If login is taking a long time, I would do some testing to see if its not the ldap server itself that is getting bogged down and not your web server. Just doing simple logins via command line and getting the wait times would be a good test.
As for your file caching times. I am guessing you are using something other than just local disk, most likely NFS. I would recommend checking out using Redis for most of your caching if you cant use a more enterprise storage (more money) solution.