Indeed, Apache's error log was filled with suggestions to raise MaxClients, as MaxClients is strictly bound to the memory we have on each server so as to avoid touching swap.
What I found was that Apache's default keepalives are 15 seconds. Ouch! Looking at /server-status showed a whole lot of Ks. Behind each of those Ks, there is an Apache process, holding on to ~15MB RAM each, waiting up to 15s for a client to make a new request.
In real life, however, browsers will give you a series of requests over a keepalive session, all back to back (fetch the webpage, images, css, etc). Once the page is loaded, it only makes another request when the user clicks on a link or form action -- definitely longer than 15s, and in that gap that process can serve several other users...
Instead, it will by default stay there for 15s until it declares the conversation over. Waaaaay to long to be there, hogging all that RAM, while other client machines want to talk to us.
In other words, if a browser is going to ask anything else, it'll do it within a second or two, or in a few minutes.
I lowered our keepalives to 2 seconds. It immediately freed up about half the processes. That means we can service about double the end-users with the same amount of RAM.
Yummy!
