What kind of server are you running on? How much RAM/CPU?
Windows Server 2008. CPU intel xeon x7350 2.93 GHz (2 processors) and 8 GB ram. I think is enough resource.
With 8GB of RAM you should be able to support at the least 100 concurrent users @ avg 50mb per user process which is above normal. Can you post your httpd.conf file?
What mod are you using for PHP? Handler? FastCGI? Or CGI?
Are you using local storage for everything? Cache, Sessions?
What about your SQL Server? What are those specs?