First thing, you don't say anything about the hardware. Could you provide the details?
Second thing is about the statement: "The hardware can handle the load, lots of RAM, fast HDD, gigabit NICs etc". How do you know? Since the "performance slowed down to a stand still for a lot of users", something hit the ceiling.
Did you go through http://docs.moodle.org/en/Performance and http://docs.moodle.org/en/Performance_FAQ ?
I have spoken to the techie regarding our server configuration. He sent me the following:
Below is the server specifications
Network card(s)—There are 2 network interface of 1000Mps. These have been bonded to make a 2000Mps pipe
- Hard Disk drive Make and model- is200 Gb Virtual Disk that is part of a LUN on the SAN
- OS and version – Ubuntu -- ubuntu-10.04.2-server- x64
- Shared hardware resources (is it a virtual server?) - CPU Capacity 8x2.26 GHX Memory capacity 50 GB
- Other hardware items related to Moodle? none
PHP Accelerator—APC attached screen shots
PostgreSQL configuration file - max_connections = 250, shared_buffers = 24MB
Apache conf -- <IfModule mpm_prefork_module> increased Max clients to 250
No compression has been setup yet as issue we have been experiencing is the lack of database connections to normal users, I have attached log files to give you an idea off isuee that have been present.
First, ensure that the PostgreSQL user used by Moodle has no connections limit. As you have plenity of RAM, try increasing progressively the max_connections value (300, 350...), work_mem to 2 MB and shared_buffers to 1024 MB (and adjust the effective_cache_size accordingly). If that doesn't do the trick, try something like Pgpool or PgBouncer. Don't forget to restart PostgreSQL to apply the configuration changes
As a sidenote, after you manage to solve the database configurations problems, consider trying Cherokee instead of Apache.
How much RAM does your server have?
Do you have PHP accelerator installed, i.e., eaccellerator or APC?
Is your site gzip or mod_deflate for Apache enabled (duh, I'm sure it is)?
How many quiz questions do you display on a page? The recommended number of questions to be loaded on a page at any one time is 5.
The Server has 8Gb of RAM installed, and the PHP accelerator we are using is APC, configuration paratmeters are in the above post from Richard. Module deflate is enabled.
Just curious, do you still find an issue with Moodle if you were to turn the session handling back to a file base versus database?
Also, do you have anytype of query cache going on with PGSQL? I couldn't see it in the conf file.