Here's this case, and how we've been working on it:
We have a moodle installation for a high school level program since about 2 years ago. Have been very successful and as a result we have been experiencing more demand of our courses. We have been upgrading to moodle 1.9.3 (build 2008105) with our actual setup of users as follows:
4096 users divided in four blocks: First, 1904 students, 47 teachers. Second, 1040 students 26 teachers. Third, 603 students 15 teachers. Fourth, 549 students 14 teachers.
27 courses divided in the four blocks above: 6 courses each plus + 3 optional course. Each course is divided in five units with a set between 15 to 40 activities (mainly forums, upload of archives, quizzes) At the end of each course is a final exam aleatory in secure mode. Each block is a level, on which the set of students above courses one curse at a time in a lapse of one month and the final exam on weekend (this is a point of major concurrency).
The hardware setup is as follows:
Two web servers balanced and sharing moodledata by NFS. Xeon Quad Core 3GHZ 64 GB RAM. PHP 4.449(with all extensions recommended, except curl)
One third web server with the database Mysql ver 5.0.41 Same processor and 20 GB RAM. All under Linux Fedora and apache web server. We're balancing with LVM. This configuration has been working fine for the last year until now, that de community of students have been grown.
The history is as follows:
We starting having very slow responses on server. Users just couldn't work, specially when teachers were grading. We found that the servers were fine on its loads, but the server with the database was having a lot of processing load, and having a lot of concurrent connections at rates of 400 with 1900 failed per hour. We handled this as a database corruption initially and a tuning problem. Worked on it, and did a tuning on the mysqld, on apache, and the moodle connection parameters and cache as is documented. The load diminished, but we still had a lot of problems with the work of students and theachers. The DB server still had a lot of load.
Searching for similar cases, found the same documented problems on the tables mdl_grade_grades, and grade_grades_history. We applied the patches and optimizations as suggested. And the load seemed to be lower, but still had problem with users.
Finally, we decided to partition de server: Four moodle installations (in the same three servers above), each containing one block of courses. Four databases (on the same database server and daemon). At the time things seems to be much better. The database server with the four databases interestingly is not experiencing high loads Users are working fine except block 1, the first in curricula wich has the bigger number of matriculation (1904). All four servers have the same 4096, but we deleted the corresponded blocks which have the courses they contained, thus only the matriculated users can use the courses. And the rest just idle.
We still are working with installation of block1,. We moved this alone to another server. And had better performance. But not as we desire.
Now we are deleting un-matriculated users, hoping this helps for performance. Our benchmarks show that we have improvements, and most users can work but still have problems in grading.
Any Feedback is very welcome. Thanks. Cesar S.