This message is to report performance problems and how we solved in Universidade Aberta (www.moodle.univ-ab.pt/moodle). The system was upgraded to 1.9 in July.
We have good hardware and all components of the system well optimized, and also we have made performance tests above the load that we expect, but in the first day of the semester, many students want to go to the courses and the system do not respond.
The database was doing about 3000 queries per second that use the table mdl_log, that has a very large table (we was maintaining records for 6 months), but even after cutting the table, the problem maintains. The 8 CPUs units of the database server were at 100%, and the two front-end servers were sleeping.
We turn off the block "Recent Activity", that was about 1700 instances (almost in all courses), and turn off the capability of the teacher and students to see reports, and the system recovered. In fact it is now answering very fast, was never before, and we have this year about 4 times the load that we have last year.
About the ability of teacher and students to see reports, the problem is that generates queries that can take 4 seconds of CPU, and depend of the size of mdl_log table. In those queries the WHERE condition starts with a time inequality, that probably would be better if it was in the last condition. This was not the main problem, but if it is turned on, the performance sometimes decreases considerably.
The block "Recent activity" it consumes too much CPU time in a heavy load situation, and when the mdl_log table is big. The difference was from night to day.
In http://docs.moodle.org/en/Performance#Performance_of_different_Moodle_modules there is no reference to this block, that was the root of our problems.
Our performance tests do not detect this situation because the block "Recent activity" and the reports was not on the pages tested. Also when we do the tests, the mdl_log table was smaller.