We've been aware that the recent poor, perhaps inconsistent, service performance has been a major issue for all users and have been working hard as a team to try to deal with such issues and to find ways to improve the user experience. This has been indicated within posts on our Moodle service blog.
A recent post on the blog indicated that we've turned a real corner with performance, so I wanted to post to this forum to elaborate on the work that we've done, and to share our solution with the community.
Last Monday (22nd February) was a particularly bad day in terms of performance. This was down to a combination of factors, namely:
- Monday's are the busiest day of the week in terms of usage. I recently showed a graph at a Moodle Advisory Group meeting which indicated this - slide 5 at http://go.bath.ac.uk/fltx. The key thing to notice is the spike at the start of each week.
- We identified that one department had four submission deadlines for Monday, where hundreds of students were submitting work online. This had a rather large impact on the database server. [I've since spoken to the department, and asked if they might be able to consider staggering submission dates. Not a longer term or scalable solution, but it'd certainly help in the short/medium term].
- The same department had a couple of hundred students logging into Moodle at the start of an "open week".
The image below shows an illustration of our Moodle database load for the period Tuesday 16th February - Tuesday 23rd February.
On a typical Monday, the load is near to 1.5 (not shown on the graph above). Monday 22nd February 2010 was the exception to the rule - you'll see the spike at 2.0 for much of the day. You'll also be able to see that during the previous week, Wednesday through the Friday, the database load isn't as much and runs between 0.5 and 1.0.
As well as having this information, we receive email pings from the database every hour when there are "too many connections". On a typical Monday we might receive 3 or 4 such emails; on Monday 22nd, we got 7!
During our maintenance period on Tuesday 23rd, we implemented something which has reduced the load on the database server. Previously, the database server was used to store information about current sessions. Now however, this information is stored on the filesystem. (To take a look at this in your own installation, go to: Site Administration >> Server >> Session Handling >> Use database for session information)
A typical load for Tuesday is between 1.2 and 1.5; the load on Tuesday 23rd February (and subsequent days) was close enough to a peak of 0.2 (see graph below). The change detailed above has had a significant impact on Moodle performance, on one of our busiest days... and we didn't receive any email pings, even during the middle of the day!
The number of users logging into the service Tuesday 23rd compared to Monday 22nd was reduced admittedly, but following the change to how sessions are handled, we did not encounter the same types of problems as the previous day (or previous Tuesday's) and the impact on the database server was markedly different.
We will continue to monitor this over the next couple of weeks, but clearly the real test will come on Monday, when usage is at its very highest. However, we're confident that our improved configuration would be able to handle the same number of users (and associated activity, such as coursework submissions) in a much, much better fashion.
A few weeks ago, we released our Moodle Development Plan for 2010 to colleagues (and the Moodle community) which details further service performance projects are planned from now until June 2010. This includes:
- Moving Moodle 'log tables'
- Moodle housekeeping, in particular, deleting 20000 old users that no longer exist in our LDAP.
I'd be happy to answer any further questions about this, or indeed, our ongoing Moodle-related work at the University of Bath
