Moodle in English: 2.4.5+ (Build: 20130712) - intermittent performance issues

2.4.5+ (Build: 20130712) - intermittent performance issues

by Susan Mangan - Thursday, September 5, 2013, 3:24 AM

Hi we just upgraded to 2.4.5+ (Build: 20130712) and have been experiencing some intermittent performance issues and can't pinpoint what is happening to cause this.

At the time of the extremely slow page loading, our DB and VM check out in terms of cpu, memory, etc.

I'm just wondering if anyone else has experienced anything out of the norm with this particular build?

I have a feeling it might have something to do with the automated backups we have scheduled for twice per week (even though they are scheduled for 3 AM, not during peak time) - I just see some odd queries continuously occurring when I'm looking at server connections via MySQL Workbench - ones I don't generally see. And when the system slows down to a halt it looks like there are a lot of these built up including session locks. This is an example of the query that continues to appear during slow periods:

UPDATE mdl_backup_courses SET courseid = '2673',laststarttime = '1373128889',lastendtime = '1373128900',laststatus = '3',nextstarttime = '1378461600' WHERE id='2675'

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Guillermo Madero - Thursday, September 5, 2013, 4:09 AM

Hi Susan,

If the backup courses table is being updated, I would say that the backup process is running then. Do you have a really big number of courses, or are they very large? Sounds like a good idea to investigate and see if the backup process is the cause.

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Susan Mangan - Thursday, September 5, 2013, 4:27 AM

Thanks - how do I determine when a particular process runs?

I receive alerts that the automated backup process has occurred and that it was successful. When I review the Backup Reports the last timestamp I see is approximately 2-3 hours past the time it started.

I recall seeing some sort of bug where automated backups keep attempting to continue if they did not finish, and this is my gut feeling however I'm not sure how I determine whether or not a specific process is continuing to run to confirm this is happening. I'll check the actual table to see if the records are increasing...

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Guillermo Madero - Friday, September 6, 2013, 1:03 AM

Hi Susan,

Moodle uses an algorithm to distribute and run the different tasks it needs to do, but the automated backups should run at the scheduled time defined in the automated backups settings. You could use the tools of the OS to check which processes are running.

I just checked the SQL query you posted:

UPDATE mdl_backup_courses SET courseid = '2673',laststarttime = '1373128889',lastendtime = '1373128900',laststatus = '3',nextstarttime = '1378461600' WHERE id='2675'

It is reporting a status value of 3

http://docs.moodle.org/24/en/Automated_course_backup#Backup_status

so the process is failing for some reason (I'm assuming that 0=OK, 3=Error). Have you checked your web server error log?

Also, I went through the timestamp conversion and I got this:

laststarttime = Sat, 06 Jul 2013 16:41:29 GMT
lastendtime   = Sat, 06 Jul 2013 16:41:40 GMT
nextstarttime = Fri, 06 Sep 2013 10:00:00 GMT

There's a noticeable shift in the scheduled time.

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Susan Mangan - Friday, September 6, 2013, 6:21 AM

That's interesting... so it looks like what is happening is that there is a process constantly running that is looking at old backups from July, that had failed apparently, and rescheduling them to the next time which is accurately set to Friday.

We did go through a transition period where we had the backups turned off, as well as switching from saving the backup in the course directory to a remote server which failed a couple of times... so perhaps this is just moodle's way catching up some messy backups...

Whether or not this was(is) the cause of our extreme performance hits is yet to be seen!

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Guillermo Madero - Friday, September 6, 2013, 1:11 PM

Yes, it seems you'll have to closely monitor everything around the automated backup process.

Do not discard other possible causes for the erratic performance. Consider normal processes like antivirus scanning or updating. Most programs usually "call-home" and update automatically by default.

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Susan Mangan - Thursday, September 12, 2013, 12:01 AM

Ack! It's happening again and I cannot figure out why updating the records for these old course backups is slowing our system down to a halt.

We have an automated backup process that runs at 3 AM on Tues and Fri. The backups are being written to a different server and I know this can be significantly slower however, I have a report indicating the backups have completed successfully approximately 1.5 hours after they started and yet, 4 hours later the database seems to be still updating old course records... is this normal?

An example query for a particular course is: the last backup was in July and the next backup will be schedule for this Friday. Which seems accurate enough... given we didn't have the automated backups running for a time during the summer.... But the question is why is this slowing our system down? Why the build up in the MySQL Workbench view? (see attached screenshot)

You'd think there might be a simple database or php setting that could be made so it could handle all these old updates without slowing the system down to a halt? Oh, if it were only that simple

I apologize in advance for the cross-post but I think I might post this in the backups forum as well since I now know this is the cause. However, with that said the resolution might still be related to a performance setting?

Thanks again in advance for any insight here!

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by CP Lau - Thursday, September 12, 2013, 5:37 PM

Hi Susan,

May i understand more about your performance problem?

1) what do you mean by "slow down to halt"? Do you mean the moodle system does not respond at all while the web browser is just kept loading for a long time?

2) what do you do when the Moodle server is halted? restart the mysql?

3) what is the load average of the OS when the performance problem happens?

4) when does the performance problem usually happen? when the backup is in progress ONLY?

5) how many courses and users are there in your Moodle site?

Thanks

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Susan Mangan - Friday, September 13, 2013, 1:00 AM

Hello, and thanks for replying! We've discovered that our intermittent performance issues are due to the automated backup process. I've started a new thread here: https://moodle.org/mod/forum/discuss.php?d=237841#p1032856 so I can delve into what is happening with that process a little more.

But, to answer your questions:

1) what do you mean by "slow down to halt"? Do you mean the moodle system does not respond at all while the web browser is just kept loading for a long time?

- Yes that is correct. It takes an extremely long time for page loads when we see these updates happening to the mdl_course_backup table (see screenshot above). Page loads can be up to 10-15 mins.

2) what do you do when the Moodle server is halted? restart the mysql?

- We contemplated restarting mysql however over enough time the problem seemed to fix itself and then re-occurs 1-2 days after the next automated backup occurs.

3) what is the load average of the OS when the performance problem happens?

- in all honesty I'm not sure. I've used top from time to time but we have an infrastructure group that tends to look after monitoring this kind of thing and whenever I have asked them to look into it the report back is that there are no performance issues whatsoever with cpu, memory, etc. I'm starting to look into learning more about this and capturing my own historical performance stats via top so I don't have to depend on others to give me this info.

4) when does the performance problem usually happen? when the backup is in progress ONLY?

- The automated backup process is not in progress as far as I know. That's the weird thing. It is set to run at 3 AM and I get a report indicating it has been completed successfully at 4:30 AM. Then, at around 8 AM when I get to work, I load MySQLWorkbench and see the screen full of these queries which is abnormal. Oddly enough, I sometimes continue to see the same problem the following day for a period in the morning. Then everything is fine until the next automated backup occurs and the process repeats itself.

5) how many courses and users are there in your Moodle site?

- I would have to do a query to give you exact numbers as far as users go but using the bulk user actions tool we have 6013 users that have accessed our system after September 1st of this year and we have approximately 550 courses that are active right now.

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by CP Lau - Monday, September 16, 2013, 11:34 AM

Dear Susan,

It is great to hear that the performance issue is solved.

Cheers

Average of ratings: -

Re: 2.4.5+ (Build: 20130712) - intermittent performance issues

by Samuel Witzig - Tuesday, September 17, 2013, 7:32 PM

Just another comment regarding the performance issue with Moodle 2.4.x: We also had a huge performance issue with our moodle 2.4.6. While the webserver had very low load, the database server was fully using its cpu-ressources. This blocked access to our moodle completely. The problem was a very inneficient sql-query (https://tracker.moodle.org/browse/MDL-35685), which Adrian fixed for us (he backported the 2.5-solution to 2.4). I wonder why this issue has been solved for Moodle 2.5, but not been backported to the Moodle code on 2.4/2.3 ... Not sure whether it's the same issue you discussed here, but might help you.

Average of ratings:Useful (1)

Hardware and performance

2.4.5+ (Build: 20130712) - intermittent performance issues