Moodle in English: Moodle crashing after upgrade to 2.4.7

Moodle crashing after upgrade to 2.4.7

by Michael Hughes - Tuesday, January 21, 2014, 8:30 PM

We just upgraded to 2.4.7 and we appear to be having performance issues.

We run 4 web servers behind an HA proxy load balancer, with a single DB machine.

The moodle data directory is mounted on the DB machine from a SAN and exposed to the front end web servers via NFS.

Following the upgrade to 2.4.7 we're finding that the DB machine starts off with a handful of connections, but this rapidly ramps up to over 500 db connections.

This then causes the whole service to fall over.

The only change introduced is the upgrade to 2.4.8 from 2.4.7+ and the addition of a new assignsubmission plugin and gradebook report.

Ideas?

Re: Moodle crashing after upgrade to 2.4.7

by Howard Miller - Tuesday, January 21, 2014, 9:02 PM

What operating system is this (these)? MySQL or Postgres?

What else is going on? What does 'top' say? How much memory are you using on front-ends and db server?

Can you get into the db server to show what the connections are doing (if anything)?

How many network connections (it *could* be coincidence that you happen to have a massive overload just as you came back up)?

How long does it take to run out of db connections and how is the system being used at the time? Is this with real users (what's it like in maintenance mode)?

Delete the new plugins (if you haven't already). Shouldn't be the issue but it does eliminate them.

Are you *sure* (and I mean prove it in a court of law) that you didn't change anything else? Don't just say 'no'

Need a clue...

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Michael Hughes - Tuesday, January 21, 2014, 10:29 PM

Further information:

This is with real users and we're running MySQL with 32gb of memory

According to our infrastructure person server is running at >90% load (sorry don't have the top output from the time)

The DB server is *only* running MySQL and the exposing the moodle data dir via NFS to the application nodes.

HAProxy was indicating that it each node was dealing with a around about 180 session on each node (each has 16gb of memory), and we could literally watch it run itself into the ground in less than 5minutes.

In maintenance mode it is as happy as larry (we're currently seeing about 300 users hitting the maintenance message according to the google analytics) so I'm not discounting the possibility it is just a "shock" reaction caused by the influx of users.

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Howard Miller - Tuesday, January 21, 2014, 10:43 PM

*which* server is running at >90% load? Again, if that's the MySQL server you need to be running 'SHOW PROCESSLIST' to find out what it is doing. If that's the web server(s) you need to be running 'top' to see what they are doing. Check the load averages too.

Is 180 sessions/node a lot? I don't think you said how many nodes you have but, if you are who you think you are, we are talking quite a large University, yes

EDIT:
4 nodes - it does seem a lot but you may be experiencing 'F5 panic'

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Michael Hughes - Tuesday, January 21, 2014, 11:14 PM

Yes the MySQL server was running at 90%, I did have a show processlists running but I've not retained (dumb part by me) the output.

From what I recall there were a lot of them sitting saying "Sending data" and there were a large number of "SELECT name, value FROM mdl_config;"s, and the queries dealing with role assignments etc.

180 sessions does seem a lot, normally we'd see this around about 20-30 sessions max

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Howard Miller - Tuesday, January 21, 2014, 11:40 PM

A complete guess... some sort of network issue between the front-ends and the db server? Although, I believe 'sending data' doesn't mean what it sounds like it does.

The sessions build up because your students go into panic mode and start constantly refreshing the pages. Once something starts that can result in slow page loads you can get into a lot of trouble quickly with relatively large numbers of users.

Could anything else have gone wrong that has killed your response? How is your caching organised?

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by J S - Tuesday, January 21, 2014, 11:49 PM

The best guess I can come up with from your description is that the DB server is struggling which is causing your environment to fall over. I would take a look at the DB server and look for the longest running process(es) and identify them and then kill them. Make sure you save them off before killing them so you have a record of what needs to be optimized. Once you kill off the thing that is causing your processes to stack up, then it should catch up.

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Michael Hughes - Friday, January 24, 2014, 7:11 PM

interestingly this has just happened again and looking at the innodb status we're seeing lots of deadlocks on the database,

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Samuel Witzig - Wednesday, January 22, 2014, 12:18 AM

We had a (maybe?) similar problem with Moodle 2.4.5 or 2.4.6, which was related to https://tracker.moodle.org/browse/MDL-35685. LIIP (Moodle-Partner) wrote a fix for us (they essentially backported the solution from Moodle 2.5 back to 2.4). Unfortunately, the fix was not backported into the general Moodle 2.4-branch on moodle.org, so our admin has to apply the patch with every 2.4-update. I'm not sure whether it's the same problem you have, but it might provide some hints...

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Michael Hughes - Saturday, January 25, 2014, 12:14 AM

Well after today's session we're seeing in the processlist (an apologies for it not being in \G format, I didn't grab this) about 109 rows along the lines of when doing SHOW PROCESSLIST;

| 11774 | moodleuser | singer:38460                 | strathmoodle | Query   |    0 | Waiting for query cache lock | SELECT * FROM mdl_course WHERE category = '0'                                                        |

| 11775 | moodleuser | point.mis.strath.ac.uk:50056 | strathmoodle | Query   |    0 | Waiting for query cache lock | SELECT * FROM mdl_course WHERE category = '0'                                                        |

| 11776 | moodleuser | nimbus:48373                 | strathmoodle | Query   |    0 | Waiting for query cache lock | SELECT * FROM mdl_course WHERE category = '0'                                                        |

| 11777 | moodleuser | hydro.mis.strath.ac.uk:42169 | strathmoodle | Query   |    0 | Waiting for query cache lock | SELECT * FROM mdl_course WHERE category = '0'                                                        |

| 11778 | moodleuser | back.mis.strath.ac.uk:38675  | strathmoodle | Query   |    0 | Waiting for query cache lock | SELECT * FROM mdl_context WHERE contextlevel = '10'                                                  |

Average of ratings: -

Re: Moodle crashing after upgrade to 2.4.7

by Dan Poltawski - Thursday, January 30, 2014, 1:02 PM

Hi Michael,

Looks like others have seen these problems with the mysql cache being locked up - there is a related discussion to this over at the developers forum: Stampeding the MySQL query cache.

Average of ratings: -

Hardware and performance

Moodle crashing after upgrade to 2.4.7