Hardware and performance

Events queue huge

 
This discussion has been locked because a year has elapsed since the last post. Please start a new discussion topic.
Picture of Conn Warwicker
Events queue huge
Group Plugin developers

We had some performance issues today, and I narrowed it down to it being the Moodle cron taking forever to run.

It seemed to get stuck at the processing events queue, as it had so many events to process. I had a look in the events_queue table and there were over 1 million records in there, dating back over 2 years.

Should those records not have been wiped after they had been processed? Would it have been trying to do all of them every time?


 
Average of ratings: -
Picture of Michael Aherne
Re: Events queue huge
Group Core developersGroup Particularly helpful MoodlersGroup Plugin developers

We've seen this kind of problem before with certain plugins. What happens is that each registered event queue handler processes it's events in order. If one fails, that event queue handler won't try to process any subsequent events for that run of the cron job and (I think) the event records are retained. It'll keep retrying this until it works, which is sometimes never.

A query we've used to find out which events are blocking the queue is this:

SELECT eqh.id, eqh.status, eqh.errormessage, from_unixtime(eqh.timemodified) handler_modified, from_base64(eq.eventdata), eq.stackdump, eq.userid, from_unixtime(eq.timecreated) event_created
FROM {events_queue_handlers} eqh
JOIN {events_queue} eq ON eqh.queuedeventid = eq.id
JOIN {events_handlers} eh ON eqh.handlerid = eh.id
WHERE errormessage <> ''

You could try running that and see what you get. It usually turns out to be bad code in a plugin.

 
Average of ratings: Useful (3)
Picture of Conn Warwicker
Re: Events queue huge
Group Plugin developers

Seems to be an error in "portfolio_handle_event":


Handler function of component moodle: s:22:"portfolio_handle_event"; threw exception :Invalid export id. Maybe it has expired

* line 944 of /lib/portfoliolib.php: call to portfolio_exporter::rewaken_object()

* line ? of unknownfile: call to portfolio_handle_event()

* line 299 of /lib/eventslib.php: call to call_user_func()

* line 343 of /lib/eventslib.php: call to events_dispatch()

* line 450 of /lib/eventslib.php: call to events_process_queued_handler()

* line 340 of /lib/cronlib.php: call to events_cron()

* line 61 of /admin/cli/cron.php: call to cron_run()


Any idea what might be causing that, or how to fix?

 
Average of ratings: -
Picture of Jeff White
Re: Events queue huge
Group Particularly helpful Moodlers
Looks like it involves this bug MDL-42802. Which version of Moodle are you running? 
 
Average of ratings: Useful (1)
Picture of Conn Warwicker
Re: Events queue huge
Group Plugin developers

2.6.2 at the moment, but will be going to 2.8 in the summer, so hopefully that will be okay in the new version.

I don't even know what the portfolio thing is to be honest, I don't think we even use it.

 
Average of ratings: -
Picture of Michael Aherne
Re: Events queue huge
Group Core developersGroup Particularly helpful MoodlersGroup Plugin developers

Sorry, I've never used the portfolio API, so I don't really have a clue what might be breaking here!

In the past, we've resorted to just deleting the broken record from the events_queue_handlers table. I wouldn't really recommend doing that unless you understand what the consequences could be though. Having said that, if the queue has been blocked for 2 years, it may not make much difference whether an event raised back then is handled now or not.

 
Average of ratings: -
Picture of Jeff White
Re: Events queue huge
Group Particularly helpful Moodlers

Do you have some form of logging for your cron jobs? I would guess that the cron job task has been failing for some time and Moodle has built up a massive list (2 years worth). Since it is impacting performance for end users, I would disable the scheduled task for Background processing for events in /admin/tool/task/scheduledtasks.php until after hours then try to run the task manually via command line after I have determined the cause. Can you provide the output of the error message in the cron job log, preferrebly with debugging on?


My events_queue is normally empty so I would take a guess that if Moodle is operating correctly that table should be fairly small. How large is you environment? 

 
Average of ratings: -
Picture of Howard Miller
Re: Events queue huge
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

I've only seen this with, essentially, broken plugins. If, say, the event has to interact with an external resource/webservice and this isn't working properly. 

The last time this happened was with a well-known plagiarism plugin wink

 
Average of ratings: -