Long running analytics tasks, expected?

Long running analytics tasks, expected?

by Séverin Terrier -
Number of replies: 8
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi,

I've just upgraded my main Moodle site from 3.5.7 to 3.7.1+ (20190816) two weeks ago.

Since then, i've got two tasks always running very long time, each time they're launched:

  • \tool_analytics\task\predict_models always takes more than 1h30
  • core\task\analytics_cleanup_task always takes between 1h and 1h15

Is it something expected?

For information : my server is based on CentOS 7, uses PHP 7.1 and MariaDB 5.5.60. It has plenty of RAM, and no problem apart these tasks.

Séverin

Average of ratings: Useful (1)
In reply to Séverin Terrier

Re: Long running analytics tasks, expected?

by David Monllaó -
Bonjour Séverin,

Thanks for reporting this potential issue. A new analytics model is enabled by default in Moodle 3.7.1, it sends reminders about upcoming activities due to students. To do it, it iterates through the list of users in the site, so the first time the model is executed it can take some good time. After this first execution, the analytics API skips all the users who have already been analysed, they won't be analysed until +1 week (this is the default value for this model).

The system has been designed keeping performance in mind. The number of database queries required to complete the task the first time depends on the number of users and the number of activities due for each of these users, but after that the number of queries should be really low (around 50 + the number of users in the site). The \tool_analytics\task\predict_models task shouldn't take that long, the equivalent cost of this task should be less than the cost of having each user in the site viewing their dashboard.

Some comments:
  • I am puzzled about the 1h15 required by core\task\analytics_cleanup_task. It shouldn't be that long as it is just running a few bunch of db queries. A couple of these queries have IN statements with a large number of params, this is the only possible cause I can think of. In any case, we are changing the "IN" approach of core_analytics\manager::cleanup method (the method executed by core\task\analytics_cleanup_task) in MDL-66498. You could try to pull the patch for 37 and try to execute the task again. I am really interested on this specific case because I have no idea of what can be wrong and I feel like I am missing something important, probably related to mysql/mariadb not handling efficiently a large number of params in an IN.
  • What is the value of your "analytics | modeltimelimit"? The purpose of this setting is to split large processes like \tool_analytics\task\predict_models in batches. The default value is 20 minutes. It is possible that the system does not have enough time to go through the list of users in the site and it just processes a few of them, so the next time you run the task more users are processed. It would be great if you could paste the output or to summarise what can you see if you can't (the output may contain users' personal info). To know if the patch in the first point right above reduces the time taken by  core\task\analytics_cleanup_task can help us narrow the issue to mysql/mariadb with IN.
  • How many users do you have?
  • Have you enabled any new model after the upgrade? These tasks are not new in 3.7, do you remember if they used to take that long?
  • Is the students at risk model enabled? I ask it because I am assuming that the tasks are slower because of the new model, but it could also be that there is a performance regression after the API changes in 3.7.
Average of ratings: -
In reply to David Monllaó

Re: Long running analytics tasks, expected?

by Tobias Reischmann -

Hey David,
I am posting here, since you requested it in MDL-66598. I can't tell if the applied patch lead to an performance improvement in our setting, since for our setting (postgres) at least the cleanup task was not runnable before the patch. After the patch we also observe long runtimes for the predict_models task. However, our longest runs are at 21min. The cleanup task is with 7.5 sec quite fast.

For comparison:
We have the upcoming_activities_due and the no_teaching model enabled but the course_dropout disabled. We have 97497 users.
The analytics | modeltimelimit is 20 min, which would explain, why our task usually takes a bit longer than 20 min. :D

Cheers,

Tobias

Average of ratings: Useful (1)
In reply to David Monllaó

Re: Long running analytics tasks, expected?

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators
Hi David,

Sorry for the late answer, i've other urgent problems to resolve with the beginning of the new academic year...

In my installation, i have :
  • 73985 users (97187 in mdl_user, with deleted ones)
  • 3019 courses (2205 visible)
  • the "analytics | modeltimelimit" is at the default value : 20 minutes (i've just changed it to try with 2 hours)
  • since some days, the predict_models task doesn't take much time (perhaps since i've disabled upcoming_activities_due model?)
  • the student at risk model course_dropout is disabled
  • with the new academic year coming, we've reset most of the courses, deleting related logs
I've not made tests with adding the index or applying the patch, will try to do it when i can.
But it's nice to see that tracker items evolve, and perhaps a solution will soon come (thanks everybody working on it).

I wonder if i will desactivate related tasks, waiting for tracker items being closed with a solution (and making the update), to avoid charging the server... and perhaps i should take time to better read about analytics, how they work and what they provide, before activating it.

Séverin
Average of ratings: -
In reply to Séverin Terrier

Re: Long running analytics tasks, expected?

by David Monllaó -
Hi Séverin,

The tasks should finish quickly if the analytics models are disabled. There is no need to disable the related tasks. A the fix will be available in Moodle 3.7.3.
Average of ratings: Useful (1)
In reply to David Monllaó

Re: Long running analytics tasks, expected?

by Alistair Spark -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
core\task\analytics_cleanup_task is taking 3h30m to run for us so looking forward to deploying the fix. For now have disabled the task. We have 262 000 users in the system.

We didn't enable any models, but did disable one to mitigate MDL-66026

Execute scheduled task: Predict models (tool_analytics\task\predict_models)
... started 16:00:03. Current memory use 19MB.
There are no enabled and trained models to predict.
... used 1 dbqueries
... used 0.032793045043945 seconds
Scheduled task complete: Predict models (tool_analytics\task\predict_models)
Average of ratings: -
In reply to Alistair Spark

Re: Long running analytics tasks, expected?

by David Monllaó -

Hi Alistair,

MDL-66498 should have fixed the issue (available in Moodle 3.7.3). MDL-66026 should not be related to core\task\analytics_cleanup_task, I assume that you disabled the "Upcoming activities due" model.

Average of ratings: -
In reply to Séverin Terrier

Re: Long running analytics tasks, expected?

by David Monllaó -
Hi Séverin,

I am pretty sure that this report matches your issue: MDL-66599. I will have a patch ready max. tomorrow.
Average of ratings: -
In reply to David Monllaó

Re: Long running analytics tasks, expected?

by David Monllaó -
The patch is ready, feel free to test it if you have a staging environment. Alternatively, you can just add an index to 'analysableid' field for mdl_analytics_used_analysables table. Any feedback is welcomed.
Average of ratings: -