High CPU Usage since Upgrade

High CPU Usage since Upgrade

av Brett C -
Antall svar: 29

Hey all,

I'm running a Moodle site on Siteground, which measures usage in CPU Seconds. Since upgrading to 4.4.1, usage has spiked 300x and the company is threatening to take the site offline.

Possibly related, I'm getting the below notification daily:

Fail delay: 1 day - Clean backup tables, logs and files (core\task\backup_cleanup_task)
Fail delay: 1 day - core\task\asynchronous_copy_task ID = 7
Fail delay: 1 min - core_course\task\course_delete_modules ID = 1334
Relevant log says:
Execute scheduled task: Clean backup tables, logs and files (core\task\backup_cleanup_task)
... started 21:35:23. Current memory use 41.7 MB.
Debugging increased temporarily due to faildelay of 86400
... used 2 dbqueries
... used 0.092118978500366 seconds
Scheduled task failed: Clean backup tables, logs and files (core\task\backup_cleanup_task),error/error_opening_file
Backtrace:
* line ? of unknownfile: call to file_logger->__wakeup()
* line 97 of /backup/util/dbops/restore_controller_dbops.class.php: call to unserialize()
* line 502 of /backup/controller/restore_controller.class.php: call to restore_controller_dbops::load_controller()
* line 243 of /backup/util/helper/copy_helper.class.php: call to restore_controller::load_controller()
* line ? of unknownfile: call to copy_helper::{closure}()
* line 233 of /backup/util/helper/copy_helper.class.php: call to array_map()
* line 262 of /backup/util/helper/copy_helper.class.php: call to copy_helper::map_backupids_to_restore_controller()
* line 54 of /lib/classes/task/backup_cleanup_task.php: call to copy_helper::cleanup_orphaned_copy_controllers()
* line 410 of /lib/classes/cron.php: call to core\task\backup_cleanup_task->execute()
* line 208 of /lib/classes/cron.php: call to core\cron::run_inner_scheduled_task()
* line 125 of /lib/classes/cron.php: call to core\cron::run_scheduled_tasks()
* line 186 of /admin/cli/cron.php: call to core\cron::run_main_process()
Gjennomsnittlig vurdering: -
Som svar til Brett C

Re: High CPU Usage since Upgrade

av Brett C -
One other note, Siteground reports:
There are three connections to the database that are almost always in a Sleep state. These connections appear to close and reopen every second or so. This behavior is likely causing a notable increase in resource usage due to the overhead associated with repeatedly authenticating and establishing connections. Each time a connection is opened and closed, CPU resources are consumed to manage this process.
Gjennomsnittlig vurdering: -
Som svar til Brett C

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

Observed on a non SG server running cron.php from command line:

php cron.php
Server Time: Sun, 28 Jul 2024 23:50:55 -0500


Ran 0 adhoc tasks found at Sun, 28 Jul 2024 23:50:56 -0500
Cron run completed correctly
Cron completed at 23:50:56 in 0.155494 seconds. Memory used: 3.6 MB. Continuing to check for tasks for 179 more seconds.

See the last line ... continuing to check for task is checking for adhoc-task ... and will loop for 180 seconds by default.   That keeps connections to DB.

Matter of fact, the cron job will run again in the middle of those 180 seconds then pick back up with whatever count it was on.

I think this is SG's issue.

The fix could be in editing config.php file - I think.

adding the following 2 lines to config.php file

$CFG->cron_keepalive='0';
$CFG->task_adhoc_concurrency_limit='1';

Don't have to restart or reboot anything.   Changes the cron job behavior ... especially the adhoc.

The next  run of cron, no more looping and adhoc_task was checked only once.   Looks like this from command line:

php admin/cli/cron.php
Server Time: Mon, 29 Jul 2024 00:07:02 -0500


Ran 0 adhoc tasks found at Mon, 29 Jul 2024 00:07:02 -0500
Cron run completed correctly
Cron completed at 00:07:02 in 0.212924 seconds. Memory used: 4.3 MB.

The 'catch22' to this ... is if you do have something qued in adhoc_task! trist

Fix to that ... might be simply commenting out the lines you added to config.php by putting // in front of them.

Cron should, upon next run, go back to old behavior!

When adhoc_task is finally cleared/cleaned up, un-comment those lines and you are back the other way!

'SoS', Ken

Gjennomsnittlig vurdering:Useful (2)
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Pol Maragall -
In my case, it doesn't seem that the cron job is encountering the problem of continuously checking for adhoc tasks. Here is the output of my cron job:

Server Time: Mon, 29 Jul 2024 09:22:38 +0200

Ran 0 adhoc tasks found at Mon, 29 Jul 2024 09:22:38 +0200
Cron run completed correctly
Cron completed at 09:22:38 in 0.018375 seconds. Memory used: 22.1 MB.

As you can see, the cron job completes correctly without looping or checking adhoc tasks repeatedly. Therefore, it appears that my cron job is functioning as expected and not exhibiting the issue of continuous adhoc task checking.
Gjennomsnittlig vurdering: -
Som svar til Pol Maragall

Re: High CPU Usage since Upgrade (II)

av Visvanath Ratnaweera -
Bilde av Particularly helpful Moodlers Bilde av Translators

Better to separate this from https://moodle.org/mod/forum/discuss.php?d=460039, your Cron is running smoothly.

Do you have shell (CLI) access? If so just watching the output of top/htop may contain clues.

Either way provide as much information as possible, the minimum is documented in Before you post.. read this...

Gjennomsnittlig vurdering: -
Som svar til Pol Maragall

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

Let me repeat ... I don't host with SG ... my only experience with SG was moons ago when helping someone else with an issue.  Saw that SB building their own panel - not cPanel/Plesk/Cockpit/or Webmin - and did not see a Terminal icon ... couldn't get to command line.

'In my case ...'

Did you run the cron.php script from the command line?

And I shared that on a non-SG hosted server with a 4.4.highest exhibited a looping behavior *when cron run from the command line*.

So if your server isn't exhibiting the same behavior, then you are seeing the same symptom but there is different cause.   If that is true, then more digging else where.   Since none of us host with SG, educate us and tell us what sort of access you have.

Comment: shared hosting isn't a good environment for a moodle.

Bret's issue is related to clean up task - part of cron - clean up from what appears to be a failed copy course and the adhoc task for cleaning up temp area ... moodledata/temp/backup/

Is your issue really the same?   Debug would help!

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Pol Maragall -
Finally, I added these two lines to the config.php file, and the CPU usage decreased immediately.

$CFG->cron_keepalive='0';
$CFG->task_adhoc_concurrency_limit='1';
 
Thank you, everyone.
Gjennomsnittlig vurdering:Useful (1)
Som svar til Pol Maragall

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

Am not 'everyone' ... but you are welcome!

Checks for you: is your cron job now running without error?  and you do need to keey an eye on add-hoc task that hopefully will not build up!

For those that can install software via their package manger there is a a small app that acts like top only for mysql that came in handy seeing results of tinkering with config settings on server with multiple moodles (not on siteground).

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade [SOLVED]

av Visvanath Ratnaweera -
Bilde av Particularly helpful Moodlers Bilde av Translators

SOLVED

Does $CFG->cron_keepalive have no equivalent on the front-end? $CFG->task_adhoc_concurrency_limit is there.

Vedlegg Screenshot_2024-07-30_08-00-51.png
Gjennomsnittlig vurdering:Useful (1)
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Malcolm Woodman -

Ken,

You are a lifesaver!

I upgraded both our test & staging sites to Moodle 4.3.3+ over the weekend (from 4.0 & 3.11 respectively). These are hosted by SiteGround. Immediately the CPU usage went through the roof, approx 11,000 secs / hour - i.e. 3 CPUs running at 100% continuously.

I have just now implemented your change to config.php - CPU usage over the last hour is 69 secs 😊

Thanks!

Gjennomsnittlig vurdering:Useful (1)
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Jon Fila -
Thanks Ken! You really bailed me out with this one. I already had upgraded my Siteground service since upgrading to 4.5 last week and my CPU Seconds shot up from under 4,000 per hour to over 20,000! I can see it trending down now an hour after implementing your suggestion to the config.php file.
Gjennomsnittlig vurdering: -
Som svar til Jon Fila

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

@Jon ... Welcome! 

Hopefully, one of these days, SG will figure out how to setup users with CLI access ... without CLI, one is greatly limited. :|

Continually figuring out and adding to config.php isn't really a good option.

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Jon Fila -
I spoke too soon! My site was just taken down because of my CPU seconds and I'm clueless about what to do next. Trying to upgrade my plan again and see if their tech support can help.
Gjennomsnittlig vurdering: -
Som svar til Jon Fila

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

@Jon ... Sigh! trist

Maybe the following SG pages will help:

https://www.siteground.com/kb/cpu_resource_usage_graph_explained/
https://www.siteground.com/kb/fair-use-siteground-hosting/

From general impression of above info, upgrade to plan is the only path for your Moodle.

Fingers crossed for ya!

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Jon Fila -
The spike happened after I upgraded from 4.2 to 4.5. Was anywhere from between 45-400 CPU seconds and after my upgrade I'm over 20,000. I've done everything that I can think of to limit that site but it holds steady at that higher rate now. I changed some of the new features back but the CPUs didn't go down (Boost Union new font, ChatGPT integration, tried disabling a bunch of scheduled tasks.) I can't find anything that is actually causing it to happen.

I ended up upgrading my plan. I guess the good news is that I can now move all of my websites over to my new cloud server. I'll keep trying to figure it out, but it's no longer an emergency.
Gjennomsnittlig vurdering: -
Som svar til Jon Fila

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

Are we assuming legitimate traffic headed to your server?

Maybe time to look @ logs for 'blackhat' activity that is persistent.

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Jon Fila -
I thought so. There was one IP address that looked like it was trying to access a bunch of pages but I blocked it. I've been having Claude and ChatGPT trying to analyze everything for me to make changes but nothing seems to have an effect. My new plan with SiteGround is more than enough, but it's really strange that it happened right after I upgraded to 4.5. I turned off statistics and everything else I can think of.
Gjennomsnittlig vurdering: -
Som svar til Jon Fila

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

If SiteGround provides a terminal, the following commands might help you discover:

free -h
less /proc/meminfo
vmstat 5 4 -S M
top then Shift +M or Shift+P
htop
ps -e -o pid,ppid,comm,%mem,rss,%cpu --sort=-%mem | head -10

Moodle specific ...

cron job where there are adhoc_task re-scheduled to run again.

In 4.2 a nasty thing that checked adhoc task for 180 seconds more after the initial cron job completed.  Only way to see it, run cron.php from command line.

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Malcolm Woodman -
Jon - is your CPU usage still high after implementing Ken's original suggested changes to config.php?

When I moved our staging site from 3.11 to 4.4.3, I immediately got the high CPU usage. The changes to config.php addressed the problem, but only after it took me 5 days to a) realise there was a problem and b) find & implement Ken's fix.

About 2 weeks later, the whole Siteground account was blocked because of the 5 days of high CPU usage. We simply waited until the start of the following month, when the total monthly CPU usage was reset and the account became available again.

I'm about to make the same upgrade on our live site this week. If Moodle 4.5 has an additional issue causing high CPU usage on Siteground, then I'm going to stick with 4.4.3! smiler
Gjennomsnittlig vurdering: -
Som svar til Malcolm Woodman

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

As with all hosting and Moodle, 'devils are in the details'!

SiteGround
https://www.siteground.com/web-hosting.htm
Web Hosting Plans
StartUp, GrowBig, GoGeek
All 3 show WP-CLI and SSH as well as 'Add Collaborators'

(am open to a looksee)

GrowBig shows Staging ... GoGeek shows staging + git

https://www.siteground.com/shared-hosting-features.htm
Server Resources line has popups Essential, Premium, Geeky.
Geeky has CPU Seconds / Program and Script executions
4000 per hour, 40000 per day, 800000 per month.

If you have CLI access:

How to find memory hogs

free -h
less /proc/meminfo
vmstat 5 4 -S M
top then Shift +M or Shift+P
htop
ps -e -o pid,ppid,comm,%mem,rss,%cpu --sort=-%mem | head -10

Not sure there is any plugin to moodle that would gather such stats.

Cron job recommenation for a moodle is to run cron.php once per minute ... so that's 1,440 times?   BUT there are task within that cron job ... automated backups, if cleaning up a site - removing courses or modules in a course, there is recyclebin, there are many cleanup task, assignments and online grading ... prep for Annotation, etc.

Do take a look at your task list.

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Malcolm Woodman

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

So the suggested settings in config.php worked for a while ... then 5 weeks later ... issues?

When one upgrades, the config.php file is copied back into new code (if that is how you are upgrading) so the settings in config.php should be retained.

Upgrading however ... make sure everything is green in current setup environment check.   Address any reds for sure.  Might want to do the upgrade at first of the month when restrictions have been reset to their max.

Does your SG panel have a terminal icon and a git icon?

If so, updates/upgrades could be done via command line and with git - takes web service out of the loop so it's just php talking to DB server.   Not sure how that would count vs whatever, but I would think less than 100% web based.

According to what little I read about SG, they will do customizations ... might be worth checking into that!

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Malcolm Woodman -

Changes to config.php are still working now on our staging site running Moodle 4.4.3 - max daily CPU usage is 3,500 seconds, our account with siteground supports up to 10,000 seconds.

Our account got shut down because we exceeded our monthly CPU usage back in September because of the high usage over the original 5 days before I made the change to config.php - when we massively exceeded both daily & monthly allowed usage. Why it then took them 2 weeks to shut us down, I don't have a clue!

Gjennomsnittlig vurdering: -
Som svar til Malcolm Woodman

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

So you have a custom setup? 10000 is over what they show.

I have a couple of Google CE sites and I've never been able to determine how they determine billing on usage.   Maybe they don't want me to!

What erks me ... legitimate traffic vs black hat scanning - the latter keeping usage higher than it should be.   Things like CloudFlare watch FQDN ... not direct IP address. trist

'SoS', Ken

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Malcolm Woodman -

Standard StartUp plan, which allows 1000 secs/hour, 10,000 secs/day, 300,000 secs/month.

So our peak normal CPU usage of 3,500/day is acceptable. But the >250,000 secs/day we jumped to after installing Moodle 4.4.3 and before changing config.php was not.

But 5 days at 250,000 is way over the daily and monthly limits according to the plan, so I don't have a clue why they allowed us to keep running for another 2 weeks!

Gjennomsnittlig vurdering: -
Som svar til Ken Task

Re: High CPU Usage since Upgrade

av Jon Fila -
Thanks for the guidance! I was finally able to figure out how to get in to SSH, but not all options are allowed.

When I tried what you suggested:
free -h
less /proc/meminfo
vmstat 5 4 -S M
top then Shift +M or Shift+P
htop
ps -e -o pid,ppid,comm,%mem,rss,%cpu --sort=-%mem | head -10

I get:

:~$ free -h
free: Memory information file /proc/meminfo does not exist

less/proc/meminfo: No such file or directory
vmstat 5 4 -S M
vmstat: Unable to create vmstat structure

It's still a mystery to me, but now my hosting plan can accommodate it.
Gjennomsnittlig vurdering: -
Som svar til Brett C

Re: High CPU Usage since Upgrade

av Ken Task -
Bilde av Particularly helpful Moodlers

What do you lease from SiteGround (SG)?  A shared account or a VPS?

Newer SG hosting is on Google infrastructure (very powerful) and SG has it's own panel (not cPanel/Plesk/Webmin/Cockpit).  Unfortunately, they do not have Terminal access to the account where one could take care of this.

There are some command line only task/scripts one could run but no terminal/no route to resolve that way. trist

Do you have automated backups turned on and scheduled?   If so, turn that off.

In the SG panel, file browser, look at your moodledata/temp/backup/ area ... anything that is in there, erase.

With autobackup, code uses moodledata/temp/backup/ to build moodle .mbz course backups and there is a task (part of cron job) that cleans up that area.  Check autobackup settings for where the backups are going.   Default settings will send .mbz files to moodledata/filedir/ where one cannot see them by humanly recognizable names.   Best, me thinks, to set auto to a designated directory outside of moodledata (you have to manually create) where can see backups by humanly recognizable names.

Unfortunately, moodle has chosen to make copying a course or backing up a course anonymous - totally background process.

Also note, that one might have to truncate certain tables in DB related as they have a 'next run' column with epoch time stamp.  This to say, you still might see errors until the tables are updated and correct.

I don't host with SG ... but did attempt to help an OP that was hosted on SG.

'SoS', Ken

Gjennomsnittlig vurdering:Useful (1)
Som svar til Brett C

Re: High CPU Usage since Upgrade

av Pol Maragall -
Hi Brett,

I have the same problem.
Were you able to solve it?

I would greatly appreciate your response, as SiteGround is also threatening to block my site.

Thank you!
Gjennomsnittlig vurdering: -