Automated backups: performance issue?

Automated backups: performance issue?

by Steven Pitteman -
Number of replies: 5

tl;dr: Server performance issues, seemed mainly the automated backups. Cronjob entirely disabled: everything fine.  
Disabled automated backups, reenabled cronjob: Execute adhoc task: core\task\course_backup_task still pops up, even after 24 hours of being disabled. 


Hi,

First of all, I'm new to Moodle and moodle.org, my apologies in advance for any rookie mistakes or whatsoever!
I do am an experienced developer, mainly .NET, so I should be able to gasp at least some concepts.
So far the intro...

We are using Moodle 3.9 on Linux.



We are struggling with the performance of our server, that hard that we are receiving calls from our hosting partner to kindly ask us not to do whatever we are doing.


It seemed that our problems have to do with the cronjobs, once disabled our suspicion was confirmed.
So I started running all tasks from the cronjob-list manually, but all seemed fine. So probably an adhoc task, after some  investigation, most certainly the automated backup task (Execute adhoc task: core\task\course_backup_task keeps busy until the time-treshold is reached).

I disabled the automated course backup task, and ran the cronjob a few times at "nice -n 19" + set the niceness of the heavy consumers to 19 as well (htop).  Backups didn't appear in the list anymore, so I thought I had it sorted.

Performance stayed bad with the cronjob enabled, but not as bad as before,  That part seemed related to our MySql-instance, so we probably have to upgrade to a higher tier.

But, tonight I saw that "Execute adhoc task: core\task\course_backup_task" was again running:

Execute adhoc task: core\task\course_backup_task

... started 20:03:38. Current memory use 113.3MB.

Processing automated backup for course

How is this possible?

Bonus questions:
What can I do to improve the performance once we sorted this out?
Any ideas?
this issue and tried 


Thanks in advance,
Steven


Average of ratings: -
In reply to Steven Pitteman

Re: Automated backups: performance issue?

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
You say, you run cron.php every miniute. Although Site administration > Courses > Backups > Automated backup setup: Active (backup_auto_active) is set to Disabled, "Processing automated backup for course" appears in the logs? That is odd.

Indeed the background tasks have been overhauled in Moodle release 3.9: https://docs.moodle.org/dev/Moodle_3.9_release_notes. So somebody who's in the bleeding edge might guide you.

What i don't like is the "we are receiving calls from our hosting partner to kindly ask us not to do whatever we are doing" part. Are they Moodle specialists? If so, they should know the cause. If they are not, you acting as an intermediary between the hosting partner and us will be painful.

Is the server a managed VPS? The hosting providers have the habbit of putting all sorts of restrictions on their servers, like limiting IOPS. If such restriction suffocates the Moodle background tasks, nobody here can find that out.

You've already found out that your database performance is your bottleneck. How exactly?

BTW, the general performance documentation is linked to the header of this forum.

P.S. Personally, the tone of my posts match the person or non-person I am talking to. "Point 45" reminds me a firearm. (Smiley omitted, non-persons don't know humour.)
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Automated backups: performance issue?

by Steven Pitteman -

You say, you run cron.php every miniute. Although Site administration > Courses > Backups > Automated backup setup: Active (backup_auto_active) is set to Disabled, "Processing automated backup for course" appears in the logs? That is odd.
Indeed it is, very odd....

What i don't like is the "we are receiving calls from our hosting partner to kindly ask us not to do whatever we are doing" part. Are they Moodle specialists? If so, they should know the cause. If they are not, you acting as an intermediary between the hosting partner and us will be painful.
No, they aren't Moodle specialists, but as  we are pushing our resources to 100% usage while running cronjob, they are kindly asking us to upgrade to a higher tier or stop pushing it.

Is the server a managed VPS? The hosting providers have the habbit of putting all sorts of restrictions on their servers, like limiting IOPS. If such restriction suffocates the Moodle background tasks, nobody here can find that out.
That's something I'll have to look into.

You've already found out that your database performance is your bottleneck. How exactly?
Htop, user MySql pretty much eats CPU.

P.S. Personally, the tone of my posts match the person or non-person I am talking to. "Point 45" reminds me a firearm. (Smiley omitted, non-persons don't know humour.)
One of my many aliases online, indeed coming from the caliber.  That being said, never fired a gun in my life, I'm rather pacifist.
Average of ratings: -
In reply to Steven Pitteman

Re: Automated backups: performance issue?

by Ken Task -
Picture of Particularly helpful Moodlers

This ...

"we are pushing our resources to 100% usage while running cronjob, they are kindly asking us to upgrade to a higher tier or stop pushing it"

sounds like shared hosting. sad  Their 'higher tier' would be a dedicated VPS with specs for spec/memory/cpu's etc.?   Fun and games ... they xfer your shared hosting environment into a jailed (but dedicated) environment and then one has to figure out the jails.

What?!!!?? No virtual prize? ;)

'SoS', Ken

Average of ratings: -
In reply to Steven Pitteman

Re: Automated backups: performance issue?

by Ken Task -
Picture of Particularly helpful Moodlers
Bonus question???!!! Alright ... what do I win? I'll start with some suggestions in the 'bonus' area ...
Install MySQL Tuner. It's a perl script so doesn't require anything related to moodle.
You've not really described set up so am assuming single server running web + moodle code + DB server + storage (moodledata).
Install moosh ... it's a command line utility knife for Moodle ... you don't install via Moodle admin GUI.
https://moodle.org/plugins/view.php?id=522

How many courses? How large are the courses? (moosh has a course info command that will help)

What are your preference settings for backups? Do you really need to include ALL options?

Cron and adhoc really do need to stay running.

Did I win? smile
'SoS', Ken
Average of ratings: Useful (1)
In reply to Ken Task

Re: Automated backups: performance issue?

by Steven Pitteman -
I'll take your advice and install MySQL Tuner & Moosh.
Then I'll report back to you with complete intel.

Erm, a price? How bout a barbecue next summer? I hope Belgium isn't too far for you though.
Average of ratings: -