Automated backups (cli) long backuping time/higher server load

Automated backups (cli) long backuping time/higher server load

by Marius S -
Number of replies: 9

Hello,

This summer we have updated from Moodle 3.3 to Moodle 3.5 and updated our Webserver.

We do weekly automatic backups by running  php  /admin/cli/automated_backups.php

We backup ~2500 courses  to Moodledata and to separate folder, it takes 75-80hours to complete backup, total size ~100gb of .mbz files.

Could you please share your backup time?
I would like to know if this is normal time to backup higher amount of courses?


Because we have updated Moodle and Webserver in the summer when there was no load. Now there is 200-300+ students connected to Moodle all the time and automatic backup is running few days in a row, server load goes to 7.0-9.0 according to 'top' or 'htop' command.
During higher load most of the time 'mysqld' or automated_backups.php shows higher CPU% usage.

Here is load example:

load


We are not sure if automated backups are slowing down webserver or maybe Moodle 3.5 is slower or maybe we should optimize php/apache/mysql settings?

We use Database for Session Handing, should we consider disabling this option?
As default setting is Default: No

Our Webserver "hardware" is the same, it's VM with 8CPUs, 32GB RAM. We just updated to PHP7.1.20 (now using php-fpm), new version of mysql, I believe we have mariadb now, as I needed to change in config from mysqli to mariadb.


Thanks,

M.

Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi Marius,

First, some information about my server : a VM with 16 CPU and 48 GB RAM, using CentOS 7.5, with PHP 7.0.32 and MariaDB 5.5.60.

Last week, we had 7 000 different users connected per day.

I use Moodle 3.5.2 and the /admin/cli/automated_backups.php script, using a dedicated folder. It's launched at full night, when (very) less people are connected, and avoid other big processes at the same time : no Moodle statistics, and no server backups !

It backuped 2 500 courses in 5h (more details of my configuration in this discussion), generating 50 GB of .mbz files.

In my case, sessions are on disk, not in DB !

I've also optimised MariaDB's configuration (using MySql Tuner and tuning-primer tools to help me), and really think it could help you to do that.

HTH,
Séverin

Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Marius

You said:
> We backup ~2500 courses to Moodledata and to separate folder

So to _two_ places. Why?

What are their file systems? How are they mounted?

With Moodle 3.3 exactly the same operation ran smoothly?
Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Ken Task -
Picture of Particularly helpful Moodlers

M ...

Not sure that anyone's comparison stats would really help you or not - unless your hardware is same or dang close ... and that could be a very deep and detailed pursuit.   Hmmmm .... I don't like 'DEEP rabbit holes' myself. smile

Having said that ...
Serving the code and also hosting the DB.

The cli scripts take apache out of the loop ... it's just PHP and the dB then.

In your top screen snap noticed the first line ... I take it that's when server was running auto backups?

user mysql ... virt memory (SWAP Space?) is 1.5 Gig. 130% of CPU (Ouch!)

Nothing runs well using SWAP ... especially DB's.

There are two other mysql processes using 57 and 55%

A server that is an 'all in one' ... Apache/Code + DB server and DB for moodle
*requires* tweaking ... balancing act between however you are running apache (mem/proceeses) and the DB server (MariaDB).

According to a Percona article/blog posting db runs best when using 80% of the memory
of the server it's on, but don't think that's possible in an 'all in one'.  Best I've ever gotten was a little over 60%.

Suggest installing and running MySQLTuner.
https://github.com/major/MySQLTuner-perl
Séverin has already said that ... I'll +20 that suggestion!

Backups ... they are gonna be heavy IF there are modules contained in courses
that are heavey processing anyway ... like quiz.

What's preferences for backups/autobackups? ie, hitories, grades, all users, etc.
That's a factor ... isn't it?
Are you autobacking up courses you don't need to?   Like courses that are hidden, test courses, etc.

If I were you ...
**** Since it's a VM, if one could, bump memory for it ... 32Gig to 48Gig - even a little might be enough if your VM server is 'crowded'.

**** Do run tuner ... before bumping mem and after bumping mem.

My 2 cents! smile

'spirit of sharing', Ken


Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Marius S -

Hello,

Thank you for replies.

We keep max 5 course backups, 2 min., we keep in backup everything (logs, users, assignments, etc.)

After last backup we have updated "Skip courses not modified since" to 14 days (it was longer last time) and new backups finished yesterday, so I looked at times when backups started and finished and here is some numbers before and after  Skip courses update:

Before:
Total: 2555 courses backed-up, 110gb .mbz files, 2018-09-15 5:46AM  -  2018-09-18 6:18PM

After  setting "Skip courses not modified since" to 14 days:
Total: 1486 courses backed-up, 85gb .mbz files, 2018-09-22 5:38AM  -  2018-09-25 6:46PM


What is interesting that courses decreased by 1000+, size decreased by ~25GB, but backup time is almost the same (new backup even took ~20-30min longer). Do you have any idea why backup time was the same?


I have found older backups in my PC, as course backup names includes date and time here is some numbers (Moodle 3.3):
2900 courses, 96gb, 2018-01-27 5:56AM - 2018-01-28 3:30PM, so it was only 1,5 day and I would be very happy having this again smile

And I have another backups (Moodle 3.3):
2881 courses, 110gb, 2018-06-23 6:19AM - 2018-06-25 3:22PM. Here we have 2,5 days, so backups in that time increased by 1 full day.

Between 2018 January and 2018 June we had Intelliboard (learning analytics) tool installed and moved to another physical server, but VM kept the same with same 32GB RAM, 8CPUs.

And at the moment when we updated Moodle 3.5, Debian, php7, mariadb we have increased backup time to 3,5 days..


Thanks,
M.

Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi,

Like you, i backup everything, and keep 5 (max) or 2 (min) backups ; and skip courses not modified for 60 days.

The difference of time for the same number of courses can be related to the number of users connected, number of actions (in logs), and datas in different tables (if you use lot of quizzes...).

Also, having your backup running while your users are massively connected, backups are less speed (and making backups also put more charge that your server must handle).

Like Ken wrote, you should really avoid DB using the Swap!

With 32GB RAM, i think you should deserve (at least) 20/24GB for DB, and fine tune it to use it efficiently. Ideally, the whole DB should be in RAM.

HTH,
Séverin

Average of ratings: Useful (1)
In reply to Séverin Terrier

Re: Automated backups (cli) long backuping time/higher server load

by Marius S -

Hello Severin,


could you let me know what is the size of your mdl_logstore_standard_log table?

Ours is 100gb and 192,000,000+ records.

We are thinking maybe this table is an issue of slow backup time because it needs to extract all logs from this table.


Thanks,
Marius.

Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi Marius,

My mdl_logstore_standard_log.ibd file is 12 GB, for 57,500,000 records.

But i think you must know several things about the log table :

  • information about deleted courses still exists in the table
  • when resetting a course, it does not delete previous logs for this course!

Information is deletted only based on the loglifetime variable (mine is configured for 2 years / 730 days).

So, depending on the way you use your Moodle, and the number of years you used it, log table perhaps still stores a big number of totally unneeded information.

You can manually delete information about deleted courses and reseted course (before the start date) with appropriate queries if you want.

HTH,
Séverin

Average of ratings: -
In reply to Séverin Terrier

Re: Automated backups (cli) long backuping time/higher server load

by Marius S -

Hello Severin,

Thank you for information. It seems that our logs are too big...

We are facing this issue: https://tracker.moodle.org/browse/MDL-56223

If I go to random old course with assignments, with no students, and go to Report > Logs, I see that "Calendar event updated" is running everyday, multiple times per day, so it generates ton of logs and stores in mdl_logstore_standard_log table.

Would it be safe to run sql query in phpmyadmin and delete "Calendar event update" records from mdl_logstore_standard_log table?


This issue is still unresolved. I will try to post about this issue in forums, maybe someone faced this issue and has a solution.

Marius.

Average of ratings: -
In reply to Marius S

Re: Automated backups (cli) long backuping time/higher server load

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi Marius,

The log table is mainly used to track details of what has been done, and be able to browse (or search) it. It can also be used to generate statistics (if enable). So, no problem to delete things in it !

Moreover deleting "abusive" information isn't a problem but better cleaning unwanted things.

I think, seeing people facing this problem, and knowing problems generated (big logs, big backups, too much time and disk needed...), that it's worth doing things to have it resolved (soon).

Séverin

Average of ratings: -