I'm trying to make sense of the backup procedures and scripts so I can dial in my server (or change policies) to get my backups working again.
I put the details to the bottom of this post, so you don't have to scroll too far down to get to my questions.
The problem:
Our site has 300+ courses. Schools are closed in Belgium now, so I've seen a huge increase in Moodle usage. Some colleagues have asked their students to take pictures of their workbooks and upload them. Movie clips for speaking exercises were handed in as well.
Whenever a course seems to go over approximately 2.2gb, I get a report one or two of them have failed. Evert few days, it's another course that had the problem. Which would imply that the course previously giving problems now were backed up.
If I manually back these courses up, through site or CLI (backup.php), they get done without a problem.
When I remove some materials from these courses, they backup just fine again.
I can use some help reviewing my server settings. And, as I am not really trained in php and server management (budgets, rules, ...), I've writting down some more general php questions, because I want to learn.
The main goal is, of course, to get the system going with minimal manual intervention. Preferably without restricting my users too much.
What have I tried and considered?
- Moved backup time to 2AM, to make sure the server isn't too busy.
- Cleaned out all temp backup files (I now remove the folders in there once a day).
- Increased php upload size and process timeout to values you can find below.
- Tried to get a script running to clean out all the failed backup folders in temp automatically.
- I've informed my users they need to have their users hand in smaller files (scan to pdf instead of picture).
- Flushing the backup tables (but I don't have the rights on my server).
The Questions
- Is this really about size alone?
Since it is always the largest two courses that seem to cause the problem, but the largest courses aren't always the same ones. Or is the concept 'large' more complicated and could it also depend on shared files and assignments? - Could it be another error that just times out the entire process?
- Why can I do a single manual backup of this course, but won't it work through automated backups/cron?
- How does time-out work vs cron? I presume the cron script calls subscripts. And I think the automated backup script calls the backup scrip for every course. Does a new time out timer start for every script? Or does the timer start at the beginning of the cron script for everything withing the scope of that script?
- Same question as above, but for memory usage.
- Can I flush the backup tables (log, control) while the site is live?
- Not all of my tables are converted to utf8mb4_unicode. (Scheduled for the end of this week). Could this be related?
Site and server
- 8 x 2,50 GHz CPU
- 32 GB RAM
- 640 GB SSD (55% full)
- 16 TB Bandwidth
php settings:
- version: PHP 7.1 (ea-php71)
- 64
bit
- memory limit: 4096 MB
- allow_url_fopen = On
allow_url_include = On
display_errors = Off
enable_dl = On
file_uploads = On
max_execution_time = 600
max_input_time = 500
max_input_vars = 15000
memory_limit = 4096M
post_max_size = 4000M
session.gc_maxlifetime = 1440
upload_max_filesize = 4096M
zlib.output_compression = Off
Error logs
Moodle backup log:
Last two lines are always the same.
10:50:02 - calculating controller checksum 65c49d43238f8c300cc74d290e909d07
10:50:02 - loading controller from db
Error_log
I've left out all the 'requireloginerror' entries. I've found two other types of error.
[01-Apr-2020 06:31:51 Europe/Brussels] Default exception handler: Fout bij het lezen van de databank Debug: MySQL server has gone away
SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = ?
[array (
0 => '88a9b8033340ba30ab2f0486019175eb',
)]
Error code: dmlreadexception
* line 486 of /lib/dml/moodle_database.php: dml_read_exception thrown
* line 1245 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 1571 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
* line 1543 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
* line 1522 of /lib/dml/moodle_database.php: call to moodle_database->get_record_select()
* line 327 of /lib/classes/session/manager.php: call to moodle_database->get_record()
* line 98 of /lib/classes/session/manager.php: call to core\session\manager::initialise_user_session()
* line 794 of /lib/setup.php: call to core\session\manager::start()
* line 29 of /config.php: call to require_once()
* line 30 of /index.php: call to require_once()
[04-Apr-2020 21:28:56 Europe/Brussels] Default exception handler: Kon bestand"1/core/preview/0/thumb/27e5128f15aa6156f1c2c09a7698e716c64417e3" niet maken Debug: Duplicate entry 'da810284efd3b7a236b3264b04d40329d683dfff' for key 'mdl_file_pat_uix'
INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,timecreated,timemodified,userid,source,author,license,status,sortorder,contenthash,filesize,mimetype,pathnamehash) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
[array (
0 => 1,
1 => 'core',
2 => 'preview',
3 => 0,
4 => '/thumb/',
5 => '27e5128f15aa6156f1c2c09a7698e716c64417e3',
6 => 1586028536,
7 => 1586028536,
8 => NULL,
9 => NULL,
10 => NULL,
11 => NULL,
12 => 0,
13 => 0,
14 => 'c3d710d587e5f43dcd0cfb4e5ce836697a9f1c10',
15 => 5860,
16 => 'image/png',
17 => 'da810284efd3b7a236b3264b04d40329d683dfff',
)]
Error code: storedfilenotcreated
* line 1452 of /lib/filestorage/file_storage.php: stored_file_creation_exception thrown
* line 438 of /lib/filestorage/file_storage.php: call to file_storage->create_file_from_string()
* line 242 of /lib/filestorage/file_storage.php: call to file_storage->create_file_preview()
* line 2609 of /lib/filelib.php: call to file_storage->get_file_preview()
* line 88 of /draftfile.php: call to send_stored_file()