Hello Dears,
before everything, I searched everywhere but I couldn't find a solution, I have a server (Described below) that hosts Moodle, and I have used PHP-FPM on the server to handle requests which is faster, I have configured it (as shown below), the server every day at 2:00 AM it starts to backup includes (Moodle, Moodledata, Moodle Mysql database) and it make .tar.gz file on the server the total size of these files around 90GB, then it sends those tar.gz files to another server which is backup server, so I notice during this process the CPU goes to more than 75% then it goes until reaching 100% after hour the server will be down, it not happened every day or at the specific day, it happens one day randomly, we have set a limit on the on rsync to transfer the files, even we limit the interface port of the server but it doesn't fix the problem.
Please help me, what is opinion, I really appreciate your help and support.
RAM :32GB RAM,
CPU: 8 Core
OS: Ubuntu 24.04.1 LTS
PHP-FPM config:
pm = dynamic
pm.max_children = 120
pm.start_servers = 12
pm.min_spare_servers = 10
pm.max_spare_servers = 14
pm.max_requests = 300
It could also explain why not everyday. It happens only when the two jobs overlap.
In fact, single thread processes don't max out the CPU, because waiting for the I/O system automatically throttle the process. My feeling.
To go deeper other than ask everybody who is involved, you need a good monitoring system which take a record, graphs, of many matrices in the machine. See for example How do you monitor your Moodle server? General information are in the forum documentation - Performance and Performance FAQ.
A couple of things ... maybe ... moodledata/filedir/ is really the only directory needed in moodledata ... what's size of the tar.gz ball of moodledata? Bet that's the lions share of the 'around 90Gig'.
How about progressive rsync of moodledata/filedir/ to that backup server? Progressive means mirroring (bascially) ... new files added are sync'd, files deleted are also deleted from the mirror. First run of that takes a while but there after and on a regular basis takes a lot less time.
Database dumps do lock the DB for the duration of the dump and, depending upon size of DB, does take some time ... not hours, but minutes.
During the backup creations is the site put into the maint mode? If not, cron job is running ... by chance is there a task in the cron job that pushes the total activity of server to 100%?
Running automated backups of courses?
We assume that traffic headed to our Moodle's are legit .... uhhh, probably need to consider there might be 'black hats' (persons or bots). Are you monitoring secure logs/error logs to see if there are bots poking and probing?
'SoS', Ken
Thank you for your responses.
The size of the moodledata.tar.gz file is approximately 87GB. Currently, we are using rsync to transfer the entire moodledata folder and its contents to a local backup folder on the same server, and then from there, to our main backup server.
Regarding your question: "Is the site put into maintenance mode during backup creation?" — no, we do not put the site into maintenance mode. We use the following command to back up the MySQL database without interrupting the site's availability:
/usr/bin/mysqldump -uroot -p'@$' --triggers --quick --single-transaction dbname -R > /home/moodle-backup-folder/mysql/moodle.$(date +"%F").sql
However, I'm uncertain whether this is the best approach for backing up the database while ensuring that the website remains operational, especially during times when it is being actively used by students, such as in the early morning.
We have also enabled the Cron job with the cron_keepalive set to 3 minutes. I’ve checked the logs, and there doesn't seem to be anything unusual.
If anyone has suggestions or improvements on this backup strategy, it would be greatly appreciated.
I'd strongly suggest running a read replica on your db and runs dumps from there. It will prevent any outages and row locking while it occurs. There is also no reason to not just rsync to the backup server directly rather than zipping and moving. It's all incrementals which would save a mass of data moving. This could be done hourly even with little impact. At the moment it sounds like your backup strategy is sub optimal
@Ken, I have a question regarding "A couple of things ... maybe ... moodledata/filedir/ is really the only directory needed in moodledata", are sure the only moodledata/filedir/ directory should be copied using Rsync ? what about other directories inside moodledata/ should I ignore them, in case if we face a problem with the main server, If I restore the Moodle backup through only moodledata/filedir/ it should works perfectly?
I appreciate your help.
It's my understanding that moodledata/filedir/ is the location of files that are needed by moodle code and where files uploaded to the moodle are stored.
The other directories ... following is from one of my active servers:
antivirus_quarantine cache environment lang localcache muc sessions temp trashdir
Are rebuilt if not present (cache localcache muc sessions) or as it suggest (temp) are temporary and change upon usage. A stale or in-correct cache after a migration causes issues.
'perfectly' .... does any Moodle work perfectly?
'SoS', Ken
Re: moodledata/filedir/ is really the only directory needed in moodledata?
Hi Barzan
You question hat nothing to do with the subject, which was Re: Moodle High CPU Usage then down. We follow, one question, one discussion etiquette. Maybe a moderator will split the discussion at https://moodle.org/mod/forum/discuss.php?d=463997#p1863518 and move it to General help.
Talking of moodledata/filedir/, the lang/ also contain data that will be useful in the new site. But not compulsory, what you need to do is to download the language packs again. Theoretically the recycle bin is also somewhere under moodledata/, if you want to shift it too.