Moodle in English: backup courses outside Moodle cron on DB server

backup courses outside Moodle cron on DB server

by Alain Raap - Thursday, 14 March 2019, 4:04 PM

I've been working on a little POC to move the backup process to our data/database server and run it on this server (without Apache) webserver. Ken suggested this option and I spent some time on investigating this subject. Finally I've got it working now in our Redhat environment.
What I did was install php and an instance of Moodle (same version as on our website), wrote a little bash script that I run in the Linux cron. I took the course that didn't backup via the Moodle cron anymore and managed to make a backup that took 3 hours now instead of the 7 hours before. We use an NFS share for the Moodledata on the DB server.

Problem I encountered was the Caching store of Memcache that prevented the backup.php script to run. I managed to solve this in my backup bash script (temporary disabled/enabled the Memcache caching store in muc/config.php)

Our environment: Redhat 7, Moodle 3.1.5+, Apache 2.4, PHP 5.6, Mariadb 10.1.29

My bash script I wrote and that I scheduled in the Linux cron for Apache user:

#!/bin/bash
# ------------------------------------------------------------------------------
# Script: backup_moodle_courses.sh
#
# - Backup a list of courses with admin/cli/backup.php
#
# Writer: Alain Raap
# Date: 24-01-2019
# ------------------------------------------------------------------------------
exec > >(tee -i /var/log/backup_moodle_courses.log)
exec 2>&1

MOODLE_ROOT="/var/www/moodle"
MOODLE_DATA="/moodledata"
COURSES="${MOODLE_ROOT}/admin/cli/courses.txt"
COMMAND="${MOODLE_ROOT}/admin/cli/backup.php"
PHP="/usr/bin/php -d log_errors=1 -d error_reporting=E_ALL -d display_errors=0 -d html_errors=0 -d memory_limit=2048M"
MUC="${MOODLE_DATA}/muc/config.php"
BACKUP_PATH="${MOODLE_DATA}/backups/"

echo "${SCRIPT}: Start backup script"

# ------------------------------------------------------------------------------
# Change MUC (Remove Moodle-memcache store)
# ------------------------------------------------------------------------------
echo "${SCRIPT}: Remove Moodle-memcache store from muc/config.php"
sed -i -e "s/'store' => 'Moodle-memcache'/'store' => 'default_application'/" $MUC

# ------------------------------------------------------------------------------
# Read input file with course-id's to backup
# ------------------------------------------------------------------------------
while read COURSE_LINE ; do
   ID="$(echo ${COURSE_LINE} | cut -d',' -f1)";
   NAME="$(echo ${COURSE_LINE} | cut -d',' -f2)";
   echo "${SCRIPT}: Backup of course: " $ID " - " $NAME;
   echo "$PHP $COMMAND --courseid=${ID} --destination=${BACKUP_PATH}"
   $PHP $COMMAND --courseid=${ID} --destination=${BACKUP_PATH}
done < ${COURSES}

# ------------------------------------------------------------------------------
# Change MUC (Restore Moodle-memcache store)
# ------------------------------------------------------------------------------
echo "${SCRIPT}: Restore Moodle-memcache store in muc/config.php"
sed -i -e "s/'store' => 'default_application'/'store' => 'Moodle-memcache'/" $MUC

echo "${SCRIPT}: End backup script"
exit 0

The inputfile for the bash script courses.txt (id and fullname of courses)

1001,course 1

1002,course 2

1003,course 3

Average of ratings: -

Re: backup courses outside Moodle cron on DB server

by Alain Raap - Friday, 15 March 2019, 7:51 PM

For who's interested, I've also got my bash backup script working with our newer Moodle version (3.5.4+).

Our development environment: Redhat 7, Moodle 3.5.4+, Apache 2.4, PHP 7.1.8, Mariadb 10.1.29, Redis Caching Server 3.2.12

My bash script I wrote and that I scheduled in the Linux cron for Apache user:

#!/bin/bash
# ------------------------------------------------------------------------------
# Script: backup_moodle_courses.sh
#
# - Backup a list of courses with admin/cli/backup.php
#
# Writer: Alain Raap
# Date: 24-01-2019
# ------------------------------------------------------------------------------
exec > >(tee -i /var/log/backup_moodle_courses.log)
exec 2>&1

# ------------------------------------------------------------------------------
# Set parameters
# ------------------------------------------------------------------------------
SCRIPT="backup_moodle_courses"
MOODLE_ROOT="/var/www/moodle"
MOODLE_DATA="/moodledata"
COURSES="${MOODLE_ROOT}/admin/cli/courses.txt"
BACKUP="${MOODLE_ROOT}/admin/cli/backup.php"
PHP_CMD="/usr/local/bin/php -d log_errors=1 -d error_reporting=E_ALL -d display_errors=0 -d html_errors=0 -d memory_limit=2048M"
MUC="${MOODLE_DATA}/muc/config.php"
BACKUP_PATH="${MOODLE_DATA}/backups/"

echo "${SCRIPT}: Start backup script"

# ------------------------------------------------------------------------------
# Change MUC (Remove Moodle-redis store)
# ------------------------------------------------------------------------------
echo "${SCRIPT}: Remove Moodle-redis store from muc/config.php"
sed -i -e "0,/'store' => 'Moodle-redis'/s/'store' => 'Moodle-redis'/'store' => 'default_application'/" $MUC
sed -i -e "0,/'store' => 'Moodle-redis'/s/'store' => 'Moodle-redis'/'store' => 'default_session'/" $MUC

# ------------------------------------------------------------------------------
# Read input file with course-id's to backup
# ------------------------------------------------------------------------------
while read COURSE_LINE ; do
   ID="$(echo ${COURSE_LINE} | cut -d',' -f1)";
   NAME="$(echo ${COURSE_LINE} | cut -d',' -f2)";
   echo "${SCRIPT}: Backup of course: " $ID " - " $NAME;
   echo "$PHP_CMD $BACKUP --courseid=${ID} --destination=${BACKUP_PATH}"
   $PHP_CMD $BACKUP --courseid=${ID} --destination=${BACKUP_PATH}
done < ${COURSES}

# ------------------------------------------------------------------------------
# Change MUC (Restore Moodle-redis store)
# ------------------------------------------------------------------------------
echo "${SCRIPT}: Restore Moodle-redis store in muc/config.php"
sed -i -e "s/'store' => 'default_application'/'store' => 'Moodle-redis'/" $MUC
sed -i -e "s/'store' => 'default_session'/'store' => 'Moodle-redis'/" $MUC

echo "${SCRIPT}: End backup script"
exit 0

The inputfile for the bash script courses.txt (id and fullname of courses)

1001,course 1

1002,course 2

1003,course 3

Add this line in the crontab of your database server (runs on Saturday night 0:00 hrs):

0 0 * * 6 /var/www/moodle/admin/cli/backup_moodle_courses.sh > /dev/null

Average of ratings: -

Re: backup courses outside Moodle cron on DB server

by Alain Raap - Tuesday, 19 March 2019, 10:05 PM

The automated_backups.php script can also be started as a Linux cron job on the DB server. I managed to get this working with a Moodle instance where I only had to make a minor change in the automated_backups.php script in admin/cli. Also I had to temporary disable the caching store in muc/config.php while running the script. I also changed the settings of the Automated backup in 'Manual' instead of 'Enabled'. And I disabled the automated_backup_task in Moodle cron tasks.

Code update in automated_backups.php:

     66 //if (moodle_needs_upgrading()) {
     67 //    echo "Moodle upgrade pending, backup execution suspended.\n";
     68 //    exit(1);
     69 //}

Bash script to add to the Linux crontab:

#!/bin/bash
# ------------------------------------------------------------------------------
# Script: automated_backup_moodle_courses.sh
#
# - Backup courses with admin/cli/automated_backups.php
#
# Writer: Alain Raap
# Date: 19-03-2019
# ------------------------------------------------------------------------------
exec > >(tee -i /var/log/automated_backup_moodle_courses.log)
exec 2>&1

# ------------------------------------------------------------------------------
# Set parameters
# ------------------------------------------------------------------------------
SCRIPT="automated_backup_moodle_courses"
MOODLE_ROOT="/var/www/moodle"
MOODLE_DATA="/moodledata"
AUTOMATED_BACKUPS="${MOODLE_ROOT}/admin/cli/automated_backups.php"
PHP_CMD="/usr/local/bin/php -d log_errors=1 -d error_reporting=E_ALL -d display_errors=0 -d html_errors=0 -d memory_limit=2048M"
MUC="${MOODLE_DATA}/muc/config.php"
BACKUP_PATH="${MOODLE_DATA}/automated-course-backups/"

echo "${SCRIPT}: Start automated backup script"

# ------------------------------------------------------------------------------
# Run automated backups PHP script
# ------------------------------------------------------------------------------
echo "$PHP_CMD $AUTOMATED_BACKUPS"
$PHP_CMD $AUTOMATED_BACKUPS

# ------------------------------------------------------------------------------
# Change MUC (Restore Moodle-redis store)
# ------------------------------------------------------------------------------
echo "${SCRIPT}: Restore Moodle-redis store in muc/config.php"
sed -i -e "s/'store' => 'default_application'/'store' => 'Moodle-redis'/" $MUC
sed -i -e "s/'store' => 'default_session'/'store' => 'Moodle-redis'/" $MUC

echo "${SCRIPT}: End automated backup script"
exit 0

Copy the bash script to /var/www/moodle/admin/cli and execute 'chmod 555 automated_backup_moodle_courses.sh'.

Add this line in the crontab of your database server (runs on Saturday night 0:00 hrs):

0 0 * * 6 /var/www/moodle/admin/cli/automated_backup_moodle_courses.sh > /dev/null

Average of ratings: -

Backup and restore

backup courses outside Moodle cron on DB server

backup courses outside Moodle cron on DB server

Re: backup courses outside Moodle cron on DB server

Re: backup courses outside Moodle cron on DB server

Empowering educators to improve our world

Moodle

Support

Get Involved

Contributions

Downloads

Tracker

Development

Empowering educators to improve our world