Dear all,
I have a 3.11 Moodle Instance with a lot of Courses, Students and Quizzes. I wonder how I can migrate these information into a new Moodle Instance based on higher Moodle version, say Moodle 4.2.1 ?
Thanks you in advance for your help.
The 'devils are in the details'! So far we know what the job is and desired goal ... what we don't know are the 'devils'!
Platform? (OS) Are you averse to using command line?
So the quickie ... 2 ways:
Migrate the entire site then march the moodle upwards.
Stand up a 4.highest server, setup categories that are the same as old server, restore course backups.
One of the things you would want to do ... on old server know the plugins you have installed and check Moodle plugins site to see if those plugins have a compat version for your destination version of Moodle. Themes are plugins.
References:
https://moodledev.io/general/releases
Using above resource see what the requirements are for higher versions of Moodle core.
https://www.syndrega.ch/blog/#php-and-dbms-compatibility-of-major-moodle-releases
https://docs.moodle.org/500/en/Moodle_migration
https://docs.moodle.org/500/en/Git_for_Administrators
Note: the version numbers in the URL's above are the same docs one would use for your version of Moodle.
'SoS', Ken
Dear Ken Task,
Thank you for your detailed guide. Indeed I dont want to migrate the whole site, because I just want to clean up the data a little bit while moving to a new site / new moodle version (especially we have been using 3.11 for 4 years already).
The problem I am facing is that backup each course becomes a cumbersome task. With GUI, the restoring process always finishes with getting stucked somewhere over 90%. So I have to do that with commandline, one by one. Furthermore each result backup file is quite big, in a range of GBytes. Eventually we have only ~ 100 courses, but the process seems long and space consuming...
So I take the opportunity to ask for your recommendation on the following tasks I am trying right now:
1-What is the efficient options I should check while backing up a course? A course having 70 topics consisting 5 activities each (not always having video) can create a backup of several GBytes seems strange for me.
2-Using commandline call to backup course, can we backup a full category of courses ?
3-I will check the upgrading method for Moodle, but in brief, I just download the code folder of the new Moodle version, setting the config file pointing to the old DB (the DB server is likely upgraded previously to an appropriate version) and moodle-data, then just visit the new moodle site, and observe the code running the upgrading process automatically, won't I? Can I make a far upgrading, say from 3.11.6+ directly to 4.2.1+
Thanks again for your precious guide.
Thank you for your detailed guide. Indeed I dont want to migrate the whole site, because I just want to clean up the data a little bit while moving to a new site / new moodle version (especially we have been using 3.11 for 4 years already).
The problem I am facing is that backup each course becomes a cumbersome task. With GUI, the restoring process always finishes with getting stucked somewhere over 90%. So I have to do that with commandline, one by one. Furthermore each result backup file is quite big, in a range of GBytes. Eventually we have only ~ 100 courses, but the process seems long and space consuming...
So I take the opportunity to ask for your recommendation on the following tasks I am trying right now:
1-What is the efficient options I should check while backing up a course? A course having 70 topics consisting 5 activities each (not always having video) can create a backup of several GBytes seems strange for me.
2-Using commandline call to backup course, can we backup a full category of courses ?
3-I will check the upgrading method for Moodle, but in brief, I just download the code folder of the new Moodle version, setting the config file pointing to the old DB (the DB server is likely upgraded previously to an appropriate version) and moodle-data, then just visit the new moodle site, and observe the code running the upgrading process automatically, won't I? Can I make a far upgrading, say from 3.11.6+ directly to 4.2.1+
Thanks again for your precious guide.
Responses to your numbered items:
1 - not so strange really.
2 - can we backup a full category of courses
No, but you can use the backup.php script to loop through course ID's of the courses in a category and save to a mount point that is on the new server and which is really a moodledata/repository/ filesystem called restores. restores goes deeper ... a directory for oh, say 'ss' courses ... (Social Studies).
Then, on new server, could use the GUI to restore all those to a category that matches the old server.
2 catch 22's to using restoring course backups to a new higher version of Moodle
1- plugins. New server must have all the same plugins that existed on old server - especially any quiz addons.
2-if you restore via GUI, and IF the 4.1.highest restore process is now totally a background process ... you have to wait on that screen until it finishes. Things like web server time outs, memory for a script to consume etc. come into play.
But, you could, for 2- above use command line to restore the courses to a category in a looping bash shell script ... one at a time ... and you might have to put that command in a 'no hang up' (nohup) wrapper or your ssh session might time out!
Example of script for backups you would have to adjust:
niclbackups
cd /var/www/html/admin/cli/;
for i in `cat /var/www/html/admin/cli/cids.txt`
do
echo "Course ID in que:" $i;
php backup.php --courseid=$i --destination=/mntpoint/onnewserver/
done
ls -l /mntpoint/onnewserver;
echo 'Done!';
On new server and run from code/admin/cli ... here's the help on the restore_backup.php script.
php restore_backup.php
Restore backup into provided category or course.
If courseid is set, course module/s will be added into the course.
Options:
-f, --file=STRING Path to the backup file.
-c, --categoryid=INT ID of the course category to restore to. This option is ignored when restoring an activity and courseid is set.
-C, --courseid=INT ID of the course to restore to. This option is ignored when restoring a course and the categoryid is set.
-s, --showdebugging Show developer level debugging information
-h, --help Print out this help.
Example:
$sudo -u www-data /usr/bin/php admin/cli/restore_backup.php --file=/path/to/backup/coursebackup.mbz --categoryid=1
$sudo -u www-data /usr/bin/php admin/cli/restore_backup.php --file=/path/to/backup/activitybackup.mbz --courseid=1
Due to the size of courses you've mentoned ... one could create a listing of course.mbz files - those are usually long but include the course short name and course ID as it existed on old system, and loop through those restoring to a category you have created for them in advance. Don't have ane example script like that but it is similar to the one above for backups.
Again, large course restores might have to be put into a nohup wrapper. Example:
nohup php ./restore_backup.php --file=/path/to/backup/coursebackup.mbz --categoryid=1 &
That will launch the command and exit starting a nohup.out log at that same location which you can watch the progress using:
tail -f ./nohup.out
'SoS', Ken
Dear Ken Task,
Really appreciate your help. Thank you very much for your script. I will give it a try.
Just one more question. If I select to backup also users and groups, these users will be created / overwritten in the new moodle instance?
Thank you again for your kind help
BR.
Really appreciate your help. Thank you very much for your script. I will give it a try.
Just one more question. If I select to backup also users and groups, these users will be created / overwritten in the new moodle instance?
Thank you again for your kind help
BR.
Welcome!
"If I select to backup also users and groups, these users will be created / overwritten in the new moodle instance?"
That's one of the catch 22's using this method ... yes, users included in the backup will be created - not overwritten.
But, if an admin level user has been made a teacher in a course on original server and you restore to new server, potential snafu with role conflicts.
And, authentication needs to be setup same way before restore of course.
Comment: depending upon the number of courses, this method will really be the one with the most work.
A site migration brings everything across, but one might have to dummy down version of PHP (package manager of OS) before the 3.11 code can run. And one needs to check out MySQL/MariaDB version as well.
Once the 3.11 up and running one could get the code under git versioning and that, by far, is the best way to update current code and upgrade to next Moodle version upwards.
If you use git, you won't be in the situation you are in now! . And it's really easier to maintain the version you are running. Considering that moodle releases updates to code every week.
'SoS', Ken
Dear Ken,
I likely use the migration method. The backup / restore courses across the Moodle versions causes the fact that the quiz does not work probably although I just have conventional Multichoice questions (available in both version) in that quiz.
Another thing I would like to ask you is that whether the upgrading process does modify somethings in the moodle_data ? Can I just run a "transient" moodle instance, having 3.11 DB and 3.11 code but with a dummy moodle_data, then I perform the upgrading process to 4.x, and finally if everything goes well: I see users, courses' structure,... I can clone the 3.11 moodle_data for the upgraded moodle instance.
You can ask why I don't just clone it BEFORE running the upgrading process
. Of course I have to backup the current system, but the simple/fast backup/restore is to clone the DB and the moodle_data (and of course the code). In my case, DB ~ 300GB and Moodle_data takes about 600GB or more. So I think I can save the space / time in case something goes wrong (cloning 600GB then cleaning up that 600GB, also I need 600GB space more). But if the upgrading process needs modifying also the moodle_data, it is inevitable to clone it before upgrading.
Thanks for your advices
Son
I likely use the migration method. The backup / restore courses across the Moodle versions causes the fact that the quiz does not work probably although I just have conventional Multichoice questions (available in both version) in that quiz.
Another thing I would like to ask you is that whether the upgrading process does modify somethings in the moodle_data ? Can I just run a "transient" moodle instance, having 3.11 DB and 3.11 code but with a dummy moodle_data, then I perform the upgrading process to 4.x, and finally if everything goes well: I see users, courses' structure,... I can clone the 3.11 moodle_data for the upgraded moodle instance.
You can ask why I don't just clone it BEFORE running the upgrading process
Thanks for your advices
Son
Sounds like what you really need to do is cleanup the current 3.11 first! Goal is to reduce size ... code is out of the question, but DB isn't. There are quite a few tables that are really 'trackers' and one could truncate them to re-coup overall size of a DB dump.
What are *all* the settings for time to retain data? Can tell ya there are too many to look at! :|
Bad idea not to migrate everything together ... DB, code, + minimal modledata/filedir/ - most of moodledata directories are cache related and should be purged upon migration - new server - new environment, etc.
One of the largest tables in a moodle could be mdl_logstore_standard_log ... the 'who done it' log.
If you are migrating to another server consider if you really need all that old who done it data. Could truncate that table and free up a bunch of DB space ... I bet. But, I really don't know ... cause it's your system and I've never seen it!
There are directories in your moodledata that could contain junk ... example: moodleata/temp/backup/
A production 4.5.6+ site
code - 1.4G
DB dump - 2.2G
177G moodledata/filedir
178G for all of moodledata
155 courses
7802 users
I don't do a tar ball backup of moodledata ... too large ... takes too long ... rather a progressive rsync to another directory on same server.
What's yours? you can give more exacting info via du -h source.
'SoS', Ken
On current 3.11 server issue the following and share back:
uname -an
php -v
mysql -V
du -h
Same on new server.
When you backup current site, don't save to local, but to a mount point on new server.
Commands issued from inside moodle code directory:
tar -cvf /mnt/clone/moodle-code-456+-$(date +%Y%m%d%-H%M%S).tar ../documentroot;
tar -cvf /mnt/clone/moodledata-data-min-456+-$(date +%Y%m%d%-H%M%S).tar /var/www/moodledata/filedir;
mysqldump -u root DBname > /mnt/clone/moodle-456+-db-$(date +%Y%m%d%-H%M%S).sql;
You would change version number above to your 311x.
'SoS', Ken