Backup/restoring courses between instances in bulk

Backup/restoring courses between instances in bulk

Eric Phetteplace -
回帖数：8
Particularly helpful Moodlers的头像

Does anyone have a good solution for automating the transfer of a large number of courses from one Moodle instance to another? For instance, a tool that syncs entire course categories across Moodle instances.

We are struggling to do major system migrations (moving to a new server, moving to the cloud, etc.) without simply creating a brand new instance and manually backup/restoring individual courses upon request. That was always a lot of work but it scales so poorly that it won't work at all given the quantity of online courses we support post-COVID-19.

I love the moosh command-line utility which can help automate backups and restores a little bit, but there's still a lot of semi-manual work transferring backups between servers and we can only do so many at a time without taking up too much disk space. I don't know how well I could transfer hundreds of courses over my home internet connection. Disk space is also the reason we can't just clone our current instance and thus transition all courses to a new instance. We would like to copy a whole course category (e.g. past semester) as opposed to everything.

平均分： -
回复Eric Phetteplace

Re: Backup/restoring courses between instances in bulk

Ken Task -
Particularly helpful Moodlers的头像

Not that anyone has shared in these forums that I've seen ... but .... some comments and food for thought - depending on info not shared about this migration ...

Concept: think server to server ... no download just to turn around and upload.  wget and rsync

Hints: wget -i getfiles on the new server ... the getfiles file contains single url lines that point to a different file ... like:

https://oldserver/temparea2transfer/backup1.mbz

https://oldserver/temparea2transfer/backup2.mbz

For something massive, like moodledata/filedir ... rsync from old to new.

Why not migrate sites?   Versions too old?

'SoS', Ken


平均分： -
回复Eric Phetteplace

Re: Backup/restoring courses between instances in bulk

Rick Jerz -
Particularly helpful Moodlers的头像 Testers的头像
Your post made me wonder. Your old Moodle has more disk space than your new, which is your reason for not simply migrating the entire site. Are you not concerned that your new Moodle will run out of space?

Before doing anything somewhat major, make sure to backup your current installation (I recommend that you include your Moodle folder, moodledata folder, and your Moodle database.)
平均分： -
回复Eric Phetteplace

Re: Backup/restoring courses between instances in bulk

Eric Phetteplace -
Particularly helpful Moodlers的头像
Thanks for the responses. Just to be clear, the only information I'm interested in is bulk transferring courses from one instance to another. We have done several migrations, we know how to do them, that's specifically not what I'm asking about. For context, we are weighing the tradeoffs between creating a totally new instance and migrating a large (≈3TB data) existing one; whether or not we can transfer a subset of courses would be very helpful to know in making this decision.

> Hints: wget -i getfiles on the new server ... the getfiles file contains single url lines that point to a different file ... like:

> ttps://oldserver/temparea2transfer/backup1.mbz

Are backups exposed at publicly accessible URLs like this? That's terrifying if it's true, anyone could just download sensitive course data. I believe this idea is close to my best guess: to rush to write some brittle, convoluted code that runs moosh backups/restores across two instances. I understand how file transfer utilities are involved there but it seems far from straightforward, especially at a scale of hundreds of courses. My hope was something substantial already existed.

> Why not migrate sites?   Versions too old?

We might migrate, we're trying to decide and bulk course transfer is relevant to the decision. Cons to migration include being unable to make a backwards incompatible grading settings change, much more downtime than a new instance, and the complexity of transferring a large amount of data to a new cloud host.

> Your old Moodle has more disk space than your new, which is your reason for not simply migrating the entire site. Are you not concerned that your new Moodle will run out of space?

No, our new one will have plenty of space, but storage space is neither infinite nor zero cost, so we still don't want to duplicate terabytes of data across two instances if it's avoidable.

> Did you study the documentation page https://docs.moodle.orgen/Moodle_migration ?

See my opening comment, my question isn't fundamentally about a migration, it has to do with selectively transferring a subset of courses between instances.
平均分： -
回复Eric Phetteplace

Re: Backup/restoring courses between instances in bulk

Ken Task -
Particularly helpful Moodlers的头像

Trying to keep this short (carpel tunnel) ...

OK, get it ... been there ... done that ... 4000+ courses ... in under 8 hours.

The wget method of acquiring full course backups from a symlinked directory protected by .htaccess that allowed only the new server IP access.

Could also do rsync of full course backups from a designated directory on old server to new server designated directory.   Plus, an alt ... the designated directory for backups on old server is a mount point to a directory on new server ...  outside of moodledata or in a symlink from moodledata/repository/restores to where the backups were xferred.

New site should have categories created in advance.

Looping bash shell script using moosh command to restore courses to an existing category.   Course backups would need to be organized into folders that are associated with the new site categories.

When one does courses like this 1 catch 22 are users that had admin role assignments in old - never user ID 2.

New site should have same plugins as old pre-installed.

Bottom line, there is no utility/plugin in moodle that does what you want.  All is done via command line using utils like wget, rsync, moosh and some simple looping scripts to rebuild on new site.

Once you decide how, suggest small test runs for process at first, then do larger.

Definitely cut off access to old server once you begin.

Getting painful to continue here ... my 2 cents!

'SoS', Ken

平均分：Useful (1)
回复Ken Task

Re: Backup/restoring courses between instances in bulk

Eric Phetteplace -
Particularly helpful Moodlers的头像
OK thank you, really appreciate the response. This tells me that there isn't any existing plugin or tool, we would have to devise our own procedure. I think I could probably manage that but it is a negative to not doing an in-place migration. I hope your wrists feel better!
平均分： -
回复Eric Phetteplace

Re: Backup/restoring courses between instances in bulk

Visvanath Ratnaweera -
Particularly helpful Moodlers的头像 Translators的头像
> my question isn't fundamentally about a migration, it has to do with selectively transferring a subset of courses between instances.

OK, the two building blocks are:
1) https://docs.moodle.org/en/Course_backup
2) https://docs.moodle.org/en/Course_restore

Step 2) is predestined for Moosh https://moosh-online.com/. The question is where do you have the course backups. I assume you've activated https://docs.moodle.org/en/Automated_course_backup. What is in Automated backup storage (backup_auto_storage). If it is "Course backup file area" selecting a subset will be cumbersome. Set it to "Specified directory for automated backups". Then you'll find them all in a directory with with human readable names.
平均分： -
回复Visvanath Ratnaweera

Re: Backup/restoring courses between instances in bulk

Ken Task -
Particularly helpful Moodlers的头像

I don't totally disagree Mr. V, but ...

"... selectively transferring a subset of courses between instances."

Auto backup not good for 'subset of courses' - with auto is more an all that match options (and moodle routine for this can't do Vulcan Mind Melds with op).  If site has been using auto already, might have to start with a clean slate of related tables by truncating them all, then run auto ...
depending upon what Op means by 'subset'.

One will end up with a bunch of courses in one directory and 'un-organized' as to 'subset' (whatever that means).   That equates to more work on the new server rebuilding.

Again ... my 2 cents! 微笑

'SoS', Ken

平均分： -