Dealing with course restores with users

Dealing with course restores with users

by Aaron Batty -
Number of replies: 3

Okay, folks, I have made a mess, and I would very much like to fix it.

For reasons that I shan't get into here, I have had to do a server migration the dumb and brutal way, with downloaded course backup .mbz files. I backed the classes up with the user data intact.

On the new server, I uploaded the user list from the old server to put the accounts in, as I assumed that they'd need to be there before pulling in the classes. I have since realized that when you back up user data in those .mbz files, it has the whole user in it. Would have been good to know...

Here's where the problem lies, though: I started getting errors on course restore because when I uploaded the user list, I standardized the email addresses to the university email address, which meant changing some people's accounts.

The course restore sees this as a different person and refuses to go through with the user data at all.

Realizing what the problem was, and that the .mbz files had the full users in them, I decided to just delete the student user accounts and let them come back in with classes, and then upload them again to overwrite the offending mail addresses and add back accounts that were in no class this past semester.

This results in no student accounts being restored, probably because (as I recently learned) when you delete an account in Moodle, all it does is flag it as deleted in the database. So I think it's seeing that there was indeed an account with that username, but it's since been deleted, so it doesn't import.

So, database folks, what's my way around this? Here are the ideas I have:

  • Go into the database itself and just delete all the records that have the deleted flag and try again. But will this gorf something else? I don't know that much about databases...
  • Go into the database and flip all those deleted flags to 0 and then go and change email addresses as necessary so the restores go through with user data. (See concern about gorfing something.)
  • Re-upload those accounts with the old email addresses. But will it make entirely new accounts or simply update the ones in the database to be deleted=0 and impute the new email address? If it makes new accounts, I imagine I'll be right back where I am now.
  • Re-upload the accounts with the "new" uni email addresses, and when I run into an issue with a restore, just change that handful of students' email addresses back to their old ones and try again? But, again, will that result in entirely new entries in the database, or simply "undeleted" ones?
Worst case scenario, I guess I could just restore a daily MySQL backup for the whole site from yesterday and pretend this never happened, but... in all these years of administering Moodle, I have yet to have ever had to do that, and I'm very worried that it doesn't work as smoothly as I hope.

(No, I haven't done a test restore of a database. I'm just too scared; I know that's bad. Please don't lecture me.)

For reference, I have about 2800 users, but the vast majority of them are inactive or just ones I've added in case they decide to take a class from a teacher using Moodle. There are maybe only 20 classes that need to be restored. It's not a big site.

(And please don't say, "Well, you should have just started with the moodle directory and database of your old site and upgraded!" —I know. But the massive update that I needed to do would have required stepping through various PHP and Apache versions, when I also needed to change OSes and hardware—so I just decided to start fresh and pull things in the dumb way.)

There's a lot here, I realize, but who here understands the user table in the database? I can't find anything detailed enough anywhere to even know where to start with it.
Average of ratings: -
In reply to Aaron Batty

Re: Dealing with course restores with users

by Ken Task -
Picture of Particularly helpful Moodlers

Qualifications for this response ... not a certified DB person, but have run many moodles over the years and for a variety of entities.

My 2 cents ...

In the new site, delete rows 3 -> in mdl_user table.   That will leave you with ID 1 ... which is guest ... set to manual ... do not remove.  And ID 2, the account you created when the moodle was installed.  But ... see below ..

You've not been successful in restoring any courses?  or you have but there are duplicate accounts for what should be student accounts in course backups from old server.  If latter, delete courses.

So site would have only 2 users (guest/first admin) and no courses.

Then take one of your backups from old site - if they do not have 'nu' in the filename of the backup, they should include users.  Pick the largest backup file first ... tweaks may be needed to php and possibly to mysql ... php time for a script to run ... memory a script can consume (php info in site admin menu) + in mysql max_allowed_packets - backups do have large chunks of data - such as in question bank (quizzes).

In those courses, your new admin account should not exist or there will be a conflict.   Moodle protects that admin level account.

Get that largest course backup restored and checked, then the rest should restore without issues.

Exception to what I've said ... plugins ... if you've used addons in old site, install compat versions in new site *before* restoring courses that used them.

Don't worry, at first, about theme ... don't immediately install new theme ... focus on getting courses restored first and site functioning properly before working on the 'fluff' ... ie, how it looks.

Now I know you don't want any 'virtual finger wags' but do make notes of your mistakes in this adventure and internalize them ... and do know that the version you are running will not be 'forever' ... you will have to upgrade the site ... which really is much easier if you do that when versions are closer.

Again ... my 2 cents.

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Dealing with course restores with users

by Aaron Batty -

Thanks, Ken!

I've actually done a lot of moves with .mbz files, but I was always using them without the user data in them. I had no idea that they had the full accounts in them. I wish I'd known that; it would have saved me a lot of trouble.

>You've not been successful in restoring any courses?

I've gotten courses to go in without trouble... as long as none of the students had changed their email address in their profile. If I restore without users, there's no problem; I just enroll the teacher back in after restoring. I'd just love to keep the users if possible.

I actually asked the teachers to do resets before I took the site down, but of course no one did.

>Exception to what I've said ... plugins ... if you've used addons in old site, install compat versions in new site *before* restoring courses that used them.

Everything is installed on the new site.

>do know that the version you are running will not be 'forever' ... you will have to upgrade the site ... which really is much easier if you do that when versions are closer.

Here's the longer form of why this has happened: I had been serving this off of a spare Mac using the old "macOS Server" application. They took webserver administration out of that awhile back, and their setup was so bizarre that I could never get the webserver to survive OS or Apache upgrades. I finally just started from scratch with another Mac Mini we had, and set up a standard Apache/PHP/MySQL install. I had hoped to just move over the moodle, moodledata, and MySQL database from the old server and then do an upgrade from Moodle 3.1 to 3.9, but I couldn't get the old Moodle to run in the new environment at all. So I decided to do it this ridiculous way instead. I'd been marooned for a long time.

Most of my backups aren't too big, but in the case of my personal classes (I'm the heaviest user, of course), I'm going to have to bump up those PHP and MySQL values for sure.

I have set expectations low for my teachers, but obviously I'd like the most stuff to transfer over as possible.

Thanks again, and I'll do some database surgery and try again.

I'll let you know what happens.

Average of ratings: -
In reply to Aaron Batty

Re: Dealing with course restores with users

by Aaron Batty -
UPDATE!

Not crazymaking!

It worked with another class that I had tried last night. Perhaps the courses I was working with had been reset, but unenrolled user accounts still hang around in the record? At this point, it simply wouldn't surprise me... Perhaps they hang around to preserve grade history?

I.e.: It might be working now. Preserving the below for reference.

===============================================================
Okay, here's where we stand now:

  1. I went into the mdl_user table and deleted all the "deleted=1" rows.
  2. I then tried restoring a course that had failed last night to restore with users intact over itself (not a new course). ⇒ It kept the user settings from last night; just the teacher user I had enrolled. However, it still created all the new user accounts. It just didn't preserve their enrollments in the class.
  3. I then tried deleting the previously-restored course and bringing it back in fresh ⇒ Same behavior.
  4. I then tried restoring the course to a different category with a different name to try to trick Moodle into actually treating it as a new course. ⇒ Same behavior.
This is now officially driving me mad. How does Moodle know that this course has been restored before, and that I had manually enrolled the teacher? Looking at the database (mdl_course), it actually does seem to delete courses when you ask, rather than bizarrely flagging them "deleted" and letting them sit there for literally 10 years like I just recently discovered is the case with users.

This is truly crazymaking.
Average of ratings: -