Migration advice needed

Migration advice needed

by Dave Spittal -
Number of replies: 18

I recently tried to upgrade to the latest version using fantastico. I thought things were not progressing at all so I clicked the upgrade button again. After some time I was taken to the admin link and when I followed the link the page was blank. I could no longer access the site in any way. Using ftp I found that the backup files 500MB (there were two - presumably because I clicked twice) had filled my server space. Did this mean the upgrade could not complete properly?

I downloaded the backup zip file to my local computer.

Using that backup file can I recreate the old site on a new server space (since I was really needing to move it anyway)? I have tried the instructions in MoodleDocs for migrating and have the following difficulties.

Change $CFG->wwwroot . In your (possibly new) Moodle directory, change the $CFG->wwwroot variable in the config.php file for the new server

Since it is a clean install of moodle on the new site may I assume that the config.php file will be OK?

Copy data directory contents (moodledata) . Copy the contents of your data directory (check for the value in $CFG->dataroot ) to the new server. Check also that permissions remain the same on the new dataroot folder and change the value if you have changed its location on the new server.

I unzipped the backup file and there is no folder called moodledata. Is this an old post and the data folder has been renamed? Where do I find my data folder?

I am pretty confident with the changes needed to the database backup file backup.sql and can use a find and replace to change the old references. I would be greatly appreciative if anyone could help me to get my old site back up and running at the new location.

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Richard Enison -
Excuse me, stewardess, could we turn this plane around? I've already seen this movie!

DS,

Seriously, the first part of this post looks like a story I've seen before, in another thread. And I think I made some nasty comments about Fantistico at the time. If I didn't, I should have, because that's what Fantastico deserves. Just do a search of this forum for posts by me containing that word.

But this time, there's more to the story. This must be the sequel. And there are actually specific questions, some of which can actually be answered:
  • You said there were two backup files, filling up the free space in your server, possibly because you backed up twice. Yes, that is almost certainly the reason for that. I trust you deleted one of them to free up some space. Which one should be deleted? Well, if one is smaller, delete that one. Presumably the other is more complete.
  • You ask if it is necessary to change wwwroot in a clean install of Moodle on a new server? I would say so, unless you have your own domain name and you have moved it to the new server. The value of that variable is the URL by which you access your Moodle site, so it would be the URL of the new server, which is usually different from the URL of the old one. The same thing is frequently true of dirroot and dataroot, especially if the two servers are running different operating systems, such as Linux and Windows, or two different distributions of Linux. Similarly for the database parameters, especially if the new server has a different type of database from the old, such as MSSQL vs. MySQL.
  • Where is the data directory? Ordinarily, I would say that depends on whether you did a full backup of your site. If so, it should be in there. But since you did a Fantastico backup, I have no idea. Who knows what backup format that abomination uses. But I can't imagine that it would rename it. Maybe it created two backup files at the same time, until all your free space was used up, so that neither one is complete. In that case, I would delete both and do the backup again -- just once this time, please!
RLE
Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed

by Dave Spittal -

Thankyou for answering and for not being too harsh with me. One of the problems is that people like me dont have the expertise to understand all of the things in the posts.

  • The old installation was using a parked domain and I did move it to the new server space.
  • The dirroot and dataroot are the same on the new server. Everything is provided by the same hosting company so I assume there are no database incompatability issues.
  • I copied the backed up uploaddata folder and simply replaced the clean install version of that folder.I have managed to get most of the site working on the new server. I could not do another backup because I could no longer access the site.
  • I have managed to restore the courses although all of the user data seems to be lost. I think thats because I could not get phpMyAdmin to restore the whole database. The posts on that subject are also hard for me to understand. If you can point me to a clear explanation of a workaround for that issue I would be obliged.

Thanks again for the help. If I get no further that I have at present then I have tpo be happy. Having my pupils recreate their accounts is much less of a problem than me recreating the entire site form scratch.

Dave

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Richard Enison -
DS,

You're welcome. Sounds like you're in pretty good shape. I should have remembered that Fantastico (wash my mouth out with soap) calls the data directory uploaddata. But it seems you found it anyway.

Your biggest problem right now seems to be that the restore of the database is incomplete, but the only thing missing is the user account data, which your students can recreate, which is why I said you're in pretty good shape. If that's what you're asking for a work-around for, that depends on whether the data in question is actually in the backup or not. If you have the database backup in a standard format (you implied it was an .sql file) and not just part of a Fantastico backup file, it might be possible to look at it and see if it's there. But I'm sure you wouldn't want to post sensitive data like that on the forum where the world can see it. Let me know if you want to e-mail it to me and I will send my address via a private Moodle msg.

RLE
Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed

by Dave Spittal -

Thanks for your offer of help. When I extracted the downloaded zip file it contained a file called backup.sql (16MB) which I take to be the backup of hopefully the whole database. When I search the sql file with a text editor I find that all the names and account details seem to be in there. I can read their profiles. Part of the backup.sql file starts with this

LOCK TABLES `mdl_user` WRITE;

/*!40000 ALTER TABLE `mdl_user` DISABLE KEYS */;

INSERT INTO `mdl_user` VALUES (15,'manual',1,0,0,1,'guest','084e0343a0486ff05530df6c705c8bb4','','Guest User',' ','root@localhost',0,'','','','','','','','','','','','','en_utf8','','99',0,1228845165,1228780360,1228793000,'66.249.71.119','',0,'','This user is a special user that allows read-only access to some courses.',1,0,2,1,1,1,0,1193741627,0,NULL,0),(2,'manual',1,0,0,1,'dave','b51c0d8a3cd00a934122287dc755f19f','','Dave','Spittal','dave@spittal.me.uk',0,'','','','','','','','','','','Kirkcaldy','GB','en_utf8','','99',0,1228135202,1227779205,1228135103,'193.63.61.35','',1,'','<font color=\"#0000ff\">I am the controller of this whole thing so don\'t mess with me ... lol</font> <img title=\"big grin\" alt=\"big grin\" src=\"http://OLDDOMAINNAME/OLDFOLDER/pix/s/biggrin.gif\" /> ',1,0,1,1,1,1,0,1212673092,0,'Me',0)

That first entry is mine. There are hundreds to follow. If I do a search and replace on the OLDDOMAINNAME/OLDFOLDER with the new domain name and folder I would then only have the difficulty of somehow restoring the backup to the new database. If I could crack that one I would be a happy chap.

Any advice on that one would be greatly appreciated.

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
If you wait until after the restore, you can use the Moodle script admin/replace.php to do a global search and replace on the database.

Sine the DB backup is just SQL, you should be able to restore it by doing something like

mysql ... some options ... < backup.sql

on the computer you are restoring too. However, I am not very good with MySQL command lines, so you will have to look it up or wait for someone else to respond.
Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Richard Enison -
DS,

Well, it now appears that you are in even better shape than we previously thought.
  1. First of all, there is something I left out of my previous post. When I mentioned the uploaddata directory name, I also meant to mention the fact that Fantastico (no comment) has the nasty habit of locating it within the web document root directory, which is not very secure and in violation of the guidelines in the Moodle docs. Unless you are able to relocate it outside the web root, I would suggest that, assuming your web server is Apache, you create an .htaccess file in uploaddata as recommended in the above-linked doc page, containing

    order deny,allow
deny from all

    If you don't, Moodle will create one containing something a little different, which is problematic as I reported to the Tracker (MDL-16957).
  2. Actually, the second entry is yours; the first is for guest. I'm somewhat surprised you are talking about editing the .sql file to change the domain name and folder name, since you previously said the domain name and pathnames were the same as in the old server since you moved the domain to the new one and both servers are run by the same company. Be that as it may, if you can't do it any other way, I would suggest (after doing the edit of the .sql file):
    1. Fire up phpMyAdmin.
    2. If you can't shut down network access to the MySQL server to everyone but yourself, go ahead and issue the LOCK TABLES query that you show in your post by copying it from the .sql file and pasting it into the query tab in phpMyAdmin, and running it.
    3. It looks like the entire contents of the user table are in that one INSERT query. Each record is listed enclosed in parentheses, with a comma between records; within each record, the field values are separated by commas. See if you can copy the whole query into the query tab and run it. If it is too big to do that, try doing it a record (or maybe a few records) at a time. For example,

      INSERT INTO `mdl_user` VALUES (15,'manual',1,0,0,1,'guest','084e0343a0486ff05530df6c705c8bb4','',
      'Guest User',' ','root@localhost',0,'','','','','','','','','','','','','en_utf8','','99',
      0,1228845165,1228780360,1228793000,'66.249.71.119','',0,'',
      'This user is a special user that allows read-only access to some courses.',1,0,2,1,1,1,0,1193741627,0,NULL,0);

      I have inserted some line breaks to make it more readable as a forum post, but when you run it as a query leave them out, especially the one within quotation marks (between some and courses). Then for the second record,

      INSERT INTO `mdl_user` VALUES (2,'manual',1,0,0,1,'dave','b51c0d8a3cd00a934122287dc755f19f','',
      'Dave','Spittal','dave@spittal.me.uk',0,'','','','','','','','','','','Kirkcaldy',
      'GB','en_utf8','','99',0,1228135202,1227779205,1228135103,
      '193.63.61.35','',1,'','<font color=\"#0000ff\">I am the controller of this whole thing so don\'t mess with me ... lol</font> <img title=\"big grin\" alt=\"big grin\" src=\"http://OLDDOMAINNAME/OLDFOLDER/pix/s/biggrin.gif\" /> ',1,0,1,1,1,1,0,1212673092,0,'Me',0);

      And so on. If you find the highest number of records you can handle at one time, it will greatly cut down on the number of queries you will have to run this way. Of course, this is all assuming that you are unable to import the table the usual way, namely:

      "If you are running mysql, import the sql file back into a newly created database on the mysql server. Be careful here, some backups try to import right back into the same working database that moodle is connected to. This causes database problems that damage a Moodle installation. The best thing to do is make a new database, restore the backed up database into it, and change the Moodle config.php file to connect to this new database (this way you still have the original database).

      once you have created the new database: 

      mysql -p new_database < moodlesqlfile.sql
      ". That's a quote from Site backup#Restoring_a_site_backup, numbered paragraph 3.
    4. There should be an UNLOCK query after the INSERT in the .sql file. If you ran the LOCK query before, run the UNLOCK now.
    5. That should do it!
RLE

EDIT: P.S. In case you're wondering why I went to the trouble of typing all this when TH has already posted a reply, I started this a couple of hours ago and was interrupted about an hour ago because I needed to take my medicine. When I finally got it posted, I found his post from half an hour earlier. That's life. Hopefully you may find something useful in both.

Also, as I mentioned at the end of my numbered paragraph 2, subparagraph 3, I suggested the alternate, more laborious and tedious, method of inserting the data a record at a time on the assumption that you had tried the usual method and it didn't restore the user table. TH seems to have ignored that.

As for the editing of domain and folder names, he is correct about the replace.php script. I just thought doing a replace all command in a text editor would be quicker. You can do it either way.
Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed

by Dave Spittal -
Thank you again for such a comprehensive reply. I have been to bed and now its Monday morning so its off to work. I will try to work through this carefully as soon as possible and report back how I get on. It may be a day or two.
Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Dave Spittal -

Managed to get a couple of hours done tonight and I can restore the users in bundles using the SQL command INSERT INTO `mdl_user` VALUES ( etc.

Having realised that that will take a very long time and many of the users have now left school but still had accounts I am simply going to get the current students to re-enrol. I managed to search for the staff names and restored them individually.

Thanks for the uploaddata info. I never fully understood that warning. Before I go any further with the student accounts I might try a non-fantastico installation.

  • Am I correct in thinking that this will automatically put a moodledata folder outside the root folder?

If that is successful I can use ftp to move the folders from my uploaddata folder into the moodledata folder and restore the courses again from the individual course backups that are there.

Thanks again for all the help and taking the time to find the questions in the original post. Hopefully it will help someone else.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Richard Enison -
DS,
  1. I'm glad you have gotten as far as you have, and I'm even happier that you are considering a non-Fantastico installation!
  2. Not exactly. The Moodle installation script does some things automatically, but not everything. The main selling point for Fantastico is that it is an easy, one-click, automatic installer. The problem is it automatically does wrong things, like putting the data directory in a bad place (the fact that it calls it uploaddata instead of moodledata is not a big deal; you can call it anything you want), refusing to honor Moodle's standard of using Unicode character encoding scheme UTF-8 exclusively for text data in the database starting with version 1.8, and not supporting the latest version of Moodle until it is no longer the latest version.

    What Moodle does do, according to the docs, is

    "The Moodle installer tries hard to create the moodledata directory for you but if it fails then you will have to create a directory for this purpose manually."

    I have looked at the source code and found out more specifically that it tries to create moodledata in the location you specify for it. It does not automatically make sure that it is a secure location.

    If your uploaddata folder and moodledata are on the same machine, instead of using ftp you could just use your host's file manager, if they supply one, or shell access. At this point I don't know whether you (or your employer) have your own server or are using a commercial web host, or whether the server's operating system is Linux or Windows. So it would be difficult to give more specific advice at this time.
RLE
Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed

by Dave Spittal -

I have now spent two nights trying to follow the non-fantastico installation instructions and cannot get moodle to install. I cannot understand the instructions because it always gets to a bit I have to do and gives me no idea HOW to do it. For the non expert it is too hard and would take me too long to identify the bits I cant follow. I have managed to get to the bit where it starts creating all the tables and  then you click one of the continue buttons and bang, a page not found error. I am sorry I cant cope with this and, problems or no, fantastico does it first time every time in less time that its taken me to type this entry. Thanks for all your help but it will have to be fantastico.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Dave Spittal -

OK decided to try again because I believe the advice about fantastico is good. Tried a clean install tonight and am now in a loop. I got to this

Your Moodle files have been changed, and you are about to automatically upgrade your server to this version:

1.9.3+ (Build: 20081217) (2007101532.11)

Once you do this you can not go back again.

Please note that this process can take a long time.

Are you sure you want to upgrade this server to this version?

I click the yes button and get a server check where all the checks are green OKs

I click continue and get a plugins check which again is OK

I click continue and get a page that stays on screen for only about 1 second that says

Database upgraded successfully, No warnings scroll down to continue button

Below this is some data about Build numbers and a line highlighted in pink saying

"there is already a role with this name!"

The page disappears after a second or so and loops back to the first page mentioned in this post. I had to quickly press print screen to grab the contents of the page that disappears.

I have searched the forums for this error message and although several mention the error message I cannot find any which have been answered.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Dave Spittal -

OK decided to try again because I believe the advice about fantastico is good. Tried a clean install tonight and am now in a loop. I got to this

Your Moodle files have been changed, and you are about to automatically upgrade your server to this version:

1.9.3+ (Build: 20081217) (2007101532.11)

Once you do this you can not go back again.

Please note that this process can take a long time.

Are you sure you want to upgrade this server to this version?

I click the yes button and get a server check where all the checks are green OKs

I click continue and get a plugins check which again is OK

I click continue and get a page that stays on screen for only about 1 second that says

Database upgraded successfully, No warnings scroll down to continue button

Below this is some data about Build numbers and a line highlighted in pink saying

"there is already a role with this name!"

The page disappears after a second or so and loops back to the first page mentioned in this post. I had to quickly press print screen to grab the contents of the page that disappears.

I have searched the forums for this error message and although several mention the error message I cannot find any which have been answered.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed

by Dave Spittal -

OK decided to try again because I believe the advice about fantastico is good. Tried a clean install tonight and am now in a loop. I got to this

Your Moodle files have been changed, and you are about to automatically upgrade your server to this version:

1.9.3+ (Build: 20081217) (2007101532.11)

Once you do this you can not go back again.

Please note that this process can take a long time.

Are you sure you want to upgrade this server to this version?

I click the yes button and get a server check where all the checks are green OKs

I click continue and get a plugins check which again is OK

I click continue and get a page that stays on screen for only about 1 second that says

Database upgraded successfully, No warnings scroll down to continue button

Below this is some data about Build numbers and a line highlighted in pink saying

"there is already a role with this name!"

The page disappears after a second or so and loops back to the first page mentioned in this post. I had to quickly press print screen to grab the contents of the page that disappears.

I have searched the forums for this error message and although several mention the error message I cannot find any which have been answered.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed; error: there is already a role with this name

by Richard Enison -
DS,

I'm glad you reconsidered, because you are so close to finishing.

I searched my e-mail inbox folder where I keep posts from this forum (I have over two years' worth) for the error msg. you got. I found a thread (http://moodle.org/mod/forum/discuss.php?d=66574) where three Moodlers reported the exact same problem you have. They were all doing upgrades. You apparently are also; you say you were doing a clean install but you wouldn't have gotten the "Your Moodle files have been changed, and you are about to automatically upgrade your server to this version:" msg. if you had been. You must have started with a non-empty database.

Unfortunately, the thread contained no resolution. In particular, one of the problem reporters (HS) asked which role it was talking about, but nobody answered that question. So I will. So that future Moodlers will be able to find this post more easily (I had to do a search on message text rather than subject to find the thread), I have changed the subject of this post.

I have a PHP script that I downloaded from the Web a few months ago that can search all the Moodle scripts files for a given string of characters. I used it to look for your error msg., and I found it. It is in lib/accesslib, in the function create_role (no surprise there). Well, that doesn't tell us what role it was trying to create. Moodle probably calls that function every time it needs to create any role.

So I looked in the cross-reference listing (xref.moodle.org) and found it is called in a dozen places, but only once in admin/db/upgrade.php. So I looked in there and found what it was doing. It tries to create a legacy role called "authenticated user" after it becomes convinced that it doesn't already exist. But when it calls create_role, it finds that it does exist, hence your error. Here's the whole story:
  1. It only does this if the version it is upgrading from is earlier than 2007021401, which seems to mean the version number is less than 1.8.
  2. It only does it if either
  • the Moodle configuration setting defaultuserroleid is empty, or
  • the setting guestroleid is empty, or
  • the two settings are the same.
Then it looks in the role table (mdl_role assuming you accepted the default table prefix mdl_, or Fantastico did) to see if there is a row in which the value in the shortname column is user. If it doesn't find one, then and only then does it call create_role. Then in create_role, it tries to add a row to mdl_role with
  • "Authenticated user" in the name column
  • user in the shortname column
  • "All logged in users." in the description column
It would also assigns it the capability of moodle/legacy:user in the role_capabilities table, except that it finds there is already a row with "Authenticated user" in the name column. So it issues that error msg. you got.
So it seems to me that the simplest solution is to use your favorite MySQL client program (such as phpMyAdmin) to delete the row in mdl_role in which name is "Authenticated user". You can do this with a query, namely

DELETE FROM mdl_role WHERE name='Authenticated user';

Then try the upgrade again (browse to your Moodle URL/admin).

RLE

Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed; error: there is already a role with this name

by Dave Spittal -

Thanks for bearing with me on this. It wa a new install but I understand what you mean about the upgrade. I have a fantastico version in one folder running. I created a new folder for a new installation and tried to install as per previous posts each time failing to install. Deleted the lot and tried again. The reason its using an old database is because its the only database name I can get the install to find. This is one of the problems i faced during attempted installs. I made a new database using MySQL from control panel, made a new user and gave the user the privilidges to the entire database. The install fails to connect with any database other than one I called mood2.

Ill try your advice and report back. Note I really need a new laptop. That last post appeared three times. I must stop clicking when I think sosmething has died! That is how I got in this mess in the first place.

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed; error: there is already a role with this name

by Richard Enison -
DS,

Rather than go back and forth another dozen posts each way on why you can't get Moodle to connect to your new database, why not use a different table prefix for the new Moodle? Then the two Moodles can share the same database without interfering with each other. At this point your old Moodle is probably unusable, because you have upgraded the database when you installed the new one, unless you can restore it from the backup.

RLE
Average of ratings: -
In reply to Richard Enison

Re: Migration advice needed; error: there is already a role with this name

by Dave Spittal -

I can feel a celebratory glass of champagne coming on! I started again and couldnt get the new database to connect. When reading posts about this problem I noticed the one that says use only alphanumeric characters in the password. Now the MySQL gives the option to generate a "strong" password when creating the database and I always used the generated one. It ALWAYS has non alphanumeric components.

I entered my own password alphanumeric only and bingo the entire process went smoothly from start to finish. It might be worth emphasising this isuue in the installation guide.

Now I have to learn how to modify my theme because I am going to make it a bit different this time. Thanks again for all your help, patience and encouragement. I only hope the pupils appreciate it! I hope this will be my last post to this forum but never say never....

D

Average of ratings: -
In reply to Dave Spittal

Re: Migration advice needed; error: there is already a role with this name

by Richard Enison -
DS,

I'm glad you were able to succeed by using a totally alphanumeric password. I was about to take your advice and edit the docs to say that non-alphanumeric passwords don't work, but I thought I should check to see what if anything the docs already say on the subject, and what do I find? A page that says Moodle 1.9.x lets you set a site policy requiring a minimum number of non-alphanumeric characters in passwords! (Site policies#Password_policy) Go figure. Obviously I'm not qualified to do this edit at this time. But I'll look into it.

RLE
Average of ratings: -