Confirm database / schema upgrade path from 1.9.14 to 2.1

Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -
Number of replies: 22

Hi there,

I might have figured out why my upgrade from 1.9.14 is not working and wanted to confirm that this is the cause and solution to my current dilemma.

For a successful upgrade, I have to:

1. dump the schema of my existing production 1.9.14 database (A).

2. install a clean 2.0.7 moodle database and dump the 2.0.7 schema (B)

3. I find the difference between A and B schema and make all he changes IN A. This means that my 1.9.14+ database will have some extra fields, modifications to existing columns, and tables which would MATCH THE FRESH 2.0.7+ schema. 

4. After I make the changes in A, I can then run the 2.0.7 upgrade USING THE production database which now has edited schema.

5. I then do the same from 2.0.7 to 2.1.x.

Am I correct?

Thank you.

 

Average of ratings: -
In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

No.

What you need to do is described on Upgrading.

  1. Copy the new code into place.
  2. Visit the admin page.
  3. Moodle will automatically make all the necessary schema updates for you.

(Well, of course, in reality, you need to make a backup first, text the upgrade on a copy of your site before doing it for real, etc.)

In reply to Tim Hunt

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

never worked for me so I'm trying another approach...

always stops at the same point in upgrade

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

OK, so what have you done to investigate the problem?

You have Debugging turn on, I assume, so you get a detailed error message. What is it?

In reply to Tim Hunt

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

I wonder if there is a debugger or verbose setting so we can follow it onscreen on exactly what it's doing... and not just stop at the error point. I want to see what it's trying to do before it broke.

Anyway, here are my steps. I'll stuck at Error 1. sad

 



Here are the general steps of our moodle upgrade

1.9.14+ (Production aka "PROD") to 2.0.7+ (Test 2.0.7+ server aka "T1 Test Server")

1.    Download the latest 2.0.7+ moodle release on T1 in a directory other than the directory where moodle would be delivered (i.e save it in your home directory). This code base is basic in that it does not have any custom plugins.
2.    Unzip/extract the 2.0.7+ moodle code base. This will extract and creat a "moodle directory" and we will use it later when we do the upgrade.
3.    Backup production database on PROD
4.    Backup production moodle data on PROD
5.    Copy all of PROD moodle source code to the T1 test server except "config.php". This will also copy over all the custom 1.9.14+ blocks and plugins.
6.    Copy over the backup production database over to T1 test server.
7.    Copy over the contents in the PROD moodle data directory over to the moodle data directory on T1 Test Server.
8.    Restore the backup production database on T1 Test Server.
9.    On T1 Test Server, run the 1.9.14+ install procedure again (no problems). This will create a “config.php” file for this 1.9.14 instance on T1 Test Server. This is the config.php file we will use to upgrade to 2.0.7 on KELTW11. Check indexes and bigints columns under the experimental administrative menu (no problems).
10.    Save a copy of the “config.php” file created in step 9 above to your home (or another) directory.
11.    Delete all moodle 1.9.14+ code base on T1 Test Server (ie. now an empty directory where the 1.9.14+ use to be).
12.    Next, copy over all the moodle 2.0.7 code base files you extracted in step 2 over to the “moodle directory” where the site is configured in T1 Test Server.
13.    Next, copy over the “config.php” file you save in step 10 (ie. the 1.9.14+ config.php") over to the 2.0.7+ moodle root directory. The upgrade process looks at this config.php file to see which database it needs to connect to.
14.    Start the upgrade via command line (sudo -u apache /usr/bin/php /var/www/html/mymoodle2test/admin/cli/upgrade.php) option.
15.    Once the upgrade starts, the process will be monitored to ensure there are no immediate errors. Errors will occur and it will be displayed in this area with and must be fixed before going any further. As it’s possible to receive a wide variation of errors, the error will have to be handled on a per-issue base.
16.    If we are lucky enough to have no errors, then we can test basic moodle functions.
17.    If the tests in step 15 are fine, then we can install and test the custom plugins for version 2.0.7+
18.    Once all the above steps are executed without errors, we can proceed to upgrade to 2.1.4+

I have both turned on debugging at application level (Developer, show all) and in my config.php

$CFG->debug = 38911;
$CFG->debugdisplay = 1;

I cannot get past number 14... tables and columns were never created ... this is the first of 2 errors:

--------------
ERROR 1:
-------------

++ Success ++
-->mod_data
++ Success ++
-->mod_feedback
++ Success ++
-->mod_folder
Default exception handler: Error writing to database Debug: Duplicate entry '9900' for key 'mdl_resoold_old_uix'
INSERT INTO mdl_resource_old (oldid, course, name, type, reference, intro, introformat, alltext, popup, options, timemodified, cmid)
            SELECT r.id, r.course, r.name, r.type, r.reference, r.summary, 0, r.alltext, r.popup, r.options, r.timemodified, cm.id
              FROM mdl_resource r
         LEFT JOIN mdl_course_modules cm ON (r.id = cm.instance AND cm.module = ?)
[array (
  0 => '13',
)]
* line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 722 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 258 of /mod/resource/db/upgradelib.php: call to mysqli_native_moodle_database->execute()
* line 46 of /mod/folder/db/upgradelib.php: call to resource_20_prepare_migration()
* line 39 of /mod/folder/db/install.php: call to folder_20_migrate()
* line 514 of /lib/upgradelib.php: call to xmldb_folder_install()
* line 271 of /lib/upgradelib.php: call to upgrade_plugins_modules()
* line 1436 of /lib/upgradelib.php: call to upgrade_plugins()
* line 146 of /admin/cli/upgrade.php: call to upgrade_noncore()

!!! Error writing to database !!!
!! Duplicate entry '9900' for key 'mdl_resoold_old_uix'
INSERT INTO mdl_resource_old (oldid, course, name, type, reference, intro, introformat, alltext, popup, options, timemodified, cmid)
            SELECT r.id, r.course, r.name, r.type, r.reference, r.summary, 0, r.alltext, r.popup, r.options, r.timemodified, cm.id
              FROM mdl_resource r
         LEFT JOIN mdl_course_modules cm ON (r.id = cm.instance AND cm.module = ?)
[array (
  0 => '13',
)] !!
!! Stack trace: * line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown
* line 722 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
* line 258 of /mod/resource/db/upgradelib.php: call to mysqli_native_moodle_database->execute()
* line 46 of /mod/folder/db/upgradelib.php: call to resource_20_prepare_migration()
* line 39 of /mod/folder/db/install.php: call to folder_20_migrate()
* line 514 of /lib/upgradelib.php: call to xmldb_folder_install()
* line 271 of /lib/upgradelib.php: call to upgrade_plugins_modules()
* line 1436 of /lib/upgradelib.php: call to upgrade_plugins()
* line 146 of /admin/cli/upgrade.php: call to upgrade_noncore()
 !!
 
 I went to check the index ...
 
 mysql> show index from mdl_resource_old;
+------------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table            | Non_unique | Key_name            | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| mdl_resource_old |          0 | PRIMARY             |            1 | id          | A         |           0 |     NULL | NULL   |      | BTREE      |         |
| mdl_resource_old |          0 | mdl_resoold_old_uix |            1 | oldid       | A         |           0 |     NULL | NULL   |      | BTREE      |         |
| mdl_resource_old |          1 | mdl_resoold_cmi_ix  |            1 | cmid        | A         |           0 |     NULL | NULL   | YES  | BTREE      |         |
+------------------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

to check I wrote ....

mysql> select * from mdl_course_modules where instance = '9900';
+--------+--------+--------+----------+---------+----------+------------+-------+--------+---------+------------+-----------+------------+------------------+------------+---------------------------+----------------+--------------------+---------------+----------------+------------------+
| id     | course | module | instance | section | idnumber | added      | score | indent | visible | visibleold | groupmode | groupingid | groupmembersonly | completion | completiongradeitemnumber | completionview | completionexpected | availablefrom | availableuntil | showavailability |
+--------+--------+--------+----------+---------+----------+------------+-------+--------+---------+------------+-----------+------------+------------------+------------+---------------------------+----------------+--------------------+---------------+----------------+------------------+
|  21283 |    474 |     13 |     9900 |    6405 | NULL     | 1204764449 |     0 |      0 |       1 |          1 |         0 |          0 |                0 |          0 |                      NULL |              0 |                  0 |             0 |              0 |                0 |
|  21300 |    474 |     13 |     9900 |    6411 | NULL     | 1204764449 |     0 |      0 |       1 |          1 |         0 |          0 |                0 |          0 |                      NULL |              0 |                  0 |             0 |              0 |                0 |
| 119243 |   1511 |     12 |     9900 |   23098 | NULL     | 1216148337 |     0 |      2 |       0 |          0 |         0 |          0 |                0 |          0 |                      NULL |              0 |                  0 |             0 |              0 |                0 |
| 209763 |   2166 |      1 |     9900 |   33982 |          | 1323845265 |     0 |      0 |       1 |          1 |         0 |          0 |                0 |          0 |                      NULL |              0 |                  0 |             0 |              0 |                0 |
+--------+--------+--------+----------+---------+----------+------------+-------+--------+---------+------------+-----------+------------+------------------+------------+---------------------------+----------------+--------------------+---------------+----------------+------------------+
4 rows in set (0.01 sec)


 
 --------------
 ERROR 2
 --------------

http://tracker.moodle.org/browse/MDL-25248


I'm stuck at error 1. How do i find the so called duplicate?

thank you.


 

 

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

Actually, it is a good idea to verify your schema, but you do that by doing a clean install of the same version you are running. Then dump the clean schema and yours, compare them and make any necessary corrections. Once the 1.9.x schema is verified and corrected, you should be able to upgrade to 2.0.x. If you can, I'd recommend running the CLI version of the upgrade rather than relying on the web version.

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

Brett,

When you say verify my schema, do you mean a clean install of 1.9 and my production 1.9?

My production 1.9 has a lot of other blocks and activities. I can safely leave these out right?

I already am using the CLI to upgrade ... not web version

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

I would install all the add-ons in your clean 1.9 install so everything but the data is there. Apparently, for sites that have been running through many version upgrades, the schema tends to deviate from where it should be. I found minor inconsistencies in mine, only having run Moodle for a few years. But the documentation says that even minor inconsitencies can cause issues in the 1.9.x to 2.0.x upgrade. So I corrected it all before the upgrade. I don't have a lot of add-ons, so maybe that's why mine went fairly smoothly. 

The idea is that a new install with all the same options should have the right schema, so matching it against your production version should give you a clear picture of the inconsistencies which you then must manually correct. Making the schema fixes shouldn't affect 1.9.x since the new install should be more correct than a production system that has been through many upgrades. Just make sure the versions of Moodle and all the add-ons are the same when you compare the schema.

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

Brett,

So what you are suggesting is to:

  1. first take the same 1.9.4+ release I have PLUS
  2. take the customize plugins (blocks, activities) that I have in production AND
  3. then start with a fresh install of the ones above WITH
  4. the difference being that I just have an empty database with no courses or add a couple of test courses AND THEN
  5. dump the schema of the fresh 1.9.14+ install AND
  6. compare the moodle table schemas with the fresh install with the ones I have in production AND THEN
  7. make the changes in production AND THEN
  8. try the 2.0.7 upgrade again via command line by replacing all the moodle core (leaving out the custom plugins initially) with version 2.0.7

Am I correct?

Thanks

Average of ratings: Useful (1)
In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

Yes, that's exactly what the upgrade docs suggest.

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

What I don't understand if I read the docs correctly is that they suggested that to do a fresh install of 1.9.x, we should leave out the custom blocks no?

I guess it does no harm as we're only comparing moodle db core schema for their own tables.

My production is innodb so I guess for the fresh install, I would have to convert it to innodb as well?

If I finally make the changes on the schema for moodle related tables (not plugins) and do the upgrade, won't it still not work as the schema for the custom plugins are not part of the moodle core tables?

thanks

 

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

You certainly can start with the core schema. That might be enough. I guess you'd just have to skip tables that didn't exist in the core schema.

My thinking is that add-on modules go through an upgrade process too. And over time, if not every upgrade has upgraded the schema perfectly, add-ons may also have schema inconsistencies. And those can also cause problems when you upgrade your add-ons to be compatibility with 2.x. I can't say from experience whether it's necessary. We don't use many add-ons and eliminated most in the upgrade from 1.9 to 2.1 because the features we needed are supported adequately in the core now.

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

Bret,

Okay, i just created a fresh install of my 1.9.14 (same release) along with the plugins I have in production with an empty database.

I created a mysqldump as a backup

I take it that after this is done, that I just replace what was on the fresh install's schema in the mysqldump of my latest production copy in the "create table" section of my production .sql file. Is this correct?

After the above is done, I use mysqldump to put it back into my production 1.9.14+ database.

I think use this "cleaned up version" of 1.9.14+ db, for 2..0.7 upgrade keeping in mind to use the same config.php

Does this sound right?

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

so I installed mysql workbench but I cannot seem to synchronize between the 2 databases... looks like a bug.

http://bugs.mysql.com/bug.php?id=61211

http://stackoverflow.com/questions/2646041/synchronize-model-in-mysql-workbench

trying to use the source db called "moodleclean" and try to target it to the destination db called "moodleprod".

I was trying to do a diff synch when the unexpected behaviour came up as shown in the 2 links above. Tried the bug fix but still did not work.

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

My experience with MySQL is limited to website creation and maintenance, and I'll have to admin I had no idea it had a way to "diff sync" schema between two databases.

I dumped the schema from each, ran diff to compare them and manually walked through the differences making changes manually in phpMyAdmin. Most of my differences were simply columns that were out of order. Those I did nothing with. There were several signed/unsigned changes, and a few others. Of course, if you have been running Moodle longer, or you upgrade more often, you may have more.

Average of ratings: Useful (1)
In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

I might have found a better one ...SQLyog .. 30 day trial for now

I'm making a copy of the production db first and then use this copy as a target for schema changes from the clean version (source) and apply the changes to the copy..

I'll return to post my results.

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by John Reese -

SQLLog is much, much better and more intuitive ..big grin

I'm now running the upgrade again (via CLI) with the schema changes and I'll see what happens...

oh the suspense!

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Henry Foster -

Hello Bret. I'm working on upgrading two production sites (that started as 1.9) from 2.0 to 2.2. I've successfully upgraded the smaller one to 2.1, but the upgrade notes for 2.2 recommend checking the schema and correcting remnant differences. Both of my sites are on a rental server, so I don't have command-line access, but I can use phpMyAdmin through my Cpanel. 

I have zero experience in working with databases. If you have a moment, could you either point me to an explanation or briefly describe for me yourself how one uses phpMyAdmin to export and/or edit schema? Also, without command line access, do you know of any way I can check the differences in the schema?

Thanks

In reply to Henry Foster

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

Henry,

The chances are very good that if your sites STARTED at 1.9, there is very little chance that there is enough schema variation to cause any problems.

But for those who might be curious, phpMyAdmin has an Export tab where you can export your database. You will want to uncheck the entire data section because you don't want to compare data--just the structure. Once you export and download the production schema along with the schema from a clean install of the same version of Moodle, you can use any number of file comparison tools to compare them. I used diff because I have command-line access (my site runs on DreamHost where SSH access is something I can give myself). But on Windows I use ExamDiff and have heard some good things about WinMerge. Both are free tools. And if you're a linux lover, diff is available in cygwin so you can use it on Windows too.

Your level of database experience comes in when there are differences. That is, I can't explain how to tell when a difference is significant. Generally, the order of columns is not. Signed vs unsigned can be in some cases. Certainly character length discrepancies can be. Essentially you have to look at the differences and where something seems significant, use phpMyAdmin to change that column definition to what the clean install uses.

Since you are matching the schema to what should work, it's unlikely you'll create problems, but as with any structural or code change, you should try this on a test site first to verify that it isn't going to break anything. Only once it's verified to work, do it to your production site.

In reply to Bret Miller

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Henry Foster -

Bret,

Thank you, that helps a lot. Just two more questions:

When exporting from phpMyAdmin, do you need to select all of the tables in the database?

In order to make changes (if any are needed), do you edit the downloaded file and then use Import, or do you go into the Structure tab in phpMyAdmin and use the Actions there, or do you click on the specific table from the list on the left and then use the [edit] function there...? (I have no idea about how to make actual edits to my database...) 

Thanks again. There are obviously a lot of us benefiting from your advice here - kudos and gratitude to you!

Henry

In reply to Henry Foster

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Bret Miller -
Picture of Particularly helpful Moodlers

The export tab in phpMyAdmin only lists databases, so you need only select the database.

To make changes you'd either need to script them yourself using "alter table" statements, or manually edit them in phpMyAdmin. I find it easier to manually edit. It's probably safer when you're doing it on a test system first, then on a production to script it. But I'm not a database wizard, so it's just way easier to adjust the column properties on the screen.

I, too, depend heavily on community support, so I try to give it when I can.

In reply to John Reese

Re: Confirm database / schema upgrade path from 1.9.14 to 2.1

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
What nobody asked is how old your Moodle site is? It's really got to have been through quite a few upgrade cycles for schema issues to be a problem.

Fixing a wrecked schema, really needs some reasonable database skills - there isn't a simple procedure to follow. There are tools to produce the necessary commands and while they are useful you still need to apply a bit of intuition.

If it's got this bad you might want to consider a clean 2.x install and moving stuff over using backup and restore. Think of it as an opportunity to leave old junk behind