Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Leo Thiessen -
Number of replies: 20

I recently tried (unsuccessfully) upgrading to the latest current version of Moodle 2.2.3.  We created a test copy of the live site, cleaned it up as much as possible, then attempted the upgrade.  The first attempt failed right away with an error about the theme directory.  This was a simple fix - I replaced the theme/ directory.  Then the upgrade seem to run, but failed after a long run time (numerous errors, not so easy to understand, never mind actually fixing them).  So, then we tried setting up a clean 2.2+ install and begain importing the 1.9.x+ backups.  This works for simple courses, but we need the user data - there is huge value in the data even to new students.

Note: we are working on a non-semester (or similar) based commercial educational site - there is no downtime at anypoint through the year as far as the paying customers are concerned - they study on their own time and primarily at their own pace.  Our Moodle install has approx 10000 users (not all active) and our single database is about 6.5GB at the present time.

So, looking around and seeing what everyone else is experiencing, it seems pretty common for 1.9.x sites to fail during upgrading to the latest 2.x version - even large universities and companies that can throw lots of man-power and time at the challenge are not really that successful yet.

Here's what we've gathered needs to be done, and I'm wondering if you all might be able to provide some feedback on this approach:

To upgrade a Moodle 1.9.x install to the latest 2.2.x install:

  1. Clean up the current 1.9 install
     - by removing any unused activities/modules/plugins and unecessary data (eg grade history, etc.) there is less data to transfer and should be less to go wrong during the subsequent upgrade(s)/steps
  2. Migrate the current 1.9 install to a new, clean 1.9.x+ install
     - *migrate*, (not upgrade)... the thinking here is to get rid of all the cruft/garbage and what not from previous upgrades that didn't upgrade/delete/etc. everything properly
  3. Upgrade the newly migrated 1.9 install to the latest Moodle 2.2.x+ version
     - this hopefully will work better once step 2 above has been completed...
  4. Migrate the new 2.2.x+ version to a new, clean 2.2.x+ install
     - ... again, this is to fix-up problems from an imperfect upgrade in the previous step

... and there you have it - all it takes is 4 steps!  Honestly!

wink

Well, have any of you experienced this in it's entirety with a decently large Moodle installation?  Were you successfull?  How did it go?  Any tips and/or advice would be welcome.

Kind regards,
 - Leo

Average of ratings: Useful (1)
In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by adam penner -

I find this issue very frustrating. Support for 1.9.X is ending this summer and there is no simple way to upgrade. The current process is brutal and expensive and time consuming and from my experience little chance of actually working.

I see others on the forums who are having this issue as well. What's the solution?

 

Adam

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

Gonna stick my lowly neck out here and present yet another thought ... site migration, as you have experienced, is full of pitfalls, so why site migrate?

Why not run parallel sites?  Courses have to end.  When a course ends in the 1.9, it's new version is restored to the 2 (archive kept but the 1.9 no longer for new students, but available for former students [for a while]).  New students in the 2. 

Only doing a course restores then - which will still require some tweaking - but it does an auto-reset because users will not carry over AND one can rid the new course of 'legacy'.  Things to do: assign teacher, setup a repo (if needed), and some 'minor tweaking' (setup enrollments, etc.).  There is little or almost no 'down time'.

Yeah, it's work, and, for a while, lots of Moodle Server administration until all the 1.9 courses are phased out and moved to the 2.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Leo Thiessen -

Hey Ken,

Thanks for the response!  I appreciate you taking the time.

Well, if we absolutely have to, really, really have to, then we may just do that in the end.  For now, we've put off moving to Moodle 2.2 at all.

In our case, yes, "courses have to end" - but only on a per-individual basis.  We have new enrolments essentially daily with course "completions" daily as well... so for a course with, say, 300 enrollments, basically each person will have a unique start date, end date, finishing date and even duration.  (I'm not working with a college/university/etc. - this is continuing education, in a sense.)  So starting up a clean slate Moodle 2.2, a course would start out with no peers, then slowly accumulate peers... the first person would be the only and most-senior peer in the course (thus the first persons would not benefit from students who are further along), the forums would be empty and virtually inactive (useless, mostly with the burden of the majority responses on the instructor).  Fast forward to the end of the year and the last man standing in the 1.9 would be all alone, again with only the instructor around.  There is a lot lost by this approach, and maybe not that much gained/saved, as far as I can tell (we'd save *some* of the up-front cost/time of doing an upgrade/migration -- both seem necessary if one wants to keep user data and transfer existing users -- but really the time and $ is just more spread out by administering 2 Moodle sites, supporting students in both, maintaining content in both, etc.).  Unless I am wrong about something here?  Please do inform me if I'm missing or not understanding something correctly.

We would really prefer to be able to "click upgrade" and go (with full user switch-over...) is this possible somehow?  Anyone?

Is there anyone out there with a successfull Moodle 2 upgrade &/or migration who can share their experience/process/tools etc.?

Average of ratings: Useful (1)
In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Leo and Adam

I know, it is a cold comfort, but you are not alone in this rocking boat!

I see you've received excellent advice. Just two thoughts to clarify your motivation:

- Is the sole reason for moving to Moodle 2 is the end of life of 1.9? Did you see "Bug fixes for serious security issues in 1.9.x by Catalyst IT will continue until Dec 2013" in http://docs.moodle.org/dev/Latest_release_notes#Moodle_1.9 ?

- Have you evaluated Moodle 2 in a test installation and found it to be the right thing for your users?
In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

You, sir, are too kind!  At least the suggestion has rendered a deeper description of what is sought and why.

But ... am gonna mention one more thought ... and then I'll quit! smile

Moodle Networking: http://docs.moodle.org/22/en/MNet

A 1.9 can be networked to a 2.2.x and reverse is true too!

'spirit of sharing', Ken

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Paul Langdon -

Hi Leo

exactly the same problem here regarding the validity of a switch over. I'm HEAVILY reliant on 1.9. (I have 300 students who on their own store their assessed work via it and I also use a commercial plugin that I use to mark their work on line!) My older students are all confident and daily users of 1.9 so you can imagine my quandry.

To cut a long story short I have decided to retain the 1.9 for my legacy students who have 1 year remaining on their existing course and start a brand new build for the latest version Moodle being released in 2 weeks because of the drag and drop features on it which should seriously speed up the whole process.

I am also looking upon this as an opportunity to clean up the whole on line look and feel of the school's system as the old Moodle was getting a bit bloated and messy to be honest.

Its a drag I know but when I started with 1.9 I was an absolute noob so will avoid some of my earlier design mistakes.

Good luck

 

Paul

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
1.5 Check the database schema (it's on the upgrade page in the docs)

Schema problems in the databases of 1.9 sites that have any kind of age to them are incredibly common. I've seen hundreds of individual problems on some sites. They all have the potential to break the upgrade. Your 1.9 installation needs to be sparkly clean for you to stand a chance.

Your characters sets and collations (in MySQL anyway) all need to be spot-on consistent and correct. Again, if you have done previous migrations or MySQL upgrades/changes they are often not. Should get picked up by a schema check.

Delete what you can. Get rid of unused courses, uses, all those old backup files etc etc. before trying to upgrade. It all lightens the load.

If it does fail, make sure you restore the original site properly. It's easy in the heat of frustration to miss a step. If you are testing the upgrade with a copy make sure it works - again, easy to assume.

Don't skimp on checking your server really is compatible with Moodle 2. "It'll do" might not do.

If you want my $00.02 - don't bother. For sites of a significant size you are probably going to have a hard time getting upgrade to work. I would see Moodle 2 as a different thing and start with a fresh site. Perhaps run in parallel for a while.

Unashamed advert - if it really matters, consider getting a Moodle Partner in. A lot of us have a lot of experience of struggling through these problems. We have the advantage of having done it more than once and we've worked out a lot of the individual issues.

Catalyst IT have indicated that they will keep up basic support for some time. There are still lots of 1.9 sites and I can promise you that many will be around for years to come.
Average of ratings: Useful (2)
In reply to Howard Miller

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by HJWUCGA INC. -

After a lot of trial and error and copious amounts of reading and making notes to myself, I was able to successful migrate my 1.9.x instances to now 2.2.2+.

All the above are correct and there is/are more to it... delete dupe records, schema from a clean version, etc...

I've now successfully migrated about 11 of 1.9.x instances over to 2.2.x. for my clients.

In reply to HJWUCGA INC.

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

Care to share your notes? ;)

'spirit of sharing', Ken

In reply to Ken Task

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by HJWUCGA INC. -

so many notes ... I'll try and compile them all...

here are some resources so far

http://www.solutiongrove.com/blogger/2011/10/03/notes-on-upgrading-to-moodle-2-1-x

 

 

Average of ratings: Useful (1)
In reply to HJWUCGA INC.

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for sharing!  And ... here's my 'contribution to the confusion'! ;)

http://moodle.tcea.org/migrating-a19copy-tovr2mdl.txt

It's a text file to facilitate copy and paste to a terminal session, if anyone ask why a text file. ;)

'spirit of sharing', Ken

Average of ratings: Useful (3)
In reply to Ken Task

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by HJWUCGA INC. -

Ken,

Interestingly enough.. your notes is very similar to mine ..about 95% of it..

I urge people to look at it closely thoughtful

In reply to HJWUCGA INC.

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

So 95% is less than 100% ... which means there is more than one route to migration ... or it means something is left to be done/discovered after the 2's have been running a while! :\

So if there's a difference in notes, maybe we should also share those differences! ;) [or what's missing]

'spirit of sharing', Ken

In reply to Howard Miller

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Leo Thiessen -

Thanks for that info - very good.  This is actually the purpose I had in mind by doing the 1.9->1.9 migration (step 2 in topic description) -- I'm thinking this would clear up a lot of the db issues, but I haven't actually tested this yet.

We are holding off for now.

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Garret Gengler -

To add my experience...

I manage the Moodle installation for a grad college at the University of Illinois.   We have been using moodle since 2006 (v1.5)... we started using it to support our distance learning program and later expanded it to on-campus classes (as well as research projects and community/intranet functions) as well.

We have something like 3000 active accounts, and 90 active courses in a typical semester.  Our moodle mysql db is around 6GB, and moodle data is around 40GB.   It's not university scale, but it is a significant site.

We migrated from 1.9 to 2.2.3+ a few days ago... (on Monday - Memorial Day holiday in the USA).

It took at least 6 months to plan the upgrade.   I probably did a dozen test migrations during the testing period.

The upgrade bombed out at least a dozen times for various reasons.  Each time I would have to invest a few hours/days of research, trace code, look for out-of-spec data in our database, and come up with some workaround.  Then I'd have to add the workaround to my upgrade plan.   By the end, it was like a rube goldberg machine.

What makes this process extra painful is the data directory migration.   It's one thing to have to reload a 5GB database dump so you can try the upgrade again.    It's another thing to have to restore a 50GB tree of 500,000 files.   The file system migration is genuinely worrisome... on some level I can't believe the moodle devs decided to do it.   I get the benefits for security... but dang that's a crazy thing to do to _every existing moodle site_.   I hope you all are right that the benefits outweigh the change costs.

We actually were hit with one final show stopper on migration day.   I did not realize that Moodle 2 is incompatible with the standard Mysql binlog format.... our testing server had the mysql binary log disabled, so we didn't notice this.    On the production server, Moodle 1.9 ran fine with the standard binlog, but the upgrader died when it executed a statement that Mysql could not store in a standard binary log.   (we had to switch the binlog format to MIXED -  somewhere there is a ticket about this in the tracker.)

You can imagine we cursed like sailors when that issue showed up.

The final upgrade plan was about 60 steps... 20 or so actions to take on the linux command line or mysql console, and 40 things to adjust in the Moodle admistrative screens once the site was running.   Start to finish, it was supposed to take 3 hours, but ended up taking 7 hours because of the binary log configuration issue.

So I guess the moral is, it can be done... but be prepared to do a lot of work to clean up your data, and plan to spend a month or two trying to get test copies of your site to upgrade.

My notes and upgrade plan are in no shape to share... but when I have time I will add them here.

-Garret

Average of ratings: Useful (3)
In reply to Garret Gengler

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by HJWUCGA INC. -

I'm doing another test upgrade again =)

did not know that 2.2.3 came out ... will now try and use that.

We believe there should a mirror of production and test platform. OS, patch level, moodle release, database versions, etc. that way when we upgrade and test, there are no suprises.

In fact when I test, I move all the data from one server to another. Not just a small sample courses.

had to clean up dupe instances

increase the column text types to big in the xmldb install files

I cleaned out the db schema comparing it against a clean and same version of 1.9.x before doing the upgrade

run the moodle checker for bigints, indexes in 1.9.x

I use the CLI install

make sure the file resources' name is not very long or else moodle will skip it.. we have 3 filenames that were sooooo long that it ignored it because it was too long... so it was never converted

make lots of backup of the backup and only use a backup copy clone as production use.

don't get rid of 1.9.14 until your users are satisfied with 2.x and sign it off.

... more to come

 

 

Average of ratings: Useful (1)
In reply to Garret Gengler

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks, Garret.  And congrats!  Can appreciate the accomplishment and am looking forward to whatever you share.

'spirit of sharing', Ken

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by Leo Thiessen -

So... from all this feedback I'm thinking that in general older/more mature moodle installations (starting pre 1.9) are VERY difficult to upgrade, even people who started with 1.9 and wish to migrate and/or upgrade to moodle 2.x must really be very careful to do it right.  Most people recommend starting fresh, if that is an option.

I'm hoping the upgrade/migration process/robustness will improve yet.  Does anybody have any idea if this will happen - are improvements like this in the works anywhere?

Kind regards,
 - Leo

In reply to Leo Thiessen

Re: Upgrade from Moodle 1.9.18+ to Moodle 2.2.3+ -- Process Check

by HJWUCGA INC. -

Actually, it's not that bad once you upgrade to at least 1.9.1x.. do it incrementally and you can upgrade directly now, including student data starting from 2.2.x

get the 1.9.x instances to the highest levels of 1.9.x instance and then go from there.

In fact do a compare of db schema from a fresh empty install of 1.9.x against the one you are planning to update to...

one tool I used is SQLyog from Webyog ... good for comparing schema's; getting the script generated (so you can use it over and over again for testing); and doing the synchronization when you need to run it within the program.

http://www.webyog.com/en/

http://static.webyog.com/images/screenshots/schema_sync.png?version=6

I have Toad (for Oracle and MySQL) but it's not the same