Reducing upgrade downtime: pre-upgrade steps (idea)

Reducing upgrade downtime: pre-upgrade steps (idea)

by sam marshall -
Number of replies: 3
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

(possibly bad idea)

I've been speculating about a way to reduce downtime for Moodle upgrades by doing part of the upgrade before actually taking the system offline. This is a lot more complicated than you might think.

The attached document is my idea for how that might be achieved - including a summary of the benefits and limitations of the approach.

If this gets anywhere near actually happening I will obviously make a Moodle tracker issue and so on. At present, I don't have any time allocated for this and it's not a priority for my employer, but that might change...

--sam


Average of ratings: -
In reply to sam marshall

Re: Reducing upgrade downtime: pre-upgrade steps (idea)

by Michael Hughes -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

This there any grounds for looking at whether or not it's possible to isolate and upgrade only parts of Moodle?

This would be good for "core" shared parts, but the impact of having to do downtime for the whole service to upgrade 1 module (e.g. for a bug fix) is a bit problematic.

Imagine if just a plugin could be put into maintenance mode (for sake of arguments lets say assign is 'optional' smile ):

  1. Admin puts "assign" into maintenance mode
  2. All end-user (including all admin pages) facing elements would be disabled (i.e. a request for anything in mod/assign/* is redirected to a disabled page)
  3. New scheduled tasks would be deferred
  4. Any existing events would be processed
  5. New events the plugin is interested in would be "queued"
  6. assign is now in maintenance mode
  7. Admin can replace the code
  8. Upgrade steps (for plugin) are run (these would have to be non-shared steps).
  9. Plugin is now in post-install mode
  10. Admin elements are re-enabled ( to allow configuration of new settings)
  11. Admin takes plugin out of maintenance mode
  12. Queued events are picked up and processed
  13. End-user elements are made accessible
  14. Plugin is now out of maintenance mode and fully usable again.

M

Average of ratings: -
In reply to Michael Hughes

Re: Reducing upgrade downtime: pre-upgrade steps (idea)

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Nice idea! But, why stick to 'optional' plugins, it could work for core ones like mod_assign as well.

There are problems though - any type of plugin can have various callbacks that can affect any page in Moodle (for example the one that lets you insert HTML code after the header). Depending on the installation some of these things might not really be 'optional'...

It would be great if this worked though - perhaps the plugin could have a marker in version.php for whether it's OK to temporarily disable it during upgrade.

For us, we don't do plugin upgrades aside from our quarterly whole-system upgrades (well we do patches but not that require upgrade), but the approach you've suggested would also benefit the normal upgrade - probably in most upgrades a significant portion of time is spent doing plugin upgrades.

I.e. it could automatically do the plugin upgrades as a second part of upgrade, with the opportunity to turn off maintenance mode at that point. Maybe it would work best like this:

  • switch on full maintenance mode
  • core part of upgrade completes
  • switch off full maintenance mode, switch on plugin maintenance mode for each plugin requiring upgrade
  • as each plugin upgrade completes, switch off maintenance mode for that plugin
An issue with this is that plugin upgrade ordering then becomes important - for example, in our system, we use a particular course format and if that requires an upgrade, there is not too much benefit bringing the system back online if nobody can look at any courses... Similar but even worse with the theme. So you'd probably want to do theme upgrades first, then auth, course formats, blocks, then modules (and you'd probably want to consider subplugins as part of the module rather than do separate maintenance mode for each of those), then other stuff? Just off the top of my head, there are probably other ones.

Still, doing the per-plugin maintenance mode would be a nice step towards this type of improvement for the main upgrade, as well as allowing you to upgrade plugins without downtime, so basically it would be great. smile

--sam
Average of ratings: -
In reply to sam marshall

Re: Reducing upgrade downtime: pre-upgrade steps (idea)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

What you propose is basically the opposite of what has been done a few times in the past, which in your terminology could be called post-upgrade steps.

The two examples I am thinking of are:

  • The question enging changes in Moodle 2.1. Here, there was a long, complex upgrade of all the data about all quiz attempts ever. There was an option to not do all of the upgrade duing the main maintenance mode window. Instead, the attempte could be upgraded a bit at a time on cron aftewards (and there was code so that if you tried on access a quiz attempt were the data was not yet upgraded, you got an error saying to try again later).
  • The new mod_assign, replacing mod_assignment. Here, the new feature was implemented as a new activity, with a conversion tool that could be run at any time. Of course, both had to be maintained for some Moodle versions, before conversion was made a compulsory pre-requisite in the Moodle release where the old assignment was removed.
The advantage of doing it this was is that rather than having to install the upgrade code into the pre-upgrade Moodle, it can instead be developed and tested alongside the new functionality, and all the code gets deployed at the same time, as part of the upgrade.
Average of ratings: -