General developer forum

Does anyone have upgrade for massive sites tips and tricks?

 
 
Picture of A Guy
Does anyone have upgrade for massive sites tips and tricks?
 

I am trying to upgrade a 1.9 to 2.2.1. And while I can get so far I get stuck on the following upgrade sql statement:

update mdl_question_sessions set manualcommentformat = $1[array(0=>'1'),]

I know it is still running even though I see no activity at the command prompt after getting here as I looked in the pg_stat_activity table and see the active/open database session.

The mdl_question_sessions table for my site has over 12 million rows in.The upgrade has been cranking away on this part for over 4 hours now.

I am using the command line. I have increased the values for php's max execution time and max upload etc.I have removed the server/load balancer, etc from the equation, for the upgrade, by doing this all locally on my Windows machine. 

My site uses Apache and Postgresql. While it runs on Linux for production, I am doing this on my laptop with Windows 7.

Any help or suggestions would be apprecaited.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Does anyone have upgrade for massive sites tips and tricks?
 
Average of ratings:Useful (1)
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

I have not even seen that page. It has not come up in a search. But after glancing at it for a bit it definitely speaks to the issue I have been having. Thanks so much for your reply.

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: Does anyone have upgrade for massive sites tips and tricks?
Group Particularly helpful Moodlers
Looks like a topic for the "Hardware and performance" forum http://moodle.org/mod/forum/view.php?id=596. You should use the "'Using Moodle' advanced search facility" linked to the introduction to that forum to find similar discussions, for example http://moodle.org/mod/forum/discuss.php?d=192089.

Note to the document editors: Only Moodle 2.1 wiki has this information http://docs.moodle.org/21/en/Upgrading_to_Moodle_2.1#Planning_the_question_engine_upgrade. There is no equivalent in http://docs.moodle.org/22/en/Upgrading_to_Moodle_2.2.

You wrote:
> My site uses Apache and Postgresql. While it runs on Linux for production, I am doing this on my laptop with Windows 7.

Aren't that somewhat optimistic?
 
Average of ratings: -
C'est moi :-)
Re: Does anyone have upgrade for massive sites tips and tricks?
Group Documentation writersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Yes, but "Upgrading to 2.2" includes "please also read Upgrading to Moodle 2.1" smile

Perhaps should be something like "please also read, and take care of informations in Upgrading to Moodle 2.1" ?

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: Does anyone have upgrade for massive sites tips and tricks?
Group Particularly helpful Moodlers
Hi Séverin

You wrote
> Yes, but "Upgrading to 2.2" includes "please also read Upgrading to Moodle 2.1" smile

The series of Upgrading articles are recursive you mean, /2N/../Upgrading_to_Moodle_2.N stands for /2(N-1)/../Upgrading_to_Moodle_2.(N-1) which in turn stands for .... until /20/../Upgrading_to_Moodle_2.0 ?
smile

Nothing to worry, I am not going to reopen that topic. (In case anybody getting curious visit "Continuing on Upgrade docs" http://moodle.org/mod/forum/discuss.php?d=197602 )

@Mod, I suggest moving this discussion to "Hardware and performance", OPs problem is performance related.
 
Average of ratings: -
C'est moi :-)
Re: Does anyone have upgrade for massive sites tips and tricks?
Group Documentation writersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Hi Visvanath,

Yes, you must read, and take care of all upgrading instructions between the current version you're using, and the version you're upgrading to.

You can do a direct upgrade, from (say) Moodle 1.9 to Moodle 2.2, but all intermediate information has to be read, as implied changes applicates...

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

In my opinion that is just ridiculous that this software would be designed that way. That the upgrade to Moodle 2.2.x instructions would be so incomplete. It is as if Moodle is in pieces and not complete. In one place it says that if you upgrade incremently from one version to the next then you would have problems. Now at least 3 of you are saying that you have to basically do that, or at a minimum, read all the various pieces of upgrade documentation scattered here and there. This is just absurd. And if this continues going forward the software will suffer. It smacks of a lack of planning and sophistication of design, to me.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

Also, I just reviewed each individual version upgrade. With the exception of the issue brought out in this post in regards to the quizzes there isn't really anything else to do for each individual version. Yes, from 1.9 to 2.0 it talks about the custom plug in. And some possible DDL issue. But then after that nothing else is as specific as this issue. So I don't really know what else you are suggesting that I "take care of". Anyway, I will just try and figure it out on my own. It is strange that Moodle doesn't give you an error message or warning or notice of any kind . . . that it just exits. There is no message on the command line ifyou are using CLI, on the browser, if you are upgrading via the browser, in the Apache logs or in the database logs. It just cranks on taht one query for hours and then gracefully exits. But you are never upgraded. Just stupid.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

Since the code bears your name, Tim. Perhaps you can help me with the installation. I dropped it in the mod directory in a folder called "qeupgradehelper", made a copy of "partialupgrade-example.php" naming the new copy "partialupgrade.php", modified "partialupgrade.php" by uncommenting the last function (the one that upgrades the session from the last 10 years, and started up the upgrade again. It is still cranking on the same query. DId I not do something that I was supposed to to set this up? I didn't log in to Moodle and view it/access it from the admin menu as my code is in a state of flux--between the two versions. So I just re-started the upgrade.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Does anyone have upgrade for massive sites tips and tricks?
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

This plugin is not a mod.

In Moodle 2.1 (and if you want to install it into Moodle 2.0) then it is a local plugin, so it needs to go in the local/qeupgradehelper directory. The plugin is already included in the Moodle 2.1 code.

In Moodle 2.2 it had to move to admin/tool/qeupgradehelper, but again, if you already have the code on your server, it will be there.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

Well I tried just in the moodle directory itself. ANd it didn't seem to get picked up. Since I am using Moodle 2.2.1 I will use /moodle/admin/tool/ Thanks.Once suggestion . . . you may want to these specifics to the readme.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

OK. I saw that the code was in \moodle\admin\tool\qeupgradehelper already when I went to copy the downloaded zip file to that location. But I made sure that the partialupgrade.php file was there where I uncommented the last function. But it has been an hour and a half and it is still cranking on "update mdl_question_session set manualcommentformat = . . . . " Any other suggestions? Or am I still not setting this up correctly?

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Does anyone have upgrade for massive sites tips and tricks?
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

That update "mdl_question_session set manualcommentformat = . . . ." query is part of the upgrade process from 1.9 -> 2.0. You have not even got to the 2.1 part of the upgrade yet.

I don't know why it should be that slow.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

But I thought you were saying that the update statement was going to be taken care of by the code you wrote? Anyway, thanks.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Does anyone have upgrade for massive sites tips and tricks?
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

No. The really slow bit of the upgrade, that you have not even got to, will be taken care of by my code.

How long it takes to execute one simple SQL statement (that, admittendly, has to alter 12 million rows) is entirely due to MySQL. Not a lot Moodle can do about that.

Since you are testing the upgrade on on your laptop, it could well be the case that your MySQL installation has not been tuned at all, and is therefore running much more slowly that it should do.

 
Average of ratings: -
Picture of A Guy
Re: Does anyone have upgrade for massive sites tips and tricks?
 

The problem was the HUGE questions tables. First the sessions table and then the states table and then . . . I know because with no where else to turn I just started truncating them one at a time as the upgrade script failed, printing out the SQL of the last attempt.  I then just truncated all the question tables. And it upgraded within a few minutes. The code you mention, even with me setting it up, was never engaged for whatever reason. I am not finished with the upgrade as the problem now (as I move this back to the server) are the files and the file tables. Obviously some planning needs to be put in place to prevent such overwhlemingly large tables. But I just started working on this site.

 
Average of ratings: -