Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Christe McMenomy -
Number of replies: 7

I diligently created a copy of my production 3.3 Moodle on my Mac to test everything before performing upgrades to 3.5 on the production system.  My upgrade on my Mac ran perfectly and took about 5 minutes.

I then tried to do the upgrade on my LINUX system; it hangs on the first step (creating top-level question categories), getting variously through 2700 to 3700 of the 3800+ categories on multiple runs, sometimes more, sometimes less.

Obviously, something is timing out here, but I don't know what to reset or increase.   There are no mysql DB error log messages, PHP log messages, or Apache messages.

The production system is on a Linux gentoo server (4.14.52) running PHP 7.1.18, Apache 2.2, and mysql 5.5.36, with the essential theme. I have installed the moodle 3.5 with all plugins upgraded to current releases, including the theme.  Since there are no errors, I must be running out of system resources, as this would be the only difference between the two systems.  Any ideas where I should check to fix / add the required extra 10 seconds I need to complete the update?



Average of ratings: -
In reply to Christe McMenomy

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I wouldn't assume it was a timeout or a resource issue without some evidence. Although I would check things like the memory limits in php.ini look sensible. 

Have you checked your web server's error log for errors?

Average of ratings: -
In reply to Howard Miller

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Christe McMenomy -

Yes. As stated, the PHP, mySQL, and Apache logs list no errors.  The upgrade simply dies. If I restart it, it runs from the beginning, and dies in a different place.  One time it died just 11 records short of completing the category upgrade, another time it died with nearly 1000 records to go.  Hence my assumption that it is a resource issue: it does not die on the same record each time (in which case I would suspect a data problem).  It does not leave the database in an irreparable state, since it restarts and reports no errors (but dies again).

Which php.ini limits should I check?

Average of ratings: -
In reply to Christe McMenomy

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Ken Task -
Picture of Particularly helpful Moodlers

On your Mac, did you setup that Moodle instance from scratch?   OR use MAMP package with pre-installed Moodle?   If MAMP, the config files for things were probably ready to take on the upgrade.

Not really fair to compare ... Apples and Oranges there ... no pun intended.

Upgrading to 3.5.1 now has heftier requirements for DB  .. so not only tweaks to php.ini increasing things like time for a script to run and maybe even max memory a script can consume, but also MySQL DB config.

Could we get a peak at the config file for your Gentoo's MySQL .... minus any logins/passwords of course.

In most Linux's  that's in /etc/ and called 'my.cnf'.  Not sure about your server OS - Gentoo.

Since it's Gentoo, am assuming it's dedicated and not remotely hosted in a shared system of some sort as most online providers offer Ubuntu or CentOS.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Christe McMenomy -

On Mac, mySQL, Apache, and PHP are all homebrew since the Apache and PHP native to my Mac OS (El Capitan) are all too old for Moodle 3.5.1.  I backported the DB from the production system to the Mac along with the previous version of the Moodle (3.3), made sure everything worked and was updated (i.e., all tables Barracuda and unicode compliant), then did the upgrade, so that I was using the same procedure I would use on the production system and the same data.  The only differences were in config.php hard links to DB names and file locations, and in the environments controlling the PHP and mySQL. Pretty much all of the Mac install is defaulted (i.e., there's very little in the my.cnf file except the required settings for unicode compliance).

Below are the resource-related settings for the gentoo my.cnf instance.

key_buffer_size = 32M

max_allowed_packet = 20M

table_open_cache = 1024

sort_buffer_size = 512K

net_buffer_length = 16K

read_buffer_size = 256K

join_buffer_size = 4M

thread_cache_size = 4

max_heap_table_size = 64M

tmp_table_size = 64M

max_connections = 128

read_rnd_buffer_size = 512K

myisam_sort_buffer_size = 8M

innodb_lock_wait_timeout = 50

max_allowed_packet = 16M

[isamchk]

key_buffer_size = 20M

sort_buffer_size = 20M

read_buffer = 2M

write_buffer = 2M


[myisamchk]

key_buffer_size = 20M

sort_buffer_size = 20M

read_buffer_size = 2M

write_buffer_size = 2M










Average of ratings: -
In reply to Christe McMenomy

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Ken Task -
Picture of Particularly helpful Moodlers

Well, that explains the Mac side ... smile  Thanks.  Rick ought to be interested in how you got php intl extension installed on the Mac.

On the Gentoo box ... for MySQL ... what setting do you have for 'handshakes' for 'clients'?

Suggest installing MySQLTuner.pl and running it on the Gentoo server environment.

https://www.google.com/search?q=MySQLTuner

'spirit of sharing', Ken

Average of ratings: -
In reply to Howard Miller

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If it had reached a resource limit an error would be logged. So, if there's no errors it either isn't hitting a resource limit or you're looking in the wrong place. 

Make sure you have Debugging full enabled in your config.php (copy section 7 from config-dist.php)

Average of ratings: -
In reply to Howard Miller

Re: Upgrade 3.3 to 3.5.1 hangs creating top-level question categories

by Christe McMenomy -

"If it had reached a resource limit an error would be logged. So, if there's no errors it either isn't hitting a resource limit or you're looking in the wrong place."  Right.  I looked in the likely places as mentioned (apache log, mysql log)....no messages.  Not sure where else to look. I don't have root access to the production server, but my sysadmin is amazing and actually does what I ask when I know what to ask.  

The question is now moot.  I reran the upgrade .... it took four tries on the test system and two tries on the production system, but it did eventually get through all of the the question category rebuilds in each environment, and then finished the upgrade for some 380+ Moodle core modules and two plugins without any hitches. My sense is that something somewhere was timing out.  Since I was doing this through a remote browser, it is even theoretically possible that my browser connection was interrupted.  I'm assuming if that was the problem, there would be no error messages ... so it sort of makes sense.

At the moment, everything on both test and production look good, and this means (unfortunately for the analysis) that I'm not in a position to figure out why there was an issue in the first place.

I really appreciate the feedback.  I know you all have a life and other problems, and it is wonderful to me (especially in the brittle environment of civility in which we now dwell) that you would take even a few moments to consider my issue and try to help.  I hope you have the most wonderful day!


Also: I do have the php intl extension running on my Mac. When I realized that the El Capitan PHP wasn't at 7.0+ and I wasn't ready to upgrade to High Sierra (which is) since I'll lose some apps I like, I installed PHP 5.6 and 7.1.18 from homebrew, and then installed homebrew Apache 2.2 so I could tweek it. After much consternation I got the El Capitan PHP and Apache turned off and everything set so that the Mac initializes and uses the homebrew environment on reboot. I don't remember doing anything special in configuring PHP 7 beyond the basics to get it working, so the homebrew install included the PHP intl as part of the installation.  If you have specific questions, I can dig into my upgrade notes to find out.  Right now, my Mac test and support environments mimic my gentoo Linux test and production environments perfectly -- same data, same results.  Back to actually writing quizzes for my courses....

Average of ratings: -