Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
Number of replies: 48

Hi all!

I’ve done a lot of pre-work/research with this error and found a possible solution that seemed to work for a number of others, but it did not work for me and I am really stuck.

This started to occur after we upgraded from 3.5 to 3.7

PHP Version 7.3.5
MySQL 5.0.12

We have been running moodle for 15 years and this is a new one for me.

There is a tracker item for this error here https://tracker.moodle.org/browse/MDL-43314 but the fix was 2.6.1.

I tried the following workaround which seemed to work for others but it did not work for me.

I got the same error today. To fix it, I just did this (I was not in maintenance mode before) :
/usr/bin/php /var/www/moodle/admin/cli/maintenance.php --enable
/usr/bin/php /var/www/moodle/admin/cli/maintenance.php –disable

https://moodle.org/mod/forum/discuss.php?d=233629#p1266649

https://moodle.org/mod/forum/discuss.php?d=212888#p1266648

I also saw this post from Ken task who ran into a similar issue but I’m not quite clear on what he did to resolve his issue.

https://moodle.org/mod/forum/discuss.php?d=351902

 

Average of ratings: -
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Hmmmm .... someone rattled my tree ... sooooo ...

First, me thinks your PHP is too high and your MySQL isn't high enough! sad

Please see: http://www.syndrega.ch/blog/#php-and-dbms-compatibility-of-major-moodle-releases

Series 3 of moodle has been an adventure in that upgrades to moodle had dependencies for upgrade to PHP and to MySQL.  Fairly easy to fall into a 'perfect storm'.  Upgrade PHP too high (too soon), then upgrade to Moodle would fail with some strange errors ... actually 'rabbit holes'.   So the 'trick' to avoid was not really a 'trick' but to research what/when using Moodle's environment check ... update component ... then stepping through each version of Moodle to see the checks on PHP version and MySQL.  There's your upgrade plan ... had to upgrade PHP prior to pulling trigger on upgrade to Moodle on some versions.  Then upgrade moodle.

MySQL less so ... there it was running Barracuda file system and a couple of InnoDB settings which several versions of MySQL could do.

Now, however, the 5.x series of MySQL is fading away now and the next latest and greatest is version 8 ... better ... yes, and will require un-doing some tweaks set in my.cnf cause defaults have changed in version 8 (more adventure).

My posting (kinda rant) was about a popular theme ... Essential ... and the number of new settings an update to that theme might have after performing an update to it.   That plus other upgraded plugins also having new settings resulted in following the recommendation (at that time) to upgrade all plugins at once taking too long via web and ending with 'stuck' where you are stuck now!

Your moodle thinks it needs updating .... something ... if not core, a plugin, or plural plugins.

You don't have to install all addons at once ... if you know that one has traditionally had many new settings in past (like Essential did), save it for the last plugin to update.   Do the other plugins in 2's or 3's if ya have a bunch - idea is to lighten the load on web service.

OR ... if one doesn't suffer from 'CLAS' (command line avoidance syndrome), take your web service out of the loop and use the upgrade.php script in code/admin/cli/.   That script does whatever there is to upgrade.

So, think your bottom line ... roll back php to version 7.2.  Restart web service.  Get over CLAS and use admin/cli/ maintenance --disable.  Did that work?  If so, then run cron from admin/cli/.  If cron hasn't been running regularly, advise running it manually maybe several time.  Many things might need 'catching up'. :|

Check site ... OK?  Then consider doing the upgrade to MySQL to 5.7.

If not OK, what's not OK and we'll work on that!

My best guess! smile

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Hi Ken

It must be the MySQL 5.0 (5.6 needed). PHP 7.3 is OK. Ref. http://www.syndrega.ch/blog/#php-and-dbms-compatibility-of-major-moodle-releases

I wonder how Moodle 3.5 was running. The minimum MySQL is 5.5.31!

In reply to Visvanath Ratnaweera

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
I entered the wrong version!!! DOH
MySql version should read 5.7
In reply to Visvanath Ratnaweera

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Yes.  True. For OP's version of Moodle 7.3 of PHP is supposed to be fine, but ... stuck in upgrading plugins loop and OP didn't share what additional plugins (non core) were installed.   MIght be faulty human logic (mine), but by rolling back PHP and CLI upgrade script the bug in an addon plugin might show.

I install moosh now on new systems and collect the shortnames of all additional plugins (not core).   Then run moosh command with grep to pull out just the non-core plugins info for version compatibility for those addons.

addons.txt contents

format_topcoll
report_benchmark
report_coursesize
local_mailtest

checkaddons script

#!/bin/bash
#
echo 'Add-on listing: ';
cat ./addons.txt;
echo '---------------------';
for i in `cat ./addons.txt`
do
   echo "Addon in que: $i";
moosh -n plugin-list |grep $i
done

For this specific example, a Moodle 3.7.highest

Execution of above results in display of this:

Add-on listing:
format_topcoll
report_benchmark
report_coursesize
local_mailtest

---------------------
Addon in que: format_topcoll
format_topcoll,1.9,2.0,2.1,2.2,2.3,2.4,2.5,2.6,2.7,2.8,2.9,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,https:// moodle.org/plugins/download.php/21284/format_topcoll_moodle38_2019111701.zip
Addon in que: report_benchmark
report_benchmark,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,https:// moodle.org/plugins/download.php/21093/report_benchmark_moodle38_2020022400.zip
Addon in que: report_coursesize
report_coursesize,2.3,2.4,2.5,2.6,2.7,2.8,2.9,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,https:// moodle.org/plugins/download.php/19641/report_coursesize_moodle37_2019052800.zip
Addon in que: local_mailtest
local_mailtest,2.5,2.6,2.7,2.8,2.9,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,https:// moodle.org/plugins/download.php/20592/local_mailtest_moodle38_2019111700.zip

If am considering upgrading this 3.7 instance to 3.8.highest, am then alerted to *possible* issue with report_coursesize as Moodle plugins shows compat up to 3.7.

Oh, yeah ... the url's above would have been clickable here had I not added a space after https://  On servers I sometimes use above to build a plugingetlist and wget them to a pluginarchive folder that can then be transported to another moodle instance ... building a dev instance of production comes to mind.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
We downgraded to PHP 7.2.
- Using command line I enabled maintenance mode, ran the upgrade script again successfully and disabled maintenance mode. I restarted php-fpm and cleared caches (via command line) just for fun.
- Accessing the system via web browser then prompted me to upgrade again. (The Thought Plickens)

It captured some of the older plug-ins that re-installed themselves because they were still in the database and I guess not uninstalled properly. 
 
And BTW I try to avoid plug-ins like the plague.  Maybe because I'm cursed from working with Moodle from the early days thoughtful.  Actually I don't think this is a curse, I think this is a good thing - it has come a long way!  And I also research the version compatibility and do full testing with each plug-in when I install them.  So far we have not had an issue up until now.
 
I will do a clean up to see if this helps. Thanks again for your input very much appreciated!
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Yep, if a plugin (old) was not properly un-installed (code + db tables), the presence of the code of the plugin, moodle will assume it's a new plugin and will attempt re-install.

Removing the offending code folder of the plugin, will render as 'missing from disk' when viewing plugins page.   Must click the button to upgrade DB to get tables removed and mdl_config_plugins table updated.

New CLI script in versions 3.7.x and 3.8.x exist now called 'uninstall_plugins.php'.  Works!

So your on your way to 'fixed'? ;)  And, guess you could upgrade your PHP again. (sorry bout that!)

'SoS', Ken



Average of ratings: Useful (1)
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

Further to my last post, I did get further!  But still stuck.

After properly uninstalling some theme plug-ins I was able to get cron running successfully via command line!  Yay. Progress.

But when I access our moodle page via the browser afterward I still get a prompt to upgrade.  When I choose to upgrade at that stage, it comes back saying nothing needs to be upgraded but it goes through the process anyway.

All seems fine at that point until I try running cron.php again and I'm back to square one.

I repeat all the steps again via command line in this order:

...maintenance.php –disable
...upgrade.php
...maintenance.php –enable
...cron.php

Cron runs as it should.

I log back into the web browser and get a prompt to upgrade.

...aaaaaand repeat.

At this stage I am thinking I need to re-upgrade from fresh (using command line).

 

 

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Clearing cache each time?

'SoS', Keen

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

Yes, I have added that step!

I have re-upgraded to latest weekly and that did not fix the problem so I will continue with a full uninstall of the plug-ins.  (And I found the forum post on how to uninstall plug-ins using cli so that is a HUGE time saver)

I still find it odd that everything was working fine (with the plug-ins and PHP 7.3) up until I upgraded from 3.5 to 3.7.

 

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

"... until I upgraded from 3.5 to 3.7." ... well that begs the question, how did you upgrade?

'SoS', Ken


In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

  Re: "... until I upgraded from 3.5 to 3.7." ... well that begs the question, how did you upgrade?

Good point.

Well I am now at the point where I have uninstalled (and removed the directories) for all our non-core plugins except one which doesn't want to uninstall.  auth_saml2 (note: it is the updated version of this plugin for moodle version 3.7 that is installed)

At this point I'm not sure if even removing this final plug-in will help?

Perhaps I should go back one step further and restore a new version of our production database and really start from scratch.

 

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

So the server you are working on is a clone of a production server?

Then yet another question ... how did you clone it?

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

We have now restored a clean version of our database (version 3.5) with the same code-base from our production server (I double checked the versions in both version.php as well as in mdl_config to make sure they were the same version.

Moodle STILL thinks it needs to upgrade something.  

Re: new server, no it's not a clone it's an entirely new set up with upgraded OS.  But, the thing is, I had Moodle running on this new webserver for some time, including cron, prior to April 16th which is when I upgraded.  So maybe I botched something in the upgrade but I am a total loss as to why a fresh database, pre-upgrade would still be giving me the same issue. 

This is a bit insane. Am I missing something??  *officially going crazy*

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Well, if we keep doing the same thing and it doesn't work, why don't we change up what we are doing?

I asked how ... it appears to me that on the server with the issues you did an install of 'same version' (that could be true, but maybe not), got it running, then dropped that database for it, and imported a DB dump from production server.   Is that right?

IF so, you just imported all internally stored URL's/paths etc. that pointed to https://prodserver.somenet.tld in a server that's *NOT* prodserver.somenet.tld but another IP or devserver.somenet.tld.

Yeah ... things will break.

What if you cloned production ... made a backup of code, moodledata, and DB dump.   Then used those site backups to restore to this dev machine.

Before you import the .sql file, however, do search and replace on the sql file for https://productionsite.somenet.tld/ and replace with https://devsite.somenet.tld/ ... then import that edited sql dump into a new fresh/blank database where it only has the proper character set and collation.

Also remember a clone is  copy of everything ... including the moodledata directory.

IF there were a serious bug in code me thinks there would me tons of folks who upgraded code from same versions to same versions that would be saying ... me too, me too, etc.

Also, let's not forget that cron is part of operating system ... what's the crontab file look like ... etc.

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

No I don't think it's a bug in the code!  I really don't.  I feel like I munged up the upgrade somehow and as you say, something somewhere is "stuck".  And I am now even second guessing whether or not cron was in fact fully functional on this particular webserver.  I just don't know how to get back to square one without recreating the wheel but it looks like mayhaps we have to.

I asked how ... 

What if you cloned production ... made a backup of code, moodledata, and DB dump.   Then used those site backups to restore to this dev machine.

What we did was copy over the code-base and moodledata after the new server was built up.  And yes we did a backup and restore of the database (from our production system) into a fresh DB schema.  Which our DBA has done for us before.  That is what you are suggesting above, correct?

IF so, you just imported all internally stored URL's/paths etc. that pointed to https://prodserver.somenet.tld in a server that's *NOT* prodserver.somenet.tld but another IP or devserver.somenet.tld.

Hmmm... I do get this concept but I thought that the "internally stored URL's/paths" would be only for "content".  I thought that making sure our config file is pointing to the right server and cleaning out all of the file caches would suffice for basic functionality of the system.  Am I wrong?

Before you import the .sql file, however, do search and replace on the sql file for https://productionsite.somenet.tld/ and replace with https://devsite.somenet.tld/ ... then import that edited sql dump into a new fresh/blank database where it only has the proper character set and collation.

Can this be done after the fact with the search and replace tool?

Also remember a clone is  copy of everything ... including the moodledata directory.

IF there were a serious bug in code me thinks there would me tons of folks who upgraded code from same versions to same versions that would be saying ... me too, me too, etc.

Also, let's not forget that cron is part of operating system ... what's the crontab file look like ... etc.

I have disabled cron in the crontab for now and just focusing on getting it running manually.

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

How did you 'copy over' to new server after 'build up'?

For that matter, just so I am clear, what is 'build up'?

Yes could run search and replace ... did you?

What did you search for?  What did you replace with?

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
You are still here!!  Yay!  Are you getting tired of this one yet??? I am!!

        How did you 'copy over' to new server after 'build up'?
 
Using SCP on the command line in the shell from one server to another.

        For that matter, just so I am clear, what is 'build up'?
 
Our web admin provisioned a new webserver rather than cloning an existing one with an upgraded OS (redhat) and then configured with PHP, apache, etc.  Basically following the guidelines on moodle.org for recommended versions, appropriate php extensions, etc.  Trying to do everything by the book as close as we can.  To avoid situations like this!  LOL
 
        Yes could run search and replace ... did you?
 
 
No, not yet. 

After more troubleshooting from hell and getting nowhere in the infinite loop of doom, I decided to go revert back to the original platform that we were using at the time of my first post.  But we now have PHP version 7.2 now, not 7.3 as originally stated.  We are back on moodle version 3.7 with the newest weekly release installed. 
 
The site is up and running and we are not getting any PHP errors during the normal course of testing basic functions.  We still get the "update pending message" when attempting to run a task via the browser or when I try to run cron.php via the shell.  Same message.
 
I CAN get past this error successfully if I re-run the upgrade.php script via the shell as you suggested. 
 
But after that, once I try to access moodle again via the browser, I start the infinite loop of doom where as an admin I am prompted to upgrade again.  And IF I upgrade, all is well!  I am functional.  Until of course I get to the scheduled tasks and they say NOPE not gonna do it, you need yet another upgrade.
 
I guess where I am confused is, why would I get this prompt to upgrade after logging into Moodle via the browser only AFTER running the upgrade.php script via the command line?  It's almost like the web version and the shell version of upgrade.php are different.
 
But I'm going to sleep on it and hopefully have a new outlook in the morning as it's 11 pm here now.
 
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Susan,
I cannot fully understand your path - got a quick glance of this thread - and maybe this post of mine will be completely not useful but... I know that in the past you hit some strange bugs due to a combination of env settings wink (MDL-51554).

To be successful in your troubleshooting you actually need to get in touch with the code since "we" are not able to replicate your specific issue in our environments - at least at the moment.

The first piece of code is the function telling you that your site needs an upgrade when running scheduled tasks: the initial one - Moodle upgrade pending, cron execution suspended. - comes from a function in lib/moodlelib.php named moodle_needs_upgrading().
The code is self explanatory, better if you follow it with an IDE: it calculates a SHA1 hash based on the list of all the items with a version in a Moodle instance, starting from Moodle core code up to 3rd party plug-ins.

One of the reason of the hash mismatch is an actual difference at least in one of those versions i.e. something has actually changed in the deployment and the upgrade is actually needed.
That being said, it could be possible - I do not know how, at the moment - that evaluating the hash could depend on the PHP version (?) or on the php.ini settings (!) e.g. PHP CLI could have different php.ini settings compared to the PHP interpreter used to run the Moodle code through your web server.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Susan, like Matteo, I don't fully understand everything going on, so sorry if I suggest something that you have already tried, or if Ken has already advised you to do this.

Have you tried installing a brand new Moodle (don't replace what you have), just a new experimental Moodle, from scratch? This might let you know that a new Moodle works fine on your server, and that your problem has something to do with your current Moodle, not your server. If the new Moodle has problems, then your problem probably has something to do with your server and settings.
In reply to Rick Jerz

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
Hi Rick,

Yes, we have considered this! And at this stage this might be what we end up having to do if we can’t find the error of our ways troubleshooting the current setup.

Thank you for the suggestion!
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Susan, you might have the skill level to easily do this.  I have this video "Install Moodle on a VPS Using cPanel," which explains how to install a Moodle using cPanel.  You might have FTP and SSH skills and can do an install in your sleep.

Well, the concept is that you install a brand new Moodle on your server, let's call it somthing like moodle_exp (for experimental.)  Do everything in parallel.  Have a database called moodle_exp, a moodledata folder called moodledata_exp, and a moodle application directory called moodle_exp.  In my video, you will see me using moodle38.

Once you try this, you might be able to prove if a fresh moodle can run on your server, of if you have some kind of server problem.

If you have success with moodle_exp, start adding some of your desired plugins to it, just to make sure that they work.  Eventually, try adding the exact plugins that mirror your production moodle.  I kind of show this in my "Upgrade Moodle on a VPS Using cPanel" video.  Some of the concepts in this video might apply, or help you think this through.

Eventually, if you get your moodle_exp working, you could even try moving your database and moodledata contents to it.  This would be in my "Migrating Moodle" video, which I have not produced yet.  (Sorry.)

And finally, when moodle_exp is working with your plugins and data, you can then swap it with your production Moodle.

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Humor me ...

in ssh shell

which php

/the/path/which/shows/php -v

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Sebastian Mantilla -

Hi Ken,

Thank you and thanks everyone that is supporting us.

Here is the output:

[root@elearnlooknv1 data]# php -v

PHP 7.2.24 (cli) (built: Oct 22 2019 08:28:36) ( NTS )

Copyright (c) 1997-2018 The PHP Group

Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

    with Xdebug v2.9.5, Copyright (c) 2002-2020, by Derick Rethans

    with Zend OPcache v7.2.24, Copyright (c) 1999-2018, by Zend Technologies


In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Sebastian,
please check any difference between PHP being executed via shell and via web server starting from their settings e.g. php --ini and the HTTP Response of a PHP page like <?php phpinfo();.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Matteo, I learn something new every day.

Yes, I can compare the results of php --ini with phpinfo(). For example, php --ini does not show mysqli, but phpinfo() does show it.

However, as a novice, I don't know what this means. As I recall, mysqli is required. If I were to rely on php --ini, by it not showing mysqli, I wouldn't know that I already have this installed.

Any thoughts about the differences would be appreciated. It appears that using phpinfo() is more accurate than php --ini. Thanks.
In reply to Rick Jerz

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Rick,

Any thoughts about the differences would be appreciated.  It appears that using phpinfo() is more accurate than php --ini. Thanks.

it's accurate against the context you're asking for wink.
Please check for php -m which lists the modules being actually loaded - you'll find mysqli there. And try to use php --ini | grep my: it could be actually there.

The reason why I'm asking for any difference is due to the serialize() PHP function being called before creating the SHA1 hash: it would be possible that the hash stored into the Moodle config table (SELECT * FROM mdl_config WHERE name = 'allversionshash') and the one evaluated using CLI cron are different due to a discrepancy in the settings of the two contexts and not for just an actual mismatch of any Moodle component versions.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Sebastian Mantilla -
Hi Matteo,

Thank you very much for your help.

A little bit more context here, it could be useful.

We're on a just created virtual machine, with Red Hat Enterprise Linux release 8.2 (Ootpa), httpd-2.4.37, php-fpm-7.2.24, php-7.2.24, all from RedHat so we're using the php stream/module.

I don't think we're using different settings/versions when running from the command line vs the web.

So this is the output.

Web
System Linux elearnlooknv1 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Wed Feb 26 03:08:15 UTC 2020 x86_64
Build Date Oct 22 2019 08:28:36
Server API FPM/FastCGI
Virtual Directory Support disabled
Configuration File (php.ini) Path /etc
Loaded Configuration File /etc/php.ini
Scan this dir for additional .ini files /etc/php.d

Command line
[root@elearnlooknv1 conf.d]# php --ini | grep php.ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File: /etc/php.ini

[root@elearnlooknv1 conf.d]# php -m
[PHP Modules]
bz2
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
intl
json
ldap
libxml
mbstring
mysqli
mysqlnd
oci8
odbc
openssl
pcntl
pcre
PDO
pdo_mysql
PDO_ODBC
pdo_sqlite
Phar
posix
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xdebug
xml
xmlreader
xmlrpc
xmlwriter
xsl
zip
zlib

[Zend Modules]
Xdebug

[root@elearnlooknv1 conf.d]# php --ini | grep my
/etc/php.d/20-mysqlnd.ini,
/etc/php.d/30-mysqli.ini,
/etc/php.d/30-pdo_mysql.ini,

I've opened the code as a project on phpStorm and I'll look into the moodle_needs_upgrading().

Thank you.
In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Sebastian,

We're on a just created virtual machine, with Red Hat Enterprise Linux release 8.2 (Ootpa), httpd-2.4.37, php-fpm-7.2.24, php-7.2.24, all from RedHat so we're using the php stream/module.

OK, that means nothing side-by-side at the moment: just one PHP version with one overall configuration.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Sebastian Mantilla -
Hi Matteo,

So I added a bit of code to the function to catch the values and I get this:

When running from the command line:
[root@elearnlooknv1 conf.d]# sudo -u apache /usr/bin/php /var/www/moodle/admin/cli/cron.php Moodle upgrade pending, cron execution suspended.

When running from the web

Prepare submissions for annotation

Moodle upgrade pending, cannot execute tasks.

If I understand correctly the run from the web and the run from the command get the same value for hash (fcab0415a9d9cd0aa7680a0d11390b5d408acc37) but it is different from the value on the database (4e310883fbfea952b27c65c90fed9544c7bd0036).

Is this helpful in any way?
In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Sebastian,
that is expected since that is the reason for the blocking message - it tells nothing to me since it strictly depends on the exact Moodle version and on the whole of your 3rd party plug-ins, quite unpredictable to me but predictable to you.

What you should try to debug/understand is why there was a time in which the overall code versions have been hashed into that value in the DB whereas now in your current setup that evaluation is different from the old one, stored into the DB.
If the code is exactly the same what are the "elements" being actually changed? The code? The environment? Both?

That value is set into the DB just after a successful installation stage: look at the function upgrade_noncore() in lib/upgradelib.php.

Long story short: if you update the value of that record with the value being actually evaluated, you'll unlock the situation but... you'll miss to find why this mismatching happened - and it could re-appear the next time you'll change "something" in your deployment.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -

Hi Matteo! 

Thank you so much for taking the time to give us some feedback on this.

"...if you update the value of that record with the value being actually evaluated, you'll unlock the situation but... you'll miss to find why this mismatching happened - and it could re-appear the next time you'll change "something" in your deployment."

If I understand you correctly, you are saying we could manually update this record in mdl_config as a workaround?  We would be willing to do this on one of our systems and continue to troubleshoot the other however, upon testing it does not get us out of this loop.  Maybe I am missing something.  This is what is currently happening:

  • The value for allversionhash is 'XYZ' in mdl_config
  • The scheduled tasks are not running and it is determined that "there is a pending upgrade"
  • The value for allversionshash gets updated to 'ABC' in mdl_config after the cli upgrade and we are now able to run scheduled tasks via cron.php via the command line
  • Admin user then logs into the web interface but Moodle expecting to see 'XYZ' not 'ABC' so Moodle prompts for an upgrade.  If we proceed with the upgrade the value for allversionshash gets updated back to 'XYZ' in mdl_config which in turn "breaks" the scheduled tasks since they are expecting to see 'ABC'

So whichever way we update the value in the database it is still broken.

I should also note that on the system we are currently working on, I have removed ALL (but one) of our custom plugins.  We still have SAML2 plug-in active (we use the one from Catalyst) as it does not have an option to uninstall via web interface or cli.  Odd.  I'll have to look at the developer docs to see how to do this I suppose to make sure we are 100% plug-in free when troubleshooting.  I suspected the problem initially may have had to do with one of our plug-ins but alas nothing changed after removing them one by one.  Unless of course the problem is with the SAML2 plug-in which is maybe why I can't uninstall it.  *sigh*  That would at least make sense to me if that was the case in the end.

 

 

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Hmmm ... learned something new ... never noticed allversionhash in mdl_config (even if I did, it would have been a mystery) and never understood it's value could be a 'show stopper'.

Just tinkered with a 3.8.x site ...

First, use mysql client to see and archive allversionhash value.

What if, one were to add a line to config.php such as:

$CFG->allversionshash='';

setting that value to nothing.

Login to the Moodle as admin level.
Will be thrown to upgrading screen for an environment check.
Continue will take one to the plugins.

My site found no plugins needed updating.

Yours might.  Upgrade it if it does.  Just that plugin.

After moodle comes back to that screen and before clicking the 'Upgrade Database' button, go into config.php and comment out the allversionhash line added.

Click the button.

https://github.com/moodle/moodle/blob/master/lib/upgradelib.php

        // Update cache definitions. Involves scanning each plugin for any changes.
        cache_helper::update_definitions();
        // Mark the site as upgraded.
1926        set_config('allversionshash', core_component::get_all_versions_hash());

That's the only line with allversionshash ... where/how is it calculated?

Am wondering if any value that looks like an allversionhash will work?

Like this one:

80d770e323398580b47470cbb8d6daa07eb1a897

Yeah, I know ... not a programmer so shoudn't throw out such wild ideas! sad

'SoS', Ken

In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Ken,

Hmmm ... learned something new ... never noticed allversionhash in mdl_config (even if I did, it would have been a mystery) and never understood it's value could be a 'show stopper'.

There are several "internals" which no one will consider 'till a strange break will happen wink.

where/how is it calculated?

Look at lib/classes/component.php:

    /**
     * Returns hash of all versions including core and all plugins.
     *
     * This is relatively slow and not fully cached, use with care!
     *
     * @return string sha1 hash
     */
    public static function get_all_versions_hash() {
        return sha1(serialize(self::get_all_versions()));
    }

In the same file you'll find how get_all_versions() has been implemented.

HTH,
Matteo

In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Susan,

Admin user then logs into the web interface but Moodle expecting to see 'XYZ' not 'ABC' so Moodle prompts for an upgrade.  If we proceed with the upgrade the value for allversionshash gets updated back to 'XYZ' in mdl_config which in turn "breaks" the scheduled tasks since they are expecting to see 'ABC'

your fourth point is now the new focus of your debug, you should follow the code: there's no reason - i.e. somewhere there is a bug! - to let Moodle evaluate the actual hash with a different value since the way that value is evaluated should be the same!

I've actually no idea why this happens in your system but you've nailed it down so it is new "big" step towards the solution: it's "just" a matter to discover why wide eyes.
Try to enable Moodle debug at DEVELOPER level: should it be a "version.php" file with wrong permissions i.e. a different list of versions to be hashed, I mean different list size when the code runs by CLI vs by WEB?
Who knows... but it sounds a first step to understand any difference between CLI and WEB running "the same code".

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
I think we may have resolved the issue. Or if not resolved, at least found a workaround.

I have modified lib/setup.php as follows:

ini_set('serialize_precision', 17);

TO

ini_set('serialize_precision', 100);

Short story long, we set up a new environment on the same web server using the latest release of moodle (3.8+) and a local MySQL database to rule out the environment but low and behold we encountered the exact same behavior. Honestly wasn’t expecting that.

I read more about how this all works in https://tracker.moodle.org/browse/MDL-43314 and so we decided to output the value of serialize_precision and sure enough, it came back with 2 different values:
 
  • Admin logs into web interface and tries to run a scheduled task manually which fails due to pending upgrade: serialize_precision value=17
  • Admin runs cli upgrade, cron runs successfully via command line and logs back into web interface to upgrade required message: serialize_precision value=100
So am I missing something as to why lib/setup.php would be setting the init value to 17 but the lib/classes/component.php sets it to 100 on the fly? While I have programming experience I’m certainly not a developer by any means so I am more than likely missing something here. Is this a bug or is there still something wrong in our environment setup?
Average of ratings: Useful (1)
In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

Not a programmer myself but wouldn't having php's Zend X-Debug on be involved with the precision difference?

Or am I getting confused now with 2 persons in this thread with similar issue and servers but not same server - Sebastian

Anyway ... congrats!   You appear to have solved it! Your 'Helpful' icon deserves a big + in it! smile

'SoS', Ken


In reply to Susan Mangan

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended” - ALMOST for the DEVELOPER forum

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Great job Susan and your Team!
You have found what I've been discussing above about env settings: it's time to file an issue into the Tracker, but some few details are still missing.

serialize_precision is set to 17 since https://github.com/moodle/moodle/commit/404ecca3761b2879b12ac2e71eb18f688c5e40ee (MDL-43314) so it looks like in the "Moodle CLI path" that setting is missing for mostly a bug.

100 looks like "arbitrary": could you look into your PHP ini settings for that value?
If your read https://www.php.net/manual/en/ini.core.php, 100 is there for historical reasons but you should read it in your ini settings:

Before PHP 5.3.6, the default value was 100. Before PHP 7.1.0, the default value was 17.

Here is my trial on PHP 7.2 on CentOS 8.1:

# dnf module install php:7.2
# dnf module list php
Last metadata expiration check: 0:00:55 ago on Fri 01 May 2020 10:45:58 AM CEST.
CentOS-8 - AppStream
Name                 Stream                    Profiles                                      Summary
php                  7.2 [d][e]                common [d] [i], devel, minimal                PHP scripting language
php                  7.3                       common, devel, minimal                        PHP scripting language

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
# php -v
PHP 7.2.11 (cli) (built: Oct  9 2018 15:09:36) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
# php --ini | grep php.ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File:         /etc/php.ini

# grep precision /etc/php.ini
; http://php.net/precision
precision = 14 ; When floats & doubles are serialized, store serialize_precision significant ; precision.
serialize_precision = -1 # php -r 'echo "precision = " . ini_get("precision") . "\n";' precision = 14 # php -r 'echo "serialize_precision = " . ini_get("serialize_precision") . "\n";' serialize_precision = -1

Why the Moodle hash depends on that value?
It relies on serialize() which relies on __toString() of those "numbers" in the list of the whole Moodle component versions, which unfortunately are a ... float number at least for the main Moodle core version:

$version  = 2020042800.00;

The end solution would be IMHO to cast the version value to a string before hashing it which is independent from any float precision issue - float is evil in any language not actually a PHP issue!

That being said, I could work on it starting even from filing the issue into the Tracker (I've almost write it here big grin) and find the "best" solution: in the mean time I strongly suggest you to set the PHP serialize_precision to 17 - the same value now hard-coded into Moodle - into your PHP ini setting; just mimic what written in lib/setup.php:

ini_set('precision', 14); // needed for upgrades and gradebook
ini_set('serialize_precision', 17); // Make float serialization consistent on all systems.

For the Tracker issue purposes, would your mind to share here:

  1. your CLI comand, w/o any sensitive data
  2. the PHP ini settings above in your RHEL 8.2 - just to save my time to install it wink

? TIA!

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended” - ALMOST for the DEVELOPER forum

by Sebastian Mantilla -
Because the changes to the PHP settings are particular to moodle or to our way to run moodle and contain "history" I created a clever virtual host configuration that captured that. The following line does that
ProxyFCGISetEnvIf "true" PHP_ADMIN_VALUE "upload_max_filesize=600M \n post_max_size=600M \n html_errors=off \n mail.add_x_header=on \n max_execution_time=4200 \n max_input_time=600 \n memory_limit=512M \n serialize_precision=100 \n gd.jpeg_ignore_warning=0 \n date.timezone = America/Vancouver \n opcache.max_accelerated_files=8000 \n opcache.revalidate_freq=60"
And because this is on the Apache virtual host configuration it only applies when you run moodle throught the web but NOT when you run it from the com,and line - that creates the different environments.

That is not all, because there is NO trace that the PHP settings where changed on the virtual host configuration we never realized that this was happening.

When we compared running php from the command line and running php through the web they were both reading the same php.ini file but this change made the web configuration different.

Thanks for all your help and sorry for all the trouble.
Average of ratings: Useful (1)
In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended” - ALMOST for the DEVELOPER forum

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Sebastian,
thanks for the follow up, I was already doing some simple improvements to propose removing the dependency on serialize().

Your clever - and imperative (evil I'd say big grin, for some of them you could use PHP_VALUE) due to the Apache php_admin_value directive - solution show me that serialization should stay there to warn the user about that overriding: it looks like we should add a warn in Moodle reports to highlight that precision and/or serialize_precision have been altered from what "required".
Beware that serialization is part of JSON I/O in Moodle so keeping the serialize_precision "unset" and managed by Moodle is the way to go.

Precision on floats started playing a (breaking) role when your Moodle has been updated to something different from .00 e.g. 2019052005.09.

HTH,
Matteo

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended” - ALMOST for the DEVELOPER forum

by Edwin Lynd -
I have the same php settings as you suggested, but I also have the same cron pending problem.

$version = 2020061500.01; // 20200615 = branching date YYYYMMDD - do not modify!
// RR = release increments - 00 in DEV branches.
// .XX = incremental changes.
$release = '3.9+ (Build: 20200618)'; // Human-friendly version name
$branch = '39'; // This version's branch.
$maturity = MATURITY_STABLE; // This version's maturity level.

php -v
PHP 7.2.31 (cli) (built: Jun 9 2020 08:18:32) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.31, Copyright (c) 1999-2018, by Zend Technologies

php --ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File: /etc/php.ini

grep precision /etc/php.ini
; http://php.net/precision
precision = 14
;precision = 12
; When floats & doubles are serialized store serialize_precision significant
serialize_precision = 17

sudo /usr/bin/php /admin/cli/cron.php
Moodle upgrade pending, cron execution suspended.
sudo /usr/bin/php /admin/cli/upgrade.php
ERROR!!! The code you are using is OLDER than the version that made these databases!

it seems very likely on my server to meet this problem when upgrading either plugin or Moodle code.
In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended” - ALMOST for the DEVELOPER forum

by Edwin Lynd -

Hi Matteo,

If there is a saying that  this value is set to 14 digits on Win32 systems and to 12 digits on Linux? 


In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Edwin Lynd -

Does the difference between local and master value of arg_separator.output make any trouble? 

I checked that it is &amp; in the local value while is & in master value.

And I have no idea if I should make it agree with each other and where to change it.

In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

2 cents worth ... in the scheduled task list, 'Prepare Submissions for Annotations' *is* the first one in the list of 90+ task.

/admin/tool/task/scheduledtasks.php - if path to php-cli set in config, there are 'run now' links for individual task.

Also ... in moodlecode/admin/tool/task/cli/ each of the task can be run separate of the whole cron.

php schedule_task.php
Scheduled cron tasks.

Options:
--execute=\\some\\task  Execute scheduled task manually
--list                List all scheduled tasks
--showsql             Show sql queries before they are executed
--showdebugging       Show developer level debugging information
-h, --help            Print out this help

Example:
$sudo -u www-data /usr/bin/php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\session_cleanup_task

As long as config.php file has necessary values to run any cli script, the above might throw something of a clue with either

--showsql             Show sql queries before they are executed
--showdebugging       Show developer level debugging information

Like I said ... my 2 cents!

Will say this is a doooozeee!  On 'leading edge' with MySQL 8.0.x however.

'SoS', Ken

In reply to Matteo Scaramuccia

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Great! Thanks much.

Yes, the php --ini, with grep shows mysqli.

The output of php -m is much easier to read.
In reply to Rick Jerz

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Rick,
please take care that loaded modules (-m) and loaded configurations (-i) should be paired but the path where each configuration is read from.

HTH,
Matteo

In reply to Sebastian Mantilla

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Ken Task -
Picture of Particularly helpful Moodlers

The XDebug extension is not required to run a moodle nor use moodle's cli scripts.  Matter of fact, actually increases resource usage in an already resource hungry app .... ie, Moodle.

The following on a CentOS 7 boxen.

Extensions have their own .ini file ... on CentOS 7 those are found in /etc/php.d/

Any .ini file in there is 'read in' when apache starts/or is restarted.

in /etc/php.d/ one should find a file called 'xdebug.ini'.

You could move it out for safe keeping to /root/ or edit the file commenting out every line in it by adding a ';' in front.

Like so:

; Enable xdebug extension module
; zend_extension=/usr/lib64/php/modules/xdebug.so

With the second line now commented out xdebug.so won't load on restart or start up of apache.  Would recommend commenting out cause you might have a need to use it at some point in time in future.  We are just 'disabling'.

Restart apache service.

Execute php -v again ... should not see XDebug load.

Not saying it's a total fix ... but ....

'SoS', Ken


In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Sebastian Mantilla -
We added xdebug to try and connect moodle to phpStorm and see if we can make any sense.

We don't usually have it.
In reply to Ken Task

Re: Running cron.php from cli or web interface gives "Moodle upgrade pending, cron execution suspended”

by Susan Mangan -
Oh! I think I made a mistake on MySQL version - it’s 5.7.

We follow the guidelines for recommended versions per moodle installation and that is why we didn’t go over 7.3.

This is not our production system but plan to roll it out soon. We have had the new webserver up for a while and tasks were running fine with PHP 7.3 prior to April 16th which is when I upgraded from moodle 3.5 to 3.7. However if you think it’s worth going down to 7.2 we can give that a shot?

I will also try running the script via command line to see if that makes a difference. Thank you for the suggestions!