Grade book background processing fails after migration

Grade book background processing fails after migration

by Steve Morrison -
Number of replies: 15

The last of the challenges to attack after the migration. The documentation helped immensely. Thanks to those who put it together.

Since the move The background gradebook processing cron task fails. Any insight on locks? and what I need to do here?

     Thanks in advance!! Steve

ERROR:

 Execute scheduled task: Background processing for gradebook (core\task\grade_cron_task)
... started 16:00:03. Current memory use 18.4MB.
!!! Coding error detected, it must be fixed by a programmer: A lock was created but not released at:
[dirroot]/lib/cronlib.php on line 99

 Code should look like:

 $factory = \core\lock\lock_config::get_lock_factory('type');
 $lock = $factory->get_lock(Resource id #550);
 $lock->release();  // Locks must ALWAYS be released like this.

 !!!

CRON lines 97 to 100>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  $scheduledlock = null;
    for ($run = 0; $run < $maxruns; $run++) {
        if ($scheduledlock = $cronlockfactory->get_lock("scheduled_task_runner_{$run}", 1)) {
            break;

Average of ratings: -

Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers

See what's in mdl_lock_db.

Also in moodledata/locks/ - are owserships/permissions ok?

cron job running how often and by what user?

'SoS', Ken

Average of ratings: Useful (1)

Re: Re: Grade book background processing fails after migration

by Steve Morrison -
I have made the mistake that I have a pet peeve over. My apologies. Moodle 3.7 php 7.3. Cron running every 1 minute. Had it at every 5. Still came problem. Old server was running cron every 15 minutes. What should the moodledata/locks permissions be? The permissions and ownership hasn't been changed that I know of. Mdl_lock_db???? Would you be as kind as to enlighten me? Thanks in advance, Steve

Re: Re: Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers
Had something similar one time on an upgrade only different task reported in the error ... don't re-call which now.

moodledata/lock ... should be readable and writable by web service user.   If you run your cron job as root user (I do sometimes from command line run php admin/cli/cron.php) and see that some files in moodledata/lock/ are tagged as belonging to root user.  So i have moodledata/lock/ readable/writable by all.

Since you've been having issues in upgrading ... am guessing that there were times when maintenance mode was off ... a task could run ... although may not finish ... so lock created but for some reason could not be changed/removed/unlocked in a cron/task run.

The 'mdl_lock_db' is a table in the database.  My table has no data in it so am thinking that table might be acting as a scheduler of some sort for crons/task to check to see what's locked and what's not. 

mysql> explain mdl_lock_db;
+-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| id          | bigint(10)   | NO   | PRI | NULL    | auto_increment |
| resourcekey | varchar(255) | NO   | UNI |         |                |
| expires     | bigint(10)   | YES  | MUL | NULL    |                |
| owner       | varchar(36)  | YES  | MUL | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+

So, what I re-call doing ... site in maintenance mode (and to assure nothing web based could get in un-seen, actually shutdown apache), truncated the data in the table mdl_lock_db, manually removed contents of moodledata/lock/ (rm -fR * in that location)  Restarted apache.  Site out of maintenance mode.

Ran php admin/cli/cron.php several times ... no errors

The also checked the task related to my problem ... yours might be ... \core\task\grade_cron_task and

\core\task\grade_history_cleanup_task

in moodlecode/admin/tool/task/cli/

Good luck!

'SoS', Ken


Average of ratings: Useful (1)

Re: Re: Re: Re: Grade book background processing fails after migration

by Steve Morrison -
database looks just like yours unless I am missing something. Attached a screenshot from MySQL in the word doc.

Background processing for gradebook
\core\task\grade_cron_task
Scheduled1/06/19, 09:380.03 secs
3 reads
0 writes
Fail 
Background processing for gradebook
\core\task\grade_cron_task
Scheduled1/06/19, 09:370.03 secs
3 reads
0 writes
Fail 
Background processing for gradebook
\core\task\grade_cron_task
Scheduled1/06/19, 09:360.02 secs
3 reads
0 writes
Fail 

Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers

First, suggest Moderator remove .docx attachment.  Might be exposing too much info.

Tip: many folks won't open M$ anything if shared here in forums.

Screen captures as images paying attention to blacking out sensitive info better.

Best is text only!!!

Ok, your table has the same structure as mine ... and as every 3.6 user in the globe.   It contained no data .. right?

I see that that table does have utf8mb4 character set?  And collation of utf8mb4_general_ci.

The error is failing to read/or write to that table via task.

In your config.php file for DB section ... is there a DB collation line?

That line must match all the tables in the DB ... and all the columns in those tables.

In moodlecode/admin/cli/

run

php mysql_collation.php -l

Summary at the bottom ...

utf8mb4_general_ci: 1494

Also, know the super user creds to the DB for the Moodle?

Could change the config.php file DB user, DB password to super user

Then try one of the failing task.

'SoS', Ken

Average of ratings: Useful (1)

Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Steve Morrison -
Your patience and knowledge is certainly appreciated.
I never thought twice about the info in the header of the screenshot. Thanks for catching that.

MySQL returned an empty result set (i.e. zero rows). (Query took 0.0003 seconds.)

In your config.php file for DB section ... is there a DB collation line? 'dbcollation' => 'utf8mb4_unicode_ci'
Table collations summary for ***************.com:
utf8mb4_unicode_ci: 1617

Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Steve Morrison -
Also, know the super user creds to the DB for the Moodle? (as far as I know there is only one sql password)
Could change the config.php file DB user, DB password to super user would this be the root p/w?

Thanks again!! Steve

Re: Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers

Superuser Creds for DB root?
Most setups recommend not using root (ssh access) logins/passwords for DB setup.   But who knows ... I can't guess that remotely ...
no vulcan mind meld possible here!

In the Background processing for gradebook \core\task\grade_cron_task
you show in https://moodle.org/mod/forum/discuss.php?d=386884#p1559737 posting.
That has a scheduled date of Jan 6 2019 or is that 1st of May?

in moodlecode/admin/tool/cli/

try running:

php schedule_task.php --execute="\core\task\grade_cron_task"

and

php schedule_task.php --execute="\core\task\grade_history_cleanup_task"

Those same task are part of the overall cron job.

If either of those error, errors will show text .. copy text ... paste to NotePad.   Come back here and post the text.

'SoS', Ken


Average of ratings: Useful (1)

Re: Re: Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Steve Morrison -
I can't possibly convey enough appreciation for your spoonfeeding here.
we have some info:
had to use the path: moodlecode/admin/tool/task/cli

here's what shows:

Execute scheduled task: Background processing for gradebook (core\task\grade_cron_task)
!!! Exception - Class 'grade_grade' not found !!!

Execute scheduled task: Background processing for cleaning grade history tables (core\task\grade_history_cleanup_task)
Deleted old grade history records from 'grade_outcomes_history'
Deleted old grade history records from 'grade_categories_history'
Deleted old grade history records from 'grade_items_history'
Deleted old grade history records from 'grade_grades_history'
Deleted old grade history records from 'scale_history'
... used 5 dbqueries
... used 0.15094304084778 seconds
Scheduled task complete: Background processing for cleaning grade history tables (core\task\grade_history_cleanup_task)

Re: Re: Re: Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers

Well, it's some info ... but since am not a true programmer, what to do with the info???!!!???

Don't understand this:

"had to use the path: moodlecode/admin/tool/task/cli"

I normally do cd /path/to/moodlecode/admin/tool/task/cli/

Then execute like:

php schedule_task.php --execute="\core\task\grade_cron_task"

and

php schedule_task.php --execute="\core\task\grade_history_cleanup_task"

So the first command reported:

"Exception - Class 'grade_grade' not found"

but the second completed ok?

Run the first one again now that the second completed:

Mine shows - yours should be something similar:

php schedule_task.php --execute="\core\task\grade_cron_task"
Execute scheduled task: Background processing for gradebook (core\task\grade_cron_task)
... used 2 dbqueries
... used 0.027482032775879 seconds
Scheduled task complete: Background processing for gradebook (core\task\grade_cron_task)

Tried to find something in tracker for bug reporting concerning the 'Exception - Class 'grade_grade' not found ' but didn't find anything.

We might have to wait for someone with a lot more knowledge about such things than me!   We've made progress, but ....

'SoS', Ken

Average of ratings: Useful (2)

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Steve Morrison -
It took me a few minutes to figure out that I wasn't getting anywhere "cd" ing /admin/tool/cli/ there needed to be a task in between. the task logs for this task shows 3 database reads and 0 writes, I wonder if it is possibly completing. It appears that there is a redirect with lib/grade/grade_grade that redirects an sql request for grade_grade to grade_grades. not sure that I am correct on that, but a wild guess. kind of wondering if during the migration there might have been some database conversion snafu's. I've disabled that task for now until it gets sorted out. sure wish the Vulcan mind meld worked. it would explain a lot of life's mysteries.

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Grade book background processing fails after migration

by Ken Task -
Picture of Particularly helpful Moodlers

Well ... migrations require getting ducks in a row and attempting to make sure they go smoothly ... ie, without error.   You had some issues however, and possibly that lead to the first issue of locks ... in and out of maintenance mode ... cron jobs sneaking in between and attempting to do something, etc. etc.   Only a vulcan mind meld historical record might determine, me thinks.

Am concerned that turning off that task might cause issues.

Will send you a PM with an offer you probably shouldn't turn down! smile

'SoS', Ken

Average of ratings: Useful (2)

Re: Grade book background processing fails after migration

by Steve Morrison -
Ken, you are an absolute savior!! I can't properly express my gratitude for your help and wisdom. For those following this: I moved from a shared hosting plan to a vps with the same hosting provider. the migration appeared to work...but....that's where the problems began. the migration resulted in Moodle being in a different root directory. after sorting out the directory issues the cron had several failing items. This was due to multiple scripts being used on the database. Ken was kind enough to talk me through creating a new database, importing data from a known good database, and pointing Moodle at that database. so far so good. all cron tasks have run properly for the past little while. I can't say it enough, Ken, thank you very much

Re: Grade book background processing fails after migration

by Erik Frangež -

Hello,

we are in the migration phase from Moodle version 3.5.1 to 3.7.1+.

Out system environment is:

  • virtual machine (on vmwere)
  • Debian 9, x64
  • MariaDB 15.1
  • Apache2 2.4.25
  • PHP 7.3 (just upgraded from 7.0.1)
Our problem is that cronjob is not running well. If we check output of cron we get (but not always same error):
Execute scheduled task: Obdelava v ozadju za predpomnilnike (core\task\cache_cron_task)
... started 12:08:20. Current memory use 14,9MB.
Cleaning up stale session data from cache stores.
... used 0 dbqueries
... used 0,0043849945068359 seconds
Scheduled task complete: Obdelava v ozadju za predpomnilnike (core\task\cache_cron_task)
... used 1 dbqueries
... used 0,0054318904876709 seconds
Scheduled task failed: Obdelava v ozadju za predpomnilnike (core\task\cache_cron_task),Napaka pri pisanju v podatkovno bazo
Debug info:
Column count doesn't match value count at row 1
INSERT INTO mdl_task_log (type,component,classname,userid,timestart,timeend,dbreads,dbwrites,result,output) VALUES(?,?,?,?,?,?,?,?,?,?)
[array (
  0 => 0,
  1 => 'moodle',
  2 => 'core\\task\\cache_cron_task',
  3 => 0,
  4 => 1563530900.1721661,
  5 => 1563530900.1818039,
  6 => 0,
  7 => 0,
  8 => 0,
  9 => 'Execute scheduled task: Obdelava v ozadju za predpomnilnike (core\\task\\cache_cron_task)
... started 12:08:20. Current memory use 14,9MB.

In Moodle, in section Scheduled task, we see that out jobs are not running well:

Example below show that "Fail delay" is increasing till 24hours and than stops - we think that cron job is not doing his job:
Name Component Edit Logs Last run Next run Minute Hour Day Day of week Month Fail delay Default
Prepare submissions for annotation \assignfeedback_editpdf\task\convert_submissions Annotate PDF

petek, 19 julij 2019, 12:05 ASAP */15 * * * * 60 No

Permission of cron.php are (777 in linux, owner is root):
-rwxrwxrwx  1 root root 1,8K jul 18 08:26 cron.php

Data in ../moodledata/lock/ are static. We have deleted manually content of folder but situation is the same. Also we have change collation of database with command php mysql_collation.php  --collation=utf8mb4_slovenian_ci, still nothing.

Please help us what can we do next?

Best regards, Erik Frangež