Rebooted server - Cron jobs not running

Rebooted server - Cron jobs not running

by eric w -
Number of replies: 11

Hello all,

 I am a new admin, running Moodle 3.1 on Ubuntu 12 using LAMP.  On Feb. 13th, I had to shutdown the server for a brief period of time. When I started it back up, everything seemed fine. No apparent problems.


PROBLEM:

Recently, I was informed that instructors were not receiving email notifications for News Forums posts. My research leads  me to believe that there is an issue with cron where the jobs which send these emails are not running.  
Furthermore, I noticed that my weekly backup did not run on February 17th (which was after the shutdown event). Again, this leads me to believe that there is an issue with cron, and that it is related to the shutdown event.


I am super busy, and struggling with deadlines, and so I am hoping that someone can point me in the right direction or provides some insight on how to fix cron so that these jobs run. 


Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by Ken Task -
Picture of Particularly helpful Moodlers

Because you say moodle's cron job + other cron jobs are not running, shall assume crond (which is a daemon/service) wasn't set to startup on reboot:

https://www.cyberciti.biz/faq/howto-linux-unix-start-restart-cron/

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Rebooted server - Cron jobs not running

by eric w -

Thank you both for your responses smile


I ran the command to start the cron daemon, but before I did that, I checked the status of crond by using the following command:


/etc/init.d/cron status


I have posted the results of the status command.  It looks as though it was running, but maybe it just needed to be restarted?   I'll check with some of the instructors to see if they are receiving the emails now.


Attachment cron status output.PNG
Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by eric w -

I had a look at  'Administration > Site administration > Server > Scheduled tasks'    and I do notice that there are several tasks that have run after the reboot, as shown in the screenshot below.  This suggests that the cron daemon is running, but maybe it's only running certain jobs.  My question would then be:  why aren't the weekly backups running, and why aren't the news forum emailer jobs running?   What is unique about these jobs that may be causing a problem?


Thanks again.



EDIT:  As I look at this screenshot, I do notice that it says the 'Automatic Backups' ran on Tuesday which is AFTER the reboot.  Although, at this point I'm not sure if this 'Automatic Backups' is my weekly backup which is supposed to run an Friday nights.

Attachment Capture.PNG
Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by Ken Task -
Picture of Particularly helpful Moodlers

Confused about what you are calling 'weekly backups' ... is that autobackups scheduled in Moodle config or is that some other cron job you have in crontab.

If autobackups turned on ... do you have the backups going to 'file area' ... which is that sea of files in /moodledata/filedir/ (ugh ... no humanly cognizable names and can be found only via Moodle UI ... no admin tool to see them all).   OR do you have autobackups going to a designated directory outside of moodledata ... like /home/backups/ Can see the files by humanly re-cognizable names ... and their sizes ... if in a space crunch can remove them easily. ;)

Is autobackups completing the backup of all courses?

Check /moodledata/temp/backups/   See any .log files that are larger than 0 bytes?   See any directories that are contenthash named ... lots of letters/numbers? Those could be in the process or they could be failed autobackups ... which is yet another posting.

I see in the image autobackups set to run again at a date past.

When the autobackup gets confused, the table that tracks completion, when to run next, etc. basically gets out of sync with things and the only way to get them back in order/working is to truncate the related tables ... then wait for the next scheduled backup .... or force a automated back up to run via moodlecode/admin/cli/autobackup.php

Do the latter only if you have time to watch it.  Obviously, killing script will once again mess with the tables tracking.  Could let it run overnight ... but then again, your shell access might time out and kill the script ... depending upon how your system behaves with loosing ssh connection and how you issued the php script command.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Rebooted server - Cron jobs not running

by eric w -

Thank you for your response smile   Please find answers below:


Q: Confused about what you are calling 'weekly backups' ... is that autobackups scheduled in Moodle config or is that some other cron job you have in crontab.

ANSWER:  Yes, refering to a crontab job.


Q: If autobackups turned on ... do you have the backups going to 'file area' ... which is that sea of files in /moodledata/filedir/ (ugh ... no humanly cognizable names and can be found only via Moodle UI ... no admin tool to see them all).   OR do you have autobackups going to a designated directory outside of moodledata ... like /home/backups/ Can see the files by humanly re-cognizable names

ANSWER: You are correct about /moodledata/filedir. The backups are there as you described. I believe the crontab weekly job takes these files, and moves them to our fileserver.


Q: Is autobackups completing the backup of all courses?

ANSWER: Yes. 


Q: Check /moodledata/temp/backups/   See any .log files that are larger than 0 bytes?   See any directories that are contenthash named ... lots of letters/numbers? Those could be in the process or they could be failed autobackups ... which is yet another posting.

ANSWER: There are no directories in /moodledata/temp/backup.  There are several log files as you described, and they all have '0' for the size value.

 P.S. I posted an image below of what's in /etc/crontab.

Attachment crontab.png
Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by Ken Task -
Picture of Particularly helpful Moodlers

hmmmm .... follow up ...

in the task schedule there is automated backups which supposedly ran at 12:00 midnight ... and the one below that Clean backup tables and logs set for 15 minutes later.

Uhhhh .... courses only grow most of the time ... they don't get smaller ... that translates into taking longer to backup.   If you were to run the automated backups script from the command line one time, how long did it take?   Adjust the clean up.

Currently, might have an issue with one task romping on the other before completed.

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Rebooted server - Cron jobs not running

by eric w -

Bumping this thread as I still have no clue what the problem is.   The cron daemon appears to be running correctly, the question remains as to why certain tasks are not running.


Any further suggestions would be greatly appreciated. 

Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by eric w -

I belive that I have tracked this down to being an issue with cron.php.   I have posted a screen cap of the results I recieve when  I run the following command : 

/usr/bin/php /var/www/moodle/admin/cli/cron.php


Attachment cron error;lkdd;lska;sdkf;s .PNG
Average of ratings: -
In reply to eric w

Re: Rebooted server - Cron jobs not running

by Ken Task -
Picture of Particularly helpful Moodlers

Guess the question is ... how did that happen?

Try this ... use something other than php ...

Assuming you have curl installed ...

curl http://site/admin/cron.php [ENTER]

In Moodle config there is a setting for running cron.php via web with a password.   Check that setting and use a password.

curl http://site/admin/cron.php?password=[yourpassword]

See if that works. ;)

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Rebooted server - Cron jobs not running [SOLVED]

by eric w -

Thanks for all your help Ken, smile


The problem was with one of the jobs in cron.php, it was failing (for reasons I wont share) and was holding up the rest of the process.  Once I fixed the issue causing the job to fail, I was able to successfully execute cron.php  which took off like a race horse, and is still running!!!


Glad that problem is fixed.  Now on to the next one (non-moodle).  ;)

Average of ratings: -