data request tasks failing

data request tasks failing

от Mark Kinoshita -
Количество ответов: 10

I have been having a number of issues caused (I think) by the the data privacy tool.

The symptom seems to be that cron tasks do not finish. 

Execute adhoc task: tool_dataprivacy\task\process_data_request_task continues to fail.

This has resulted in a number of odd issues I think. 

The data request task ends in the preprocessing stage:

Pre-processing request...
  Fetching data from 494 components (Friday, 11 October 2019, 3:28 PM)
    Processing antivirus_clamav (1/494) (Friday, 11 October 2019, 3:28 PM) ....
 
    Processing mod_journal (40/494) (Friday, 11 October 2019, 3:28 PM)
I have noticed that there are old journal entries for deleted users (for courses that were never reset, but are still be used).

The most major problem was that forum email notifications would not send. Also Data Requests never seem to finish processing. Thus I have users whose data requests are always "processing" or "approved" but never finish.

I have "solved" this by emptying (or deleting users) in the database directly in the table:  mdl_tool_dataprivacy_request. When I run Cron manually, I get an error that users cannot be found, and then after a minute or so (other scheduled task likely run), the manual cron runs without incident and the moodle seems to work just fine (no data requests processing, forum mail sends, etc...)

I am using moodle 3.7.2. The issue seems to have occurred when I upgraded to 3.7.1 or with the migration of our moodle from a VPS I operated to a hosted site.  Thus I am not sure whether the issue is a configuration/conflict issue in the new hosted environment or  a bug caused perhaps by either my privacy configuration or the number of assignments/journal entries which had been orphaned as users were removed in August.

I do recall having a similar "forum notifications not being sent" issue with database locks in the past - but that was fixed with $CFG->lock_factory = "\\core\\lock\\db_record_lock_factory"; in 2017.

Any help on this would be useful. My "fix" is likely not good for my moodle, but it has worked and no other suggestions in any posts have worked.

Theses are some of the related posts I have found:

Thanks,

Mark


Средняя оценка: -
В ответ на Mark Kinoshita

Re: data request tasks failing

от Mark Kinoshita -
Update - cron fails when Data requests are processing (if they are awaiting approval, cron runs)
Средняя оценка: -
В ответ на Mark Kinoshita

Re: data request tasks failing

от Mark Kinoshita -
I have manually removed all journal entries no longer associated with users and cron seems to run fine. Even with data requests processing.
So - the issue seems to be with journal entries (perhaps any assignments, etc) which have been orphaned (ie without a user in the database anymore).
Perhaps the deletion of the users did not remove these entries. Hard to say at this point.
Средняя оценка:Useful (1)
В ответ на Mark Kinoshita

Re: data request tasks failing

от Mark Kinoshita -
No luck, the database repairs itself and the data requests reappear. Also the same errors are reoccurring (cron stops on the journal entries, even with the old entries removed).
Thus, I will have to continue to delete the users in the data requests table in the database to send forum notification emails.
Next I will try to also delete the users from the user table and see what happens.
Средняя оценка: -
В ответ на Mark Kinoshita

Re: data request tasks failing

от Greg Wright -

We're seeing the same issue. Cron stops suddenly right after printing "Processing mod_journal" to the screen and no emails are going out. We are also on version 3.7.2. Is there any permanent solution yet?

Средняя оценка: -
В ответ на Greg Wright

Re: data request tasks failing

от Greg Wright -
I managed to get it to complete by modifying privacy/classes/manager.php to skip mod_journal. I added the following at line 236
if ($a->component == "mod_journal") {
print "Skipping mod_journal\n";
continue;
}
After that cron ran to completion. The first run took about 10 minutes and sent out a boatload of forum emails. Since then it completes in 3 seconds or less.

I took the modification out after a few runs and cron is still completing successfully. I know that not every task runs on each invocation of cron so the error could come back tomorrow. We'll see.
Средняя оценка:Useful (3)
В ответ на Greg Wright

Re: data request tasks failing

от Mark Kinoshita -
Amazing !
thanks for the tip.
This little fix works as you describe.
I'll have wait to tomorrow to see what happens with the rest of the cron tasks as well.

[Now data requests finishes - and the users are marked as deleted, forum post were also sent]
Средняя оценка: -
В ответ на Mark Kinoshita

Re: data request tasks failing

от Mark Kinoshita -

Update - June 2020.

Since May (with no major upgrades in between - I think journal was updated April 26 ) this no longer works.

Have now done a full upgrade to 3.8.3 and tried it again - no luck.

I have looked at Michael's link : https://github.com/elearningsoftware/moodle-mod_journal/issues/47, but the code has changed in the mod/journal/privacy/classes/provider.php since his post.

It looks like somethings is broken in the mod/journal - or I have some inconsistent data which kills it.

Any ideas? 

Thanks!

Средняя оценка: -
В ответ на Mark Kinoshita

Re: data request tasks failing

от Michael Hawkins -
Изображение пользователя Core developers Изображение пользователя Moodle HQ Изображение пользователя Particularly helpful Moodlers Изображение пользователя Peer reviewers Изображение пользователя Testers
Hi Mark,

Though that link did have a comment with a proposed solution (which sounds like it didn't work for you), it's actually an issue report for what you are seeing. That issue appears to still be open/active, so I don't think the issue has been fixed yet. It looks like a proposed solution was submitted in March, but hasn't been pulled in by the maintainers.
Средняя оценка: -