Hi,
We have been doing a bit of cleanup on our Moodle platform recently and noticed course backup, restoration and especially deletions are extremely slow on our server. Course deletions sometimes lead to server crashes which is concerning.
The backups of the courses range from 1.5 - 2 GB, however to reduce the course deletion time we try to delete everything (i.e. content & quizzes) in the course prior to deleting the course in hopes to reduce the size but that doesn't really help.
We have noticed timeout 300 errors when deleting the courses.
How have you guys handled course deletions of especially large courses has it been especially slow?
Note we handle multiple moodle instances on our server, but it should be robust enough to handle these tasks.
Any assistance is appreciated!
For Deletions: check Recyclebin settings. Set time to retain for 1 second.
That will hurry along deletions to moodledata/trashdir/ where you can delete the contents manually if you are getting into a 'space' crunch.
Just make absolutely sure the course you delete is one you truely want deleted. Once trashdir emptied, no way to recover cept from a course backup restore. No course backup ... nothing to restore!
'SoS', Ken
For Reusing (reset) courses ...
moosh a cli only utility knife
https://moosh-online.com/
https://moodle.org/plugins/view.php?id=522
has a command:
https://moosh-online.com/commands/
course-reset
Reset course by ID. With -s or --settings option you can provide any supported setting for the restore. The value for -s option is a string with each setting in format key=value separated by space. This means you'll need to quote this string with double or single quotes when running the command. unenrol_users setting requires an array as a value - put at least one comma character (,) as a value to make moosh convert that into an array. Example:
moosh course-reset -s "reset_forum_all=1 reset_data=0 unenrol_users=6," 17
Add -n or --no-action to display all reset setting that would be used on course restore. When -n is used, no action is actually performed.
All posible backup settings I've found in Moodle 2.9 are listed below. Most of them is set to eiher 0 or 1.
delete_blog_associations
reset_assign_submissions
reset_chat
reset_choice
reset_comments
reset_completion
reset_data
reset_data_comments
reset_data_notenrolled
reset_data_ratings
reset_events
reset_forum_all
reset_forum_digests
reset_forum_ratings
reset_forum_subscriptions
reset_forum_track_prefs
reset_forum_types
reset_glossary_all
reset_glossary_comments
reset_glossary_notenrolled
reset_glossary_ratings
reset_glossary_types
reset_gradebook_grades
reset_gradebook_items
reset_groupings_members
reset_groupings_remove
reset_groups_members
reset_groups_remove
reset_lesson
reset_lesson_group_overrides
reset_lesson_user_overrides
reset_notes
reset_quiz_attempts
reset_quiz_group_overrides
reset_quiz_user_overrides
reset_roles_local
reset_roles_overrides
reset_scorm
reset_start_date (timestamp as a value)
reset_survey_analysis
reset_survey_answers
reset_wiki_comments
reset_wiki_tags
reset_workshop_assessments
reset_workshop_phase
reset_workshop_submissions
unenrol_users (array of user roles to unenrol)
Example 1: Reset course with id=17 using default settings.
moosh course-reset 17
Example 2: Show default settings when resetting course id=17
moosh course-reset -n 17
Example 3: Set unenrolment of participants with role id 5 and 6, and reset course with id=17
moosh course-reset -s "unenrol_users=5,6" 17
And a comment: with multiple instances on same server, command line is the way to go - period! Too slow and time consuming with GUI admin.
CLI would involve only the bash shell scripts running php cli only commands talking to DB. Web services taken out of the loop.
'SoS', Ken
moosh a cli only utility knife
https://moosh-online.com/
https://moodle.org/plugins/view.php?id=522
has a command:
https://moosh-online.com/commands/
course-reset
Reset course by ID. With -s or --settings option you can provide any supported setting for the restore. The value for -s option is a string with each setting in format key=value separated by space. This means you'll need to quote this string with double or single quotes when running the command. unenrol_users setting requires an array as a value - put at least one comma character (,) as a value to make moosh convert that into an array. Example:
moosh course-reset -s "reset_forum_all=1 reset_data=0 unenrol_users=6," 17
Add -n or --no-action to display all reset setting that would be used on course restore. When -n is used, no action is actually performed.
All posible backup settings I've found in Moodle 2.9 are listed below. Most of them is set to eiher 0 or 1.
delete_blog_associations
reset_assign_submissions
reset_chat
reset_choice
reset_comments
reset_completion
reset_data
reset_data_comments
reset_data_notenrolled
reset_data_ratings
reset_events
reset_forum_all
reset_forum_digests
reset_forum_ratings
reset_forum_subscriptions
reset_forum_track_prefs
reset_forum_types
reset_glossary_all
reset_glossary_comments
reset_glossary_notenrolled
reset_glossary_ratings
reset_glossary_types
reset_gradebook_grades
reset_gradebook_items
reset_groupings_members
reset_groupings_remove
reset_groups_members
reset_groups_remove
reset_lesson
reset_lesson_group_overrides
reset_lesson_user_overrides
reset_notes
reset_quiz_attempts
reset_quiz_group_overrides
reset_quiz_user_overrides
reset_roles_local
reset_roles_overrides
reset_scorm
reset_start_date (timestamp as a value)
reset_survey_analysis
reset_survey_answers
reset_wiki_comments
reset_wiki_tags
reset_workshop_assessments
reset_workshop_phase
reset_workshop_submissions
unenrol_users (array of user roles to unenrol)
Example 1: Reset course with id=17 using default settings.
moosh course-reset 17
Example 2: Show default settings when resetting course id=17
moosh course-reset -n 17
Example 3: Set unenrolment of participants with role id 5 and 6, and reset course with id=17
moosh course-reset -s "unenrol_users=5,6" 17
And a comment: with multiple instances on same server, command line is the way to go - period! Too slow and time consuming with GUI admin.
CLI would involve only the bash shell scripts running php cli only commands talking to DB. Web services taken out of the loop.
'SoS', Ken
Thanks Ken!
I'll have a look a the Moosh plugin. Would we be able to directly delete courses using Moosh via command line and would this be faster?
Also thanks for the recycling bin tip, but I don't think its a storage issue we are running into, I think its an issue with the SQL query times been exceptionally large for the course deletion process which is leading to a slow and cumbersome process.
I'll have a look a the Moosh plugin. Would we be able to directly delete courses using Moosh via command line and would this be faster?
Also thanks for the recycling bin tip, but I don't think its a storage issue we are running into, I think its an issue with the SQL query times been exceptionally large for the course deletion process which is leading to a slow and cumbersome process.
Think the bottom line is this ... CLI scripts are faster than GUI scripts - period.
If you have only one bash looping script to delete courses it's only the one script hitting DB server ... one course at a time - which, I would think, be better than 10 persons with admin rights hitting the web server first to delete courses.
Courses consist of multiple modules (activities/resources + quizzes) which would require query of DB for multiple tables - just to extract those that need to be backed up (recyclebin) and then deleted.
I don't know of any way to speed up the basic processes for what you are doing ... cept maintenance mode and CLI scripts.
Let's not forget your cron job is still running if you do NOT put site in maintenance mode, also.
Depending upon size of course and number of them had to wrap the looping bash shell script to delete courses in a 'nohup' command.
A lot also depends upon how healthy the DB's are. I admin one K12 school's moodle and we have massive 'clean ups' at end of every academic year. One of the things I do in prep for massive stuff is to run MySQLTuner and fix any tables that need optimizing, and any tweaks for maximizing memory consumed by massive actions.
If one can 'fit' most of the DB in memory as opposed to disk, fewer IO's to disk ... faster it is.
'SoS', Ken
Follow up ... through some non-evasive sluething, found one of your Moodle servers - it's behind ClouldFlare (CF). So 'slowness' (even though we might be talking milliseconds) might actually be due to coming in front door - internet -> CF -> Server. Advantage CLI ... no CF involved there!
'SoS', Ken
Hi Ken,
Thanks for all your help, appreciate it! We have discovered some of our local plugins were triggering when we were doing course deletions and restorations i.e. event observers for user_created and user_deleted causing an overload of SQL queries to our database during these processes. We have since disabled these plugins for time being while we cleanup our database which has significantly reduced the course deletion time by large margin.
Appreciate your prompt assistance, however!
Thanks for all your help, appreciate it! We have discovered some of our local plugins were triggering when we were doing course deletions and restorations i.e. event observers for user_created and user_deleted causing an overload of SQL queries to our database during these processes. We have since disabled these plugins for time being while we cleanup our database which has significantly reduced the course deletion time by large margin.
Appreciate your prompt assistance, however!