Creating large backup file is problem in course

Creating large backup file is problem in course

by praveen kota -
Number of replies: 17

Hi, 

I use moodle 3.9.1 on my server

whenever I try to create a backup of any of my course fully at a time as one single backup file  it never performs if that is more than an average size like 1GB or 1.5 GB(its works if it 300MG or 800MB not the exact size between any amount)

It starts creating it do some percentage 0% to 95% and stuck there only. like this below snippet


Then I do make parts of course from 1 to 4 topics then 5 to 9 similarly do entire course backup but is a time-consuming process.

Is there any way to create 1 single backup of a course no matter the size of the file (it should work for more than 1GB cause less than that I can create a backup file)?

I found this link but much understand 

https://docs.moodle.org/36/en/Course_backup

Course backup stops above 90%, not showing any errors

any help would be appreciated

with regards

Praveen Kota

Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
I would start by checking your server timeout settings in php.ini (extend them out and see if that helps) - max_execution_time is one that I normally extend out to about 360 seconds. There is a script timeout that you should check too in the same file. Also, check your file upload size and post max size and make sure they are set to large enough to cover the backup size.
Average of ratings: Useful (1)
In reply to Emma Richardson

Re: Creating large backup file is problem in course

by praveen kota -
Thank you @Emma Richardson for your quick response
Sorry for the late replay I haven't checked
As you suggested to " increase post_max size and server timeout " I will try this and let you know
Can you please tell me what time would you prefer set for server timeout and post_max size
one of my courses max size is like 3GB

thanks in advance
Average of ratings: -
In reply to Emma Richardson

Re: Creating large backup file is problem in course

by praveen kota -
Hi Emma,
I have tried to increase my server timeout(30) and also postmax size(260M) in php.ini
Both I have increased up to timeout(180) and postmax (3096M) But Not any effect on the backup process same problem.
Process:
Logged in Hostgator Cpanel -->php.ini--> Selected domain--> started changing-->saved and logged out.
Then login into Moodle started taking a single course backup but it stopped in mid of the process waited for 20Mins but not a moment in that so then again reloaded moodle started taking 2 topics, 3topics, multiple backup of a single course then those were processed and successes.

Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

So are you ready to try  the backup.php file in code/admin/cli/ yet?

'SoS', Ken

Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

Without knowing what's in the course ... knowing only that if you do backups selecting say section 1 -> 3 and no more, that works ... how about a different approach that might discover where the issue might lie?

Do a GUI backup and this time de-select all quizzes in the course ... when it comes to naming the backup insert -noquiz- in it's name.   If that works, make another backup and this time de-select all other resources in the course *except* quizzes and in the name of the backup insert '-quizonly-'.

Do both fail?  Does one succeed?  Which?

If one had debugging turned on while attempting above might get some more info.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Creating large backup file is problem in course

by praveen kota -
Thanks, Ken I will try this approach and let you know.
By the way, what would be the problem with the quiz or without the quiz while taking backup?
Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

Been my experience that when one has a 'large' (heavy processing) backup that times out during backup process, it was due to one of two things ... files (like an assignment all users in course had to turn in - such as a video) or many quizzes.   There is always the other ... an addon that has issues with backups ... but I think that one would raise it's ugly head when one split course backups ... and you've not reported issues with split backups.

One way to find out with split backups is to decompress downloaded backups that you've split and compare the size of questions.xml file in each un-compressed backup as well as the files.xml + files directories.  That involves commmand line on local computer and you don't appear to use the command line much ... if at all. :|

Could have an issue with the quiz bank ... which you could check prior to making backups with a hidden admin tool at:

https://yoursite/admin/tool/health/

It is a heavy processing tool so you have to be patient and let it run. 

After health runs, it presents you with issues that it finds and has a link for suggestions as to how to fix ... normally an sql query.   That query has fixed many issues in quiz banks for me - not all - but many.

But ... do the command line backup first before diving into anything else which just might confuse things as well as make things worse with the course as it is right now.

'SoS', Ken

Average of ratings: Useful (2)
In reply to Ken Task

Re: Creating large backup file is problem in course

by praveen kota -
Hi Ken
as you suggested to check the health of my site I did that
and I found these two issues in the quiz question bank
1. Question categories should belong to the same context as their parent
When one question category is the parent of another, then they should both belong to the same context. This is not true for the following categories

2. Question categories tree structure
The question categories should be arranged into tree structures by the question_categories.parent field. Sometimes this tree structure gets messed up.
Let me try to sort it out and get back to you
I think this is not an actual issue

Thank you for helping me Ken
Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

Earlier I did say the try backups using resources criteria ... ie, one backup with nothing but the quizzes contained in the course ... regardless of section the quiz is in.   One backup with all the assignments in the course.  Etc.   That will be just as time consuming as what you are doing now.

Unfortunately, we took a side road ... health check ... and you've determined that quizzes are not the problem in the backup of full course.

But, we don't seem to be doing anything different with the problem thus results won't be any different and we are no further down the road in finding the issue.

So you've tweaked php settings upwards ... not solved.

The fact you can do those tweaks does give a hint as to how hosted, but not enough information about hosting setup, memory, etc.

Have you looked at server logs .. web service error log ... DB error log or see anything in error log of web service related to DB ... like 'MySQL has gone away'.

And we have yet to actually try the cli backup script.   Why?

It sounds like a technical issue so I would think one would have to seek and share more technical info ... yep, talk techie even though it's not our 'lingo'.

'SoS', Ken

Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

Yeah, there is no admin tool to tell you when a course is getting to large to backup! :\

Have command line access to your server?   If so, there is a command line only backup.php script in admin/cli/ that might be able to work ... takes web service out of the loop and is just php talking to your DB and file system then.   Uses whatever the defaults are for backup preferences.

Then, however, if that does work, now have the other side of that coin ... can you restore that course backup?   No command line script to restore.

However, there is moosh ... which does have a command line restore.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Creating large backup file is problem in course

by praveen kota -
Thank you @Ken Task for your quick response
Sorry for the late replay I haven't checked
As said using CLI I can try but this is messing with the production server so is there any suggested code which you use.
Because I never tried CLI
thanks in advance
Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

After you try Emma's suggestion ... if still failing ... then the code to use is the command line script ... production server or not.

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Creating large backup file is problem in course

by praveen kota -
Yes, Ken, But I have a testing site that I can use to try the command line script but both my test and production sites are on the same server. So can I try?

thanks in advance
Average of ratings: -
In reply to praveen kota

Re: Creating large backup file is problem in course

by Ken Task -
Picture of Particularly helpful Moodlers

As long as the data directories for production and dev are unique ... separate (same for DB's and code), yes, you can run the CLI only backup.php script.

Takes web service out of the loop ... and it will report errors as well ... which will give you some clues.

Just make sure you give a destination directory outside of sea of files in moodledata/filedir/

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Creating large backup file is problem in course

by praveen kota -
Thank you, Ken.
I will try and get back with the outcome.
Average of ratings: -
In reply to Ken Task

Re: Creating large backup file is problem in course

by Helen Foster -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hello,

Just noting that in Moodle 3.10 there is also a CLI script for restoring large backup files - MDL-69307.

Average of ratings: Useful (1)
In reply to Helen Foster

Re: Creating large backup file is problem in course

by praveen kota -
Thank you for sharing with us Helen.
Average of ratings: -