Back-Up File Too Large

Back-Up File Too Large

by josh love -
Number of replies: 22

I have a course that I am trying to back-up/restore, but whenever I back-up the course, the file size is coming out to be a LOT larger then it should. It is currently coming out to be 28.9MB. 

I figured maybe user data might be causing this, so I reset the course, deleting all user data, but this did not change the file size at all (I wasn't backing up user data anyway, but thought it would be worth a shot). 

I then tested this by backing up the course, but only included a couple of labels in the back-up. The file size then came out  to be 27.8MB. This shows me that I really only have 1.1MB of content. That means there is 27.8MB of some kind of baggage that is coming along with each back-up.

Can anyone help me with this? I can't restore a file this large because the server times out every time I do (I know there are fixes for the server side of things, but my course shouldn't be anywhere near this large in the first place). 

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Version of Moodle?  (makes a difference in where to look and suggestions).

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

Hey Ken,

We are running Moodle 2.5.3.

As you can see in a post further down, we were able to locate these 'hidden' files and it dropped my course size down to 1.2MB. 

But now I have a new problem that has arisen. Whenever I try to back up the course now, nothing happens. It takes me to a new page, but nothing loads on it. I figured since I tried to restore such large files so many times, I should clear the 'cache' via Site Admin->Develpmental->Purge All Caches. This did not solve the problem.

Also, yesterday, we added the following lines to an .htaccess file to see if we could restore the course as it was. 

php_value upload_max_filesize 128M
 php_value post_max_size 128M

After adding this, we were still not  able to back-up the large file size. Think this would cause the problem? 

If not, have any ideas?

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Sounds like debugging needs to be turned on.  Then try whatever process.

When files are removed from the new file system they are actually tagged as 'trash' and remain in /moodledata/trashdir for the default of 4 days.   Only the cron job empties the trash.   Run cron from browser more than once to see if there are any references (towards the bottom of cron output) for emptying the trash ... ie, actually deleting those files.

Also check your logs ... apache error logs and mysql log if ya have access to it (or if your mysql is actually logging).

Check your moodledata/temp/backup/ directory.  Moodle uses that directory for course restores and it's where .mbz files get 'unzipped'.   You might find files of 0 bytes ... with long names:

20208a0a786fe22ab01a2ebb1cc99cbf.log

that's normal and indicates successful whatever.

However, you might also see directories or files with long names like:

d135674680f61bc70d6c627a117d7be8

Those are either backups/restores in progress or failed backups/restores

If it's a file: file -b d135674680f61bc70d6c627a117d7be8 will show the file type.

Zip archive data, at least v2.0 to extract

it's a zip!!!

unzip -l d135674680f61bc70d6c627a117d7be8

will list the contents.

unzip -j 'd135674680f61bc70d6c627a117d7be8' 'moodle_backup.xml' -d './'

will extract just the moodle_backup.xml file which is a 'map' of the contents of the backup.

head -n 10 moodle_backup.xml

will show the first 10 lines of that xml file and give info that might be useful.

In restoring 2 backups have found that max_packet_allowed had to be increased (500M?) as well as open_files_limit (6000?) in MySQL.   Those (values) a guess.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

Hey Ken,

I ran the cron job a couple of times. The trash folder has only one file in it at the moment (it is the 29MB back-up file I deleted).

I don't have access to the server to check the error logs. 

I checked the moodledata/temp/backup folder, and there are a bunch of files there. It seems the back-up I tried to do of the smaller version of the course is there 5 different times (they are all 1.2MB). There is also a couple of .log files, and a bunch of folders.

I am accessing this through an SFTP, can I just delete these files? If so, do I delete everything, or just the large files?

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Check the contents of the .log files if they are larger than 0 bytes.   Could give a clue.

The other files/folders one can delete manually - safe to do now as those appear to be failed attempts.  However, the next time you do a backup/restore, check that same folder to see what is there.  Moodle should 'clean up' that area leaving only 0 byte .log files IF backups/restores were successful.  Anything else that remains should probably be inspected with whatever tools you have to use.  Can't tell you what to look for specifically.  Do compare .xml files and their contents with the related folders ... example: files.xml with 'files' folder contents.

'spirit of sharing', Ken

 

Average of ratings: -
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

Hey Ken,

I cleared out the failed back-ups, but I am still not able to restore the course. I am however able to restore other courses. 

I have looked through the moodle_backup.xml file, but that hasn't helped me out.

You did mention in an earlier post about max_packet_limit and open_files_limit. This course has a lot of activities in it. Whenever I open the 'activities' folder within the back-up, there are 400 folders, each with 6-7 files in it. 

Think this may have something to do with it?

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by josh love -

Also, I went through and cross-checked the files.xml with the files folder. I was able to locate two <files> in the .xml sheet that wasn't in the files folder. 

Can I go into moodledata and edit this on the backup file? Or do I edit the file I have unzipped on my computer, re-compress it and turn it back into a .mbz? If I do need to turn it back into a .mbz, what is the best way of doing that?

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Ok, you've changed the extension from .mbz (which stands for moodle backup zip) to .zip and unzipped.  Right?

One could remove the references in the .xml file and also make sure there are no references to the content hashed folders in the course_files folder.

Once it's corrected, re-zip.   The moodle_backup.xml and other xml's must be at the root of the zip file tree.

Let's say you un-zipped and made edits/corrected in a 'test' folder.   To make a valid zip one must use the zip command while in the 'test' folder.   Those .xml's will then be at the root of the archive when un-zipped during the restore process.

After zipping, change the extension from .zip to .mbz and then use the .mbz to restore.

You just reverse the process you used to inspect the .mbz backup.

Folder for the edited moodle_backup.xml etc. is 'test' ...

cd test

zip -r * somebackupfilename.mbz

-r does recursive ... gets all files folders in the 'test' folder.  The extension .mbz tells the zip command to save the file name that way.  It's still a zip.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

I was able to edit the changes in the xml and rezip it as a .mbz. I switched around the command line a bit. I used:

zip -r somebackupfile.mbz *

The one you gave me was only zipping the first file in my test folder.

I tried using this file to restore, but still no luck. But I'm wondering if it worked correctly, because whenever I look at the folder created in the moodledata->backup, it only shows the first file in that test folder (but if I unzip the file on my computer, it shows all the files). 

But if this isn't the problem, then I will follow your advice from the next post, which is to dive into the error logs.

So first off, I turned on debugging, but I have no idea where to check the logs once I turned it on. Could you tell me where they would be?

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Oops ... sorry bout that ... giving you in-correct command. :\

In moodledata/temp/backup/ there might be some clues.  Moodle uses that directory to unzip into a hashnamed folder consisting of letters/numbers.  Inside that folder are the un-zipped contents.  Is there a moodle.xml (if a 1.9 course backup) or a moodle_backup.xml (if a 2 backup) present at that level.   That one xml file is like a roadmap as to what to restore.   There will be a log file that is larger than 0 bytes.  Can open it in a text editor and inspect what it says/shows.   Clues could be found in there as well.

Turning on debug will report errors to the screen.   However, one might find same references in logs ... if your server is logging errors.

Do you have command line shell access to your server?   If so, you could upload the .mbz to a test directory (outside of Moodle data) and via commands test your server ability to unzip.

In test folder - file is somebackup.mbz (the file you uploaded):

mv somebackup.mbz somebackup.zip

unzip somebackup.zip

You should see the files/structure (ie, the folder names, etc..) replicated as they existed on your local machine's 'test' folder.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

Hey Ken, 

First off, thanks for all of your help in this matter. It's been a long process.

I checked the log in the temp/backup/randomlettersandnumbers folder and it doesn't tell me anything. I took a screenshot of it if you want to look at it. Just let me know.

 

I turned on the debugging in browser, and I also combed through the apache logs, and I found the following errors:

In browser:

  1. Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/vhosts/ihopuonline.ihopkc.org/lms/cache/stores/file/lib.php on line 370

In Apache:

  1. PHP Fatal error:  Maximum execution time of 30 seconds exceeded in filepath/lms/lib/moodlelib.php on line 782, referer: sitename/lms/backup/restorefile.php?contextid=41060
  2. PHP Fatal error:  Maximum execution time of 30 seconds exceeded in filepath/lms/lib/adminlib.php on line 1541, referer: sitename/lms/backup/restorefile.php?contextid=41060
Apparently it is timing out which is stopping the back-up. I have a feeling it may be looking for a file that isn't there and is getting tripped up (like I said, when I download the course, I found in the file.xml references to 6 files, but only 3 files were in the files folder). Don't have command line access to server (just my ftp).
 
So what would be the next best step?
Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

You've found your problem ...

"Fatal error: Maximum execution time of 30 seconds exceeded"

One must increase max_execution_time in the configuration of php (php.ini)

I see by information shared in a previous post it appears you are remotely hosting and don't have access to the php.ini OR you've not read up on how to use a php.ini on your remotely hosted system.   Providers normally provide how-2's docs on such things.   Check with provider docs/helpdesk etc. for how to change max_execution_time variable.

You could try: php_value max_execution_time = 120 in .htaccess but if that results in an error 500 your package is restricted so then one will have to use a php.ini file in the moodle code directory (or maybe serveral moodle code subdiretories) to be able to get php scripts a longer time to run.

In php.ini file:

max_execution_time = 120

Changes to php require restart of apache.

BTW, the server PHP info in Moodle does show what php.ini file it's using.   If that is NOT in your space, again, check with provider.

Not sure a restore would fail IF it could not find a file ... it might pause a long time, however.

Next time ... report where you host your Moodle.   That will at least give someone attempting to assist information and they might be able to locate the providers FAQ/how-2 doc on the issues (that's really going the extra mile as a customer could do the same with very little effort + you get to test how reponsive your provider is).

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Back-Up File Too Large

by josh love -

Hey Ken,

Talked with my server guy and we increased the max_execution_time and the back-up went through flawlessly. Guess it was 'pausing' for a long time trying to get the restore through. 

Thank you for your help! I really appreciate this.

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

An assignment is an activity.   Those could be old version assignments turned in by students and the 400 folders are student ID numbers with the contents being the files that were submitted by students.   Think you'll have to look at an existing course to figure that out.

Lot's of potential issues if the backups were full backups from a 1.9.x version of Moodle.

When restoring a 1.9 course to a 2 Moodle must first convert the course to a 2 formatted backup and then it restores (or attempts to).

The max_packet_limit and open_files limit increase is what worked for me with a 'troubled' restore in the past.

So debugging turned on shows no errors ... or one just gets the 'white screen of death'?   How about any hints clues in MySQL logs or apache logs or php error logs?

'spirit of sharing', Ken

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Giancarlo De Pol -

My quiz backups (2.6.1) are usually about the same size (30 MB), without users data. These are math or physics quizzes, 30 questions each, with several pictures. I do resize my pictures to the least acceptable size for display before uploading them, they might be about 50kb each. One quiz might have 15 such pictures in it. In 1.9, I had noticed that I had the choice whether to "include course file", and I needed to include them, because  pictures linked in quiz questions were "course files"; but, either all of the course files (= pictures in my case) were included or none, not only those that were linked in the quiz being backupped. I used to ascribe the size of backups to that fact. Now, in 2.6 files are managed differently and I haven't yet understood rightly how things work, but backups are still about the same size, so I assumed that all the picture files, from all quizzes in the course, are included in the backup also in 2.6. I don't kow for sure and would be glad to hear from others usres if it is so (which would seem somewhat of a waste?).

Giancarlo De Pol

Average of ratings: -
In reply to Giancarlo De Pol

Re: Back-Up File Too Large

by Ken Task -
Picture of Particularly helpful Moodlers

Math and/or Physics courses would probably have equations/forumlas and use a Math editor to build.   Those get converted into images.

Suggest downloading a backup (.mbz).   Change the backup extension from .mbz to .zip and unzip into a work folder.  Inspect the contents of the extracted backup.   The xml files are pointers to all things in the backups - folders extracted and the files contained there in are in course_files by content hash names.

Unfortunately, there is no GUI Admin user interface into the new file system.   One has to use DB queries to find humanly recognizable file names to see the content hash assigned to the file and then a file browser of some sort to browse the directories of moodledata/filedir/ to find the content hashed named files.

Might shed some light:

http://docs.moodle.org/26/en/Course_files

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Back-Up File Too Large

by Giancarlo De Pol -

Thank you Ken and Przemek,

formulas are not converted to images in backups: that is done on the fly by the TeX filter, when the page is requested. Images of (already requested) formulas are indeed cached on the server (I don't know for how long), but that cache does not end up in the backup; only the LaTeX code does. What makes the backups huge are ALL the pictures (jpeg's) from ALL the questions in the course. I tried to unzip the backup as suggested, and they are all there, even if I choose to backup one single quiz. I saw the solution put forward by Przemek, it looks a good way out, BUT there seems to be no easy way to pick the jpeg's themselves: when I unzip the backup, they are all inside a directory structure like

file/a6/676e7a09bd5ac3d70aafc7c4b68a63aa501c41cc

with apparently random names (and no extension). There's a lot of them, for one single coures there are 168 such folders with one to 4 files in each.

Is there no other way to expose the original jpeg's? I'd be able to recognize them by name...

Thank you, Giancarlo

Average of ratings: -
In reply to Giancarlo De Pol

Re: Back-Up File Too Large

by Przemek Kaszubski -
Picture of Particularly helpful Moodlers Picture of Testers

Dear Giancarlo,

I don't think I understand precisely your case - I'm still experimenting with an uprade of our production site...

I think you need to verify that all the quizzes and assignments have been upgraded. See the upgrade tools in the site adminsitration panel.

Then perhaps work your way through the quizzes, or have your teachers do so, and ensure that they work, i.e. that the images are there. etc

If you want to recover the original image filenames - the names may be in the XML files or in the question edits. Otherwise I think your only hope are database searches IF you know the new file strings.

To extract jpeg files from a backup (and get all the **new** file strings in the process) you could try to search iteratively through a backup using regular expression search for the JPEG file header string. See this source: http://www.mikekunz.com/image_file_header.html .

A screenshot from my own attempt is attached to this post.  I used Total Commander for Windows, and my case-sensitive regex search was for:

(?-m)\A\xFF\xD8\xFF\xE0..\x4A\x46\x49\x46

Some pdf files sneaked through, but otherwise the search produced fine quality results.

I'm glad I was able to get that far, as I may shortly be in the same situation as yours.

Regards.

Przemek

PS. If your case is one of an upgrade, previous backups of the courses under Moodle 1.9 - if you have any - should store all the files directly.

Attachment 2014-05-04 , 15_34_56 - Total Commander 8.01 - Pracownia Komputerowa IFA.png
Attachment Find_Files_002_pcnt80.jpg
Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Przemek Kaszubski -
Picture of Particularly helpful Moodlers Picture of Testers

Hello,

I am experiencing the same problem - on an upgraded site. To my understanding this is because there are legacy files in upgraded courses, which - unfortunately - get backed up all en masse with any backups made in such a course. The solution that works, if I remember correctly, is to re-visit all the resources and activities that may be relying on legacy files and to refresh / recreate them in the settings by assigning files using the file picker, and then to delete the legacy files.

Average of ratings: Useful (1)
In reply to Przemek Kaszubski

Re: Back-Up File Too Large

by josh love -

Hey Przemek,

I went to the course and went to the Legacy files, and guess what? I found 27.7 MB of files sitting there. It brought my course size down to 1.2MB. The guy before me must of stored all these files onto the course. 

Thank you for sharing this!

Average of ratings: -
In reply to josh love

Re: Back-Up File Too Large

by Przemek Kaszubski -
Picture of Particularly helpful Moodlers Picture of Testers

Hi Josh,

Glad it worked for you. Verify if all the activities and resources work OK now, however. Some of the deletes may have been linked to the still existing course modules. If you find anything broken, download one of the old backups onto your machine, open it with an unzip facility, extract the file needed and re-upload it through the resource's / activity's file-picker. Might not be straightforward as Ken has explained above.

I just hope my own trying to help did not cause all these new problems.., :/

Average of ratings: -
In reply to Przemek Kaszubski

Re: Back-Up File Too Large

by josh love -

No worries Przemek, the weird thing was that none of these files were being used within the course. So deleting them should have no effect on the back-up/restore process. I made a copy of all the files before I did delete them so I can bring them back if needed.

I am just glad my course is now 1.2MB instead of 28.9MB smile

That eliminated one possibility of why my course isn't backing up. 

Average of ratings: -