Backup of large course (3.85GB)

Backup of large course (3.85GB)

by Tim Dressel -
Number of replies: 11

Hi folks,

I have a teacher whom has created a very large course (large from a content standpoint, lots of embedded videos, etc) where the /moodledata/coursenumber folder is 3.84GB. When I try to backup I get an error while copying the zip file to the course directory (when watching top (this is ubuntu 32bit linux) I see zip running for several minutes).

I have read a bunch of stuff here on the forums, but none of which as worked. Here is what I've tried so far:

In php.ini:

upload_max_filesize = 10000M

post_max_size = 10000M

max_execution_time = 6000

max_input_time = 6000

memory_limit = 2048M

Within moodle I have done the following:

System paths (set path to zip, unzip, du, aspell)

Performance (set Extra PHP memory limit to 1024M)

The physical system (ubuntu 32bit) has a 100GB disk, 50% of which is free so disk space is not an issue.

This is a virtual server (I'm working with a copy of it so I'm safe to do dangerous things), and I've given it 2 vCPU's and 3.5GB ram.

It was running 1.9.9, but as a troubleshooting test I upgrades to 1.9.12+ (weekly). Upgrade went flawlessly, but still no joy.

Permissions everywhere are good.

I don't think there is anything more in the FAQ that I haven't already tried.

Any thoughts out there?

Tim

Average of ratings: -
In reply to Tim Dressel

Re: Backup of large course (3.85GB)

by Gavin Henrick -
Picture of Plugin developers

You could try to get it smaller, by systematically substituting large files with stub placeholders to get size down below 2 gig perhaps? and where ever you are moving it to, do the reverse afterwards?

However, are there old unused files in there?

Average of ratings: -
In reply to Gavin Henrick

Re: Backup of large course (3.85GB)

by Tim Dressel -

Unfortunately the majority of the large files are already heavily compressed MP4 videos. For sure I could go through that process, but I'd just like to know if 2GB is the limit, or if I need to go even smaller. It's a key piece to the puzzle, and important feedback I will need to communicate to my teachers that are using Moodle.

I guess in the same vein, is there a "flat file" way of restoring a course that doesn't go through the backup/restore process? Like could I copy at the file level the course folder to a new location, and then tweak the database to "see" the copy? I know that's pretty ugly, and probably not supported, but thought I'd ask.

Average of ratings: -
In reply to Tim Dressel

Re: Backup of large course (3.85GB)

by Steve Bluck -

Have you checked that you don't have limits in an .htaccess files e.g.:

php_value  upload_max_filesize  8M
php_value  post_max_size  16M
php_value  memory_limit  24M

Average of ratings: -
In reply to Steve Bluck

Re: Backup of large course (3.85GB)

by Tim Dressel -

I'm running PHP as a module, so no htaccess applies (could someone confirm my understanding of that please?).

Average of ratings: -
In reply to Tim Dressel

Re: Backup of large course (3.85GB)

by Colin Fraser -
Picture of Documentation writers Picture of Testers

Hi Tim,

Yes, I would not think so either. You have control of the ini file then? Unless your server has a limiting variable, there would be no point in using a .htacces file.

There is a huge issue here, pardon the pun, that goes to the heart of Moodle actually. Moodle is based in the idea of "write once - read anywhere" Web technologies, obviously, be we still need to remind ourselves of that. The technologies themselves are not that great yet, the hardware and software limitations are constraining development, which is obvious here. PHP cannot handle that size file (yet) so how do you deal with it? Am I right in thinking that until this is resolved then you are unable to do a site backup at all?

The only thing I can suggest is that the course is broken down into several smaller courses, perhaps one lesson per course. Using a  metacourse structure will resolve login issues and if it means the the author is inconvenienced a little, well tough. If average course is around the 500mb range, then that is the price that has to be paid to continue the good order of the site and the fulfilment of the responsibilities of the Administrator, (you?).

<soapbox id="interferingsagebusybody">

As an aside, I know I have no idea of the circumstances as to how this happened in the first place, but I am somewhat appalled that any Course Creator or Editing-Teacher would allow a course to get so large. To me this suggests poor course design and structure, a definite lack of forethought, or worse, understanding, about the impact of the materials used in the course.  I know videos can get large, easily, and in many courses, videos are an absolute godsend. Camtasia, CamStudio or even Wink will produce small tight videos for any one thing. I suspect that these videos are showing high quality, multistep processes or something similar. Can these be broken down into atomic components?

That is what I am telling all the teachers who are asking about making their own videos for Moodle. Each video is, I am suggesting, a self-contained specific purpose video that should not last any more than about 5 minutes (2-3 minutes is, I suspect, about best). I would suggest that hand made movies can be broken into individual scenes that last little more than 5 minutes. I would contend that, with some effort, even Hollywood blockbusters can be broken down into individual clips with very few scenes that last more than that. NatGeo specials, well that would be a challenge..smile

</soapbox>

Average of ratings: -
In reply to Colin Fraser

Re: Backup of large course (3.85GB)

by Tim Dressel -

Hi Colin,

Thanks for the reply, and confirming my suspicions that this is a PHP issue and not a moodle one. And I agree that this course is silly huge. The teacher in question is actually a star as far as I'm concerned, and she has done great things to create engaging content for students.

I'm thinking the best way for us to address this going forward is to simply give the teachers ftp access to a directory on the web server where they can dump their videos. They can then link from their moodle courses to those videos so backup should be easy peasy. It also avoids the issue of having to break up a course into smaller chunks. Hopefully. smile

 

Cheers,

Tim

Average of ratings: Useful (1)
In reply to Tim Dressel

Re: Backup of large course (3.85GB)

by Colin Fraser -
Picture of Documentation writers Picture of Testers

What a great idea, and one i had not considered. Keep the course structure but the intense bits are moved somewhere else and linked to. That will get me out of an impending hole at some stage, I can see something similar happening in my environment, and been giving a lot of people the basics in using metacourses to avoid it. Will have to talk to the Admin Group, but it is an approach that might work...mmmmmmmm Thanks Tim, excellent suggestion.

Average of ratings: -
In reply to Colin Fraser

Re: Backup of large course (3.85GB)

by Tim Dressel -

Sorry, forgot to answer the site backup question,,, this moodle server (we run three actually) are dedicated to hosting moodle, and nothing more, so we don't actuall do any "moodle" backups, ever. Instead we run full image backups of the virtual machines, once using backupexec monday to friday night, and then 7 days a week in the mornings using the vmware vstorage tools which gives file level recovery granularity of we need it.

The only reason we do course backups is to make a copy of the courses for a teacher. Over time we have discovered that teachers are exceptional at making courses, managing students, and all that great stuff that teachers do. When it comes to administration (backups, restores, system management), teachers tend to forget that they are working in a system that is not exclusively theirs, so we don't let our teachers do things like backups and restores, instead we leave this to the folks who professionally manage our servers. I mean no disrespect to teachers, its just the balance that we've found that works the best.

Average of ratings: -
In reply to Tim Dressel

Re: Backup of large course (3.85GB)

by Robert Brenstein -
If backups are handled by your IT staff, then may be you should backup your site and forget backing up individual courses through Moodle. The latter seems better for archiving but with restore restrictions which invariably come with newer Moodle versions, that is of limited utility in reality. When backing up the site as the site, IT people can dump the database into sql or xml file and tar or whatever the moodledata and moodle program directories, and backup the three. This gives full restore now and makes it possible to recreate the site to access content in the future, so it can be used for both running and archival backups.

The important aspect here is that working on the back-end, the IT people are not subject to the memory limits, script timeouts, etc that Moodle scripts are.
Average of ratings: -
In reply to Colin Fraser

Re: Backup of large course (3.85GB)

by Richard Wallace -

One solution as well for the large file size, presuming it is video files that are making the site massive (and I totally agree with Colins comments above) is to use an external Video hosting provider to hold these videos.

We use VIMEO (which is like a private youtube system) that allows users to uplaod a video, embed in a site and have all teh functionality of youtube or teacher tube, but also allow for the video to only be viewed from a particular URL.

By doing this, the load is taken off the server, backups are of the core content, are much smaller, simpler to restore and much more manageable, with just links to the heavy video content.

Cheers Richard

Average of ratings: -
In reply to Richard Wallace

Re: Backup of large course (3.85GB)

by Tim Dressel -

Thanks for that,,, I did consider a hoster for the video content, but then it puts that content needing to be traversed over the public internet when the majority of our students are "in-house" connected to our gig backbone.

I'm meeting with the teacher in question tomorrow, and we'll make something work in house. I'll post back here what our solution was.

Cheers,

Average of ratings: -