How to upload REALLY big scorm modules

How to upload REALLY big scorm modules

by Christian Peters -
Number of replies: 19

Heyhey,


I'm down to my last with the new File API and really big scorm packages.
Scenario: I have a client with really big packages: 500 MB / 12.000 files (it's generated with a software named TTS).


I'm coming from the german forum and I've heard all the talk about "Who would make such big courses?",  "Why don't you use something else?" etc.; now I really need a solution.


I've expanded max_execution_time to nearby infinite and managed to upload (and get to work ...) an 360 MB course. But it took me 4 hours (the fails not counted).


There has to be a simpler way! I've come to understand the file API & i don't want every file used this way (when I use the scorm activity).


Now I upload the ZIP file ('cause Moodle accepts only ZIP files), the problem is not the unzipping (at least i think so); it's the huge "make-a-hash-update-the-database-and-copy-the-file-process".

I've managed to upload HTML-Deserts via Folders (Verzeichnis in German). Can it be done similar?

I didn't understand the two other options available: When I link to a manifest, moodle complains "File not found", even if everything is right in place.
Please help me. If this wouldn't work, we have to seriously consider to move to another LMS - cause such big courses are very common in germany (due to a very aggressive Marketing Strategy by TTS).

What's to do?

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Amy Groshek -
scp or at the very least sftp are the best ways to move large files onto a server, *not* a browser-based upload.

I totally agree with you: the 2.0 file API is lame. We are avoiding it with all our instructional design work, and instead placing the files in the theme or in a subdirectory of public_html. You could do the same with SCORM packages, and link to them with an absolute URL. The file API may be an improvement for students and teachers, but it is a big hassle for anyone trying add any kind of nonstandard interactive content to a course/site. I have also been able to serve SCORM packages directly from Alfresco, either in a public folder, or using a webscript to access a private folder. So it's not like you don't have options.

Unfortunately, we can't help you make huge files play well on the Internet. There is no magic switch coded into Moodle which makes SCORM authors not have to follow good Web design practices (keep file size small, stream multimedia from a cdn, etc). SCORM is online content, and all of the normal standards you would apply to online content apply to SCORM packages. If Germany's popular software doesn't know enough to output usable file sizes (most don't), and you can't educate the people authoring the content to do it correctly, there isn't much we can do for you on the LMS end. You can start shopping for alternative LMSs, but I can tell you right now that there is no LMS on the market which can increase the end-user's bandwidth.

Regards,
Amy

Average of ratings: Useful (2)
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -

Well, first of all, I don't want to go shopping for another LMS than moodle. I love the old lady!


My problem is not the end-users bandwidth. My problem is getting the course into moodle. The course itself works nice (well, as nice as scorm can get), once it's on - it worked nice up to Moodle 1.9.x; the mess comes with the file API!!!

It's also not about uploading through the browser (I use a repository & FTP) & as said it's not about unzipping. It's just about the File API!!!

You could do the same with SCORM packages, and link to them with an absolute URL. [...] I have also been able to serve SCORM packages directly from Alfresco, either in a public folder, or using a webscript to access a private folder. So it's not like you don't have options.

I'd really love to get some details on this, because I failed with this procedure: I didn't get this to work & always receive an error (file not found).

Also: The SCORM API has to stay intact; I don't want to use static courses, I need the API that logs access and completion of the courses (some - like working safety - have legal binding and have to be tracked).

Thank you!

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Hi Christian,

Yes, when you link to a scorm.zip file by pasting a URL into the SCORM activity editing in 1.9, the API stays intact. Moodle goes and gets the .zip and unpacks it in moodledata. That is what the Auto-update frequency setting is about; it determines how often to refresh the SCORM content from that remote URL.

But your question prompted me to test that solution on 2.0. It is no longer possible in 2.0 to paste a URL into the SCORM activity editing screen. You *have* to select "Choose a file" and use the file browser. This is a very poor implementation, and is going to waste a ton of time for admins who have to add hundreds of SCORM activities to a Moodle site. We should not be implementing in this way!!

Dan, can you weigh in here? Is there a way that we can still allow users to get around having to use the file browser? Web developers will find it most intuitive to just copy and paste a URL.

Regards,
Amy
Average of ratings: Useful (1)
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -

We'll you can link to a scorm activity, that resides in your htdocs folder. But this'd mean: The course is available to everyone - this is not an option, since we're talking about bought courses (we even do not own the licence).

Thanks so much for taking the time to think about a solution with me smile

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Hi Christian,

If you're going to use a repo, I don't think there's a workaround to using the file browser, with one exception: *if* the SCORM activity editing pane were altered to allow you to paste in a URL, *and* you wrote a PHP script to go and fetch the secured package from your repository outside of the Moodle interface with the repo, then you could add files in the repo by linking to that script, without navigating the Moodle file browser to do so. But I don't see any other way to have both secured content and not use a file browser of some sort when adding that content to a SCORM activity.

Aside from that, what we've established (correct me if I'm wrong) is that your issue is really to do with the functionality within the file browser, not about the SCORM activity...

I still think it should be possible to paste in a URL when adding a SCORM package to a SCORM activity. To clarify, you might be pasting in a URL with query string data that pulls content from a secure area of a content vendor site. Or you might be pasting a URL to a script which first validates with the repo, then accesses the content on a secure repo and returns it to the SCORM activity (this is what I tested and implemented with Alfresco in 1.9). While the number of people using solutions like this is always going to be smaller, they are usually high-volume users. We should not lock people into functionality they may not want.

Regards,
Amy
Average of ratings: -
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -

We'll my problem is the file API: The whole process takes way to long and is way to complicated and my server regularly crashes after hours of hours of working (and contaminating my mdl_files).

The perfect solution to me would be the way it's handled with files: I'm uploading a zip, create a directory (within the file api), unzip it their & select a main file.

None of this horrible hashing takes place. It takes seconds!

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Christian, you shouldn't have to unzip. You should be able to link to the zip dir, and Moodle will unzip for you. But imsmanifest.xml needs to be in the immediate child directory. Have you unzipped your content packages to make sure that's so?
http://docs.moodle.org/en/SCORM_FAQ#Manifest_not_found

But there are some issues with the unzip utility stalling with large SCORM packages:
http://docs.moodle.org/en/SCORM_FAQ#Unzip_issues
(That is 1.9-related so I'm not sure how it transfers to 2.0.) I don't see any documentation indicating what unzip utility M2 uses, or where to change that. The option to change the path to zip, which existed in 1.9, is gone. Though Tim Hunt's post here http://moodle.org/mod/forum/discuss.php?d=114786 seems to indicate that the newer version of PHP won't have the same unzip issues.

Also, there are some causes of the File not found error that are documented in the FAQ; have you eliminated those?
http://docs.moodle.org/en/SCORM_FAQ#File_not_found_error
Average of ratings: Useful (1)
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -
Thanks for your help!

But as said, the unzipping is not the problem.The files are getting unzipped. I've read really EVERYTHING on this topic ^^

. The process of hashing them to the moodledata folder takes too long. I'm looking for a way to get around these WITHOUT pushing the SCORM Package in a public folder.

I have erased my file not found Problem, and can call the scorm package now via giving an url. But the SCORM Package has to lie in a public folder then. This is not an option, 'cause the courses are not my property

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Right, I get you re: the unsecured content. In 1.9 I could recommend other ways of directing the SCORM module to the content in your repo, but without the ability to paste a URL into the SCORM activity, none of that is available. Gotta' love that new file browser. ;)
Average of ratings: -
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -
I haven't buried my dreams about finding a solution yet smile
Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

by default when adding a new SCORM object/editing an old one you only get the file picker as a way of selecting a SCORM object...but... if you dig around in:
admin > plugins >Activity Modules > SCORM
you might find the settings "Enable external package type", "Enable downloaded package type"

if you enable one or both of these you will get a few extra options when editing/creating a scorm - here's an explanation of the options:

  • Uploaded package - Enables a SCORM package to be chosen via the file picker
  • External SCORM manifest - Enables an imsmanifest.xml URL to be specified. Note: If the URL has a different domain name than your site, then "Downloaded package" is a better option, since otherwise grades are not saved.
  • Downloaded package - Enables a package URL to be specified. The package will be unzipped and saved locally, and updated when the external SCORM package is updated.
  • Local IMS content repository - Enables a package to be selected from within an IMS repository

don't ask me if the IMS content repository still works - it was very messy in 1.9 - haven't tested it in 2.0...

but.. the other option that works quite when dealing with larger scorm objects is:
set up a FIle repo: http://docs.moodle.org/en/index.php?title=File_system_repository_configuration - call it something like "SCORM repository"
Set up FTP access to the new file repo so you can ftp files into the directory
ftp your SCORM zip files into the new folder
Then create a new SCORM in Moodle - select the SCORM repository in your filepicker - select the zip - done smile

The file repo will only work with zip files at the moment(not unzipped scorms and direct linking to manifes) - I'm hoping some imaginative person will create a new repo plugin that allows direct linking to the manifest inside the repo at some point in future.

hope that helps!

Average of ratings: Useful (5)
In reply to Dan Marsden

Re: How to upload REALLY big scorm modules

by Christian Peters -

Thank you so much for your help, Dan.

Unfortunately: This won't work for me. (If I do understand you right. Thus, I hope I don't ^^).

Except for Local IMS content repository (will try it later), I've tried all of your options:

  • Upload package, Download Package & the file repro come with the same problem: My Problem is not to get the SCORM-ZIP onto the server. My problem kicks in, when Moodle tries to save the unzipped stuff into it's moodledata/filedir & mdl_files structure.
  • This works, but it means that the SCORM course has to be publicity available. This is not an option.

I'm hoping some imaginative person will create a new repo plugin that allows direct linking to the manifest inside the repo at some point in future.

Well, this is exactly what I'm looking for.

Average of ratings: -
In reply to Christian Peters

Re: How to upload REALLY big scorm modules

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

unfortunately the IMS repo just puts the files in a "public" place so if you're looking for a "secure" place to locate the files it won't work.

I think it shold be possible to create a repo based on the existing file repo that allows you to link directly to a manifest file - that way it's all still stored in a secure manner. If you're interested in throwing some $$ at developing a new repo type that does this feel free to drop me a line smile

Average of ratings: -
In reply to Dan Marsden

Re: How to upload REALLY big scorm modules

by Christian Peters -

Well, I'm a webdeveloper myself. Thus, I'll try for myself at first.

I'm fairly sure, that my client won't give me extra budget to build a plugin that gives him the opportunity to do something with Moodle 2.0, that he could do with moodle 1.x for free.

But thanks. I hoped for an existing solution, now I know what to do, when I'm back from holidays smile

Average of ratings: -
In reply to Dan Marsden

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Dan, awesome work! I didn't even know that was available. I'll have to find time to update the scorm docs to cover those new options.

Christian, you should submit a bug report re: performance of the hashing. I think this might be related to some of the other performance improvements that developers are looking at: http://docs.moodle.org/en/index.php?title=Development:Performance_2.x&oldid=82979
Average of ratings: -
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Christian Peters -

Yeah, I will. Thanks for your help. I'm not really sure where and how, but I'll take a look.

Average of ratings: -
In reply to Dan Marsden

Re: How to upload REALLY big scorm modules

by Amy Groshek -
Dan, I've added your info here to a new SCORM docs page. Could you check to make sure it's accurate?

http://docs.moodle.org/en/SCORM_Admin_Options

Regards,
A
Average of ratings: -
In reply to Amy Groshek

Re: How to upload REALLY big scorm modules

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

almost - the key difference between "external" and "dowloaded" package types is that external allows you to link directly to an imsmanifest.xml and the "downloaded" package type allows you to link to a zip.

Average of ratings: -