Purging, archiving, disk space

Re: Purging, archiving, disk space

by Lee Cujes -
Number of replies: 8

Hi Justin

Thanks. We will have thousands of students a year submitting to this particular assignment, so there's no option for us to delete the assignment itself. I want to be able to delete the file submission of the student once it's been graded.

There doesn't appear to be a way to do this in the UI - even for a site admin.

What path on the webserver does PoodLL place the uploads?

Regards

Lee

In reply to Lee Cujes

Re: Purging, archiving, disk space

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

The files all go into the Moodle file system, which is a black box really. Its not an easy thing at all to find them and delete them via the OS file system. You need to use the Moodle APIs to delete them. Right now we do not have a 'delete all' Poodll files feature. We just let Moodle handle the deleting, when the time is right. 

But actually it is on the roadmap. Just no promise of when

In reply to Justin Hunt

Re: Purging, archiving, disk space

by Lee Cujes -

Hi Justin

So we've been running PoodLL for a few months now happily collecting and grading videos from hundreds of students.

This morning Moodle crashed with a server partition full error, and I'm fairly convinced it's PoodLL's video storage that is the main culprit.

You: "Right now we just leave it up to Moodle to manage those files."

I just don't understand this. I don't see how Moodle actually manages anything. It just stores the files in its "black box".

We don't have unlimited storage (who does?). I need to understand how much disk space is being consumed by PoodLL's video store, and I need to be able to actively manage them to ensure our production server doesn't fill up and crash.

Your advice/guidance here would be much appreciated.

Regards
Lee


In reply to Lee Cujes

Re: Purging, archiving, disk space

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi Lee

I am really sorry about the trouble. You will need about 9MB for each minute of video. So you do need to make sure you have ample disk space.

The key thing here is that Moodle manages the activity(quiz, activity, forum etc) and by extension the files associated with that activity. So to remove files, you need to remove the attempts or submissions or posts that contain them. Moodle then deletes the files or moves them to trash. The actual restoration of file space (true deletion of file) takes about 4 days for some reason.

If a file is no longer necessary then presumably neither is the activity/post/submission containing it. So that really is the best way to manage the files. Poodll could do better in reporting about files, I agree. But its just not a good idea for Poodll to try to remove files that are contained in activities.

I know that you are using assignments and students have relatively short enrolment periods. Since Moodle does not offer the ability to delete assignment submissions individually, my suggestion for pruning files is to have two courses and alternate them. When one course no longer has users enroled, then reset the course. This will remove the files associated. You can be specific and reset only the assignments in the course if that is better. 


In reply to Justin Hunt

Re: Purging, archiving, disk space

by Lee Cujes -

Hi Justin

Thanks for the response, and the ideas for possible management. The two courses idea is a good one but doesn't work for us as we never have the situation of zero users enrolled.

At this point I have been able to build a report to see the culprit files. As expected, they are already consuming a significant portion of our overall server allocation.

This seems to be exacerbated by the system seeming to store multiple versions of the same file - see below:

cvxb

We've managed to clear out about 30GB by removing no-longer-needed backups, but I know that within 1-2 months we will be back here again and will either need to commission more server space, or... I don't know!

We really need Cloud PoodLL and the ability to have these files stored in a separately managed repository (e.g. AWS bucket). Any news in this regard?

Kind Regards
Lee

In reply to Lee Cujes

Re: Purging, archiving, disk space

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi Lee

Those multiple file entries are a bit of a red herring (fortunately). Moodle only stores the file data once, though it has separate file entries in the database. One of the fields you have in that table is the contenthash field. For each of those files it will be the same contenthash, and that is how Moodle ties them all together (draft files and permanent files).


CloudPoodll is working well and we have built the new CloudPoodll assignment, and another plugin on top of it. The Cloud Poodll assignment is in alpha/beta now. I think we can get it to you in a fortnight. 

In reply to Justin Hunt

Re: Purging, archiving, disk space

by Lee Cujes -

Hi Justin

Thanks for that - I've altered my reporting to exclude the duplicate database entries to get a more correct representation.

Very good news on the Cloud PoodLL front - I'll touch base in a couple of weeks if I don't hear from you sooner. Thanks!

Kind Regards
Lee

In reply to Lee Cujes

Re: Purging, archiving, disk space

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi Lee

In the end I fast tracked the Cloud Poodll Assignment Submission plugin. 

Here it is here:

https://github.com/justinhunt/moodle-assignsubmission_cloudpoodll

The regular Poodll registration key is irrelevant here. This is not dependent on the Poodll filter or any other plugin. You just need a new thing called API username (thats your poodll.com username) and API secret. You can pick the API secret up at Poodll.com as described here:

https://support.poodll.com/support/solutions/articles/19000083076-cloud-poodll-api-secret

Hope it helps


In reply to Justin Hunt

Re: Purging, archiving, disk space

by Lee Cujes -

Thanks Justin - will message you directly on this. Cheers!