Moving course images to S3

Moving course images to S3

by Martin Millmore -
Number of replies: 11

Some background - I started out with the goal of moving my moodle hosting to AWS, but came to the conclusion that it wasn't practical until Moodle can natively store files like the course backups to S3. Instead I decided to speed up my site by moving as many of the static files as I could to S3 - primarily the course images, but also the site logo, some static js and so on. The files are readable by the world on S3 (if they know the URL), so I only moved content which I don't mind anyone accessing. Also, I have been using moodle for 10 years (since v1.5), with courses which have been cloned as new courses year after year, resulting in lots of old obsolete legacy files that I just don't need clogging up the course backups, and making my moodledata directory unnecessarily large.

So, I wrote some code that looks through all of the lessons in a course, reads both legacy and new style images, pushes them to S3, and updates the course to point to the new location. It allows you to delete the old files after that is done - either fully deleting them if they are only in use by that course, or just unlinking then from the course if they are in use elsewhere. The net result for me is that page load time for the entire course (I have a bespoke printable page that shows me the entire course) is down from a typical full load time of 30s to 6s. 5s of the 6s is pulling together all of the lessons and assessment in to one page, so the image load time is down from 25s to 1s. The size of the backup of each course is down from about 10MB to about 100kB. And the daily cost of the S3 storage and serving up the files is a few cents. In short, it's a massive win for us.

I have attached my code in case anyone else finds this useful. To use it, you need to enter your account details against the S3 repository plugin. You'll also need to edit the code to enter your bucket name at line 78. Upload the aws library to your site from http://pear.amazonwebservices.com/get/aws.phar. I put both in a local plugin directory. Then you just call the file passing the parameter id=course number, e.g. http://www.mysite.com/local/myplugin/course_images_s3.php?id=123. Once the images have been moved, you can press the Delete button at the bottom of the screen. You'll need course update privileges to do all of the this.

The actual screen is messy - no navigation and so on. It's just something I wrote to get the job done for the way I use my site. However, it's robust, and anyone is free to use it as it, or make it better if they want. I'm sharing it in case anyone wants to do somthing similar - it might just save you a few days work. If you find it useful, let me know.

Thanks

Martin

Average of ratings: -
In reply to Martin Millmore

Re: Moving course images to S3

by Thom Rawson -
Thanks for this, Martin.

Just a couple questions to help me clarify this as I've been researching using AWS for a Moodle cluster:

1. Are only static files being served from the S3 bucket?
2. What about dynamic uploads from the student side - are those stored in S3, too?
3. Is the Moodle data directory in S3?

Thanks again for sharing your utility and your experience.

-Thom
In reply to Thom Rawson

Re: Moving course images to S3

by Martin Millmore -

Hi Thom,

Yes, I only serve static files from S3. I host moodle itself on a regular LAMP server elsewhere. S3 is not a php engine - it's only designed for serving static content. If you want to run php, you need to use EC2 instances. However I didn't do that for some of the reasons that it looks like you are also concerned about. I believe that to take true advantage of AWS, you would want to run moodle on an elastic beanstalk cluster. But that prevents you from mounting any permanent drive storage to more than one instance at a time, but EBS will expand to run on multiple instances - a big problem meaning that your regular upload data has no home. Until someone re-writes the code that maintains moodledata to allow it to write to something other than the file system, I don't think that elastic beanstalk is suitable for moodle hosting.

I didn't do anything with student uploads because my courses don't allow student uploads, so I had no need for it. I'm not even familiar enough with the nuances of student uploads to try and go near that I'm afraid!

Martin

In reply to Martin Millmore

Re: Moving course images to S3

by Ajeet Kumar -

Hi MArtin

I am using Moodle 2.6 . I want to access file from s3. I have given my s3 access key and secret key in s3 plugin setting. When i am trying to access s3 files from Moodle course, it is showing all bucket that I have created on s3 but when I click on any one bucket, it is displaying "Error while communicating with the repository 's3moodle'." 's3moodle' is name of my bucket that I have created on s3.

Kindly help me what I am doing wrong?

Regards

Ajeet

In reply to Ajeet Kumar

Re: Moving course images to S3

by Martin Millmore -

I ran in to a problem like that too (although you don't actually need to browse the repository to use my code). What I found is that the S3 repository code assumes you are using a US bucket at AWS. I had to change the code repository/s3/S3.php to change the endpoint to s3-eu-west-1.amazonaws.com at line 52, 76 and 1697. Your endpoint may be something different though. Or you may be experiencing a different issue - look in your php logs for more info.

 

Martin

Average of ratings: Useful (1)
In reply to Martin Millmore

Re: Moving course images to S3

by Ajeet Kumar -

Dear Martin,

Can you please describe how to upload content from moodle to s3.

 

Regards

Ajeet

 

In reply to Martin Millmore

Re: Moving course images to S3

by Ajeet Kumar -

Hi Martin,

Can you please explain how to install Moodle on Amazon Elastic Beanstalk.

 

Regards

Ajeet

In reply to Martin Millmore

Re: Moving course images to S3

by Andrew Boag -

Hi Martin,

Just wanted to let you know about a new Moodle course backup solution CourseBank (https://coursebank.biz) that allows your Moodle courses to be pushed externally to cloud storage.

Currently, we are in preview mode for a select group of Moodle clients. Details are on the site.

There is a free tier of up to 5GB storage that will allow you to push your automated Moodle course backups into the cloud. Larger storage capacities have a paid-for model.

Hope this is of use to you.

Cheers

In reply to Martin Millmore

Re: Moving course images to S3

by Ahmad Arshad -

Can your code be used to get audio files stored in the filedir of moodledata?