Extend Amazon S3 repository support to S3-compatible services

Extend Amazon S3 repository support to S3-compatible services

by Basil Gohar -
Number of replies: 5
Relating to the issue Extend Amazon S3 repository support to S3-compatible services, I wanted to see if there was any community interest in extending the Amazon S3 repository support to non-AWS S3-compatible services.  There are many services that now provide S3-compatible APIs for object storage that, for all intents and purposes, function identically to AWS's S3 service.  The only difference lies in the assumption of a global AWS URL prefix.

Per Helen's request, I am posting here to see if there are others that would find this feature interesting enough for the Moodle dev team to allocate resources for it.  Of the advantages I can think of are the following:
  • Not needing to rely on a single provider for an object storage-based repository
  • Being able to make use of your own host's object storage service if you are not on AWS (e.g., Linode, Digital Ocean, and more)
  • Taking advantage of cheaper S3-compatible storage options (e.g., Backblaze has a compelling one!)
  • Host your own S3-compatible service (Ceph and MinIO are two free and open-source such options)
All of these would be possible with the just the ability to choose the S3 endpoint without the implicit assumption that it'll be AWS-only.  As I mentioned in the linked feature request, AWS S3 can still be the default, but allow the definition of a custom endpoint that would then take precedence over AWS if specified.

If this kind of feature interests you, please vote for the issue in the tracker.
Average of ratings: Useful (2)
In reply to Basil Gohar

Re: Extend Amazon S3 repository support to S3-compatible services

by Renaat Debleu -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
The global AWS url prefix is in my humble opinion not the only change needed. A long time ago I tried this out for the S3bucket repository plugin and I ran into more problems that I could handle.

Also, I do not see this as a Moodle issue, but a problem of the storage providers. Why do services like Blackblaze have not an easy to use and up to date b2-sdk-php available? If you find a well maintained PHP Software Development Kit that supports a lot of S3 compatible services, it is a piece of cake to create a compatible local plugin. But nobody is interested to maintain such a sdk, as people ask again and again to add some obscure storage service, and complain that certain compression settings or other options only generate errors.

If it as simple as replacing a url prefix, why do you not add a hardcoded prefix to the local/aws/sdk/Aws/data/endpoints.json.php file and/or repository/s3/S3.lib ? I am a 100% convinced that, if you can prove this solution works in all cases on at least 5 compatible services, this solution will be implemented in plugins in a couple of days.
Average of ratings: -
In reply to Basil Gohar

Re: Extend Amazon S3 repository support to S3-compatible services

by Jimmy Lynn -
I am interested in using the Linode object storage for this purpose. S3 is wayyy to expensive for many smaller sites. 
Average of ratings: -
In reply to Jimmy Lynn

Re: Extend Amazon S3 repository support to S3-compatible services

by Renaat Debleu -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
Please prove your point with a detailed price calculation sample. 

For smaller sites, a price calculation per GB (AWS) is far more interesting than a fee per 250GB (Linode). 

I do agree that it is more expensive, but stating that it wayyy to expensive is too bold a statement. For me, the price difference stays below 20 cents a month, a small price to pay for a +-well maintained php-sdk.
Average of ratings: -
In reply to Renaat Debleu

Re: Extend Amazon S3 repository support to S3-compatible services

by Ilan Schoenfeld -
Hello,

The price of storage on Backblaze is really "wayyy" cheaper than on AWS, at least for small ONG's like us with less than 1TB of data in the repo and really tight budgets.

We currently use about 850GB of B2 storage at Backblaze which get's accessed by our 8000+ users (through direct URLs we manually insert in moodle) on a constant basis, and we are paying US$6 per month, compared to the more than $50 per month that would cost in AWS, according to their calculator.

Note that our downloads average is less than 3 times our used storage, so we do not pay anything for downloads, and we don't pass the different types of free-transactions thresholds (https://www.backblaze.com/cloud-storage/transaction-pricing).

I can not speak for the quality of php-sdk documentation of providers, but makes sense that is not as good as AWS's and not being a developer I don't know what level of effort would requiere to allow the user to choose the SDK when setting up the S3 repository, but the basic functions of the AWS php-sdk (v2) are well implemented at least on Backblaze and Wasabi, maybe things have improved since you tested.

I wouldn't mind having a big warning shown when creating a non AWS S3 repository, letting me know of the possible issues we could have, and not sure if a list of "available" API calls could be shrunk when a non-AWS repo is created, but can someone clarify what are all the exact files that need to be modified to set the non-AWS provider. I'll be happy to run tests.
Average of ratings: -