Future major features

File system abstraction

 
Picture of Andrew Nicols
File system abstraction
Core developersMoodle HQParticularly helpful MoodlersPlugin developersTesters

Hi all,

As some of you are already aware, I've recently been working on changes to the Moodle File Storage API to allow storage of filedir files in alternative file systems. This is being worked on in MDL-46375 and documentation is currently a work-in-progress.

Once this change has been integrated, it will allow developers to write their own storage backend for files currently stored in the Moodle filedir. Please note that files currently stored in either the temp directory, or the cache directory are not stored through this API, and must still be shared on disk between servers in a cluster.

What this change does mean is that the filedir can be replaced with a range of alternatives. The most popularly suggested backend is Amazon's S3, however you could also write backends for systems including:

  • Amazon S3;
  • Rackspace Cloud Files;
  • Ceph;
  • FTP;
  • WebDAV;
  • etc.

This change should not require any changes for developers or administrators as standard - well, unless you're calling parts of the File Storage API which you shouldn't be ;)

For developers wishing to implement a File System alternative, there should be generally minimal work required.

The new API has the concept of both local and remote file paths, with remote file paths beings consumed where possible.

Any path using a standard Protocol Wrapper is accepted (or a standard path-on-disk).

I hope to finish this change in the next week, but comments and queries are welcomed as ever,

Andrew

 
Average of ratings: Useful (2)
Me!
Re: File system abstraction
Core developersMoodle HQParticularly helpful MoodlersPlugin developersTesters

There is one FTP example attached to the issue but I don't see any benefits of using a FTP server as a remote file system ever.

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: File system abstraction
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

I am really happy to see this change landing. I had a look over the code, and it seems like a good implementation. The changes are almost all inside the file API code. Code using the file API will hardly have to change at all.

The only remaining thing I am not sure about is whether different file system back-ends should be a full Moodle plugin-type in their own folder, with version.php etc. That is a bit more heavy weight. (Currently they are just one class file in a particular folder. Oh, and hopefully another file of unit tests.) However, if they were plugins, it would be easier for people to share them through the Moodle plugins DB. That might seem like a small thing, but actually, I think it is quite important.

Anyway, thanks to Andrew for working on this.

 
Average of ratings: -
Picture of Tomasz Muras
Re: File system abstraction
Core developersParticularly helpful MoodlersPlugin developersPlugins guardiansTranslators

Could you have a look at http://knplabs.github.io/Gaufrette - would it be possible to use it here in some way? It provides quite a few FS adapters out of the box, AWS included.

 
Average of ratings: -
Dan at desk in Moodle HQ, Perth
Re: File system abstraction
Core developersMoodle Course Creator Certificate holdersMoodle HQPlugin developersTesters

would it be possible to use it here in some way

Not as part of this change, its just about making the internals more easily extendable from the current on-disk implementation. Plugin developers can use that to write a replacement file system though.

 
Average of ratings: -