General developer forum

Files API: time for MUCification?

 
 
Picture of Matteo Scaramuccia
Files API: time for MUCification?
Group DevelopersGroup Particularly helpful Moodlers

Hi,
any plan in HQ to MUCify the stored_file class, in combination with file_storage to let eventually file_storage supervise stored_file invalidation too (I'm not sure right now if there are needs for such a supervisor)?

I actually know the first criticism: tune your database to perform better and even better wink ... and probably the second: it's too risky having DB and Cache out of sync at that level.

Besides, caching at that level shouldn't be activated without the availability of an application cache configured with a memory-based store, if the aim is to increase performances - e.g.: pluginfile.php is a great, in terms of hits, read-only consumer of the Files API.

I've filed an issue to explore the possibility to increase the cache_definition parameters (MDL-36557).

Any kind of feedback (criticism first!) is welcome smile,
Matteo

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Files API: time for MUCification?
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

What is the performance bottleneck in files that you think is a problem?

Without identifying the specific performance problem to try to solve, you are just saying "We have a sledgehammer (MUC). Let's hit something."

 
Average of ratings: -
Picture of Matteo Scaramuccia
Re: Files API: time for MUCification?
Group DevelopersGroup Particularly helpful Moodlers

Hi Tim,
you're right wink, here it is the use case: lowering concurrent connections to the database when delivering a (big) amount of files referenced by an HTML file provided by Moodle, typically when delivering Learning Packages (AICC/SCORM) with complex items - not always under your control, if they come from Content Vendors - to a large number of users.

While tuning the DB could address the perfomance issue in terms of better hits/sec and lower the time window for the per-request DB connection thanks to its own cache mechanisms, resolving the same set of queries most of the time could be a good candidate for a local cache at the application level too, starting with stored_file up to, maybe, kind of cache even for those data regularly used in the chain started by pluginfile.php.

Having memcached on board somewhere - part of each front-end of a Moodle clustered instance or (at least 2) dedicated simple servers - is not so difficult and a common practice for other webapps so why not improving the delivery of files in Moodle via MUC providing a memory based cache definition to be used in this scenario i.e. explicitly activated by the Administrator and mapped into a good memory store?
I can argue against myself (already done before filing this post wink) that there are already browser caching mechanisms ($CFG->filelifetime, that helps a lot on average but with same set of regular users in the near time) as well as the possibility to introduce other mechanisms in front of the Moodle clustered instance.

That's the picture I've in mind, maybe strictly from a SysAd POV and with a specific scenario (delivering Learning Packages). It's not a requirement for some specific projects but something I immediately realized when reading/learning about MUC.

Matteo

 
Average of ratings: -
Picture of sam marshall
Re: Files API: time for MUCification?
Group DevelopersGroup Particularly helpful MoodlersGroup Testers

Hm - are you talking about performance of file serving requests i.e. if the user requests a large number of supplementary files (images, JS, CSS etc) all at once?

I think that's a fairly specialist issue - and the system makes a number of database requests as part of a pluginfile.php request, the majority of which are not to the files table. In fact the files table queries are usually rather simple. So basically, I'm not sure it is the first place to look for performance improvement in fileserving. There's probably still room for more generic improvements to the access-checking/setup.php type code.

Regarding file API and performance, what I'd personally like is an allowed way to access the real path of the file - if available - for read-only use. This would save unnecessary copies of files to temp directory for various processing in various custom modules/tasks I've written, which can take significant time and I/O in the case of large files. In other words, if there was a
function like


$path = $file->allocate_read_only_path()

$file->release_read_only_path($path)

So if your file is stored in the local filesystem it just returns the path and does nothing, if it's stored elsewhere then the allocate/release functions would do the same as the current function that saves it to a temp path, then delete it on release.

--sam

 

 
Average of ratings:Useful (1)
Picture of Matteo Scaramuccia
Re: Files API: time for MUCification?
Group DevelopersGroup Particularly helpful Moodlers

Hi Sam,
yes, talking about CSS, IMG, JS, TXT, XML, ... .

OK, so the first useful step could not be caching stored_file objects but reviewing what I've quickly referenced as the chain: probably a good point for the final goal but looking at a baby steps approach I was guessing about stored_file, having already found a TODO about MUC in the new related code, repository::sync_external_file().

About your proposal: really interesting and probably a required improvement, indeed get_content_file_location() is protected. BTW, never played with complex dev tasks w/ Files API except an exercise, DAVRoot, which gave me the opportunity to file MDL-30946.

TNX for your reply: you gave me new interesting points to review my initial thoughts,
Matteo

 
Average of ratings: -
Picture of Matteo Scaramuccia
Re: Files API: time for MUCification?
Group DevelopersGroup Particularly helpful Moodlers

Hi All,
just for the record, an issue has been already filed before my initial question: #34340, as a subtask of MDL-34224.

Matteo

 
Average of ratings: -
Picture of Matteo Scaramuccia
Re: Files API: time for MUCification?
Group DevelopersGroup Particularly helpful Moodlers

Again, just for the record, here is an interesting and natural evolution of the abstraction provided by file_storage (and stored_file): http://docs.moodle.org/dev/File_Storage_Plugintype.

Matteo

 
Average of ratings: -