Slow file download and playing SCORM videos

Slow file download and playing SCORM videos

by Sali Kaceli -
Number of replies: 14

After upgrading to version 3.1 of our LAMP Moodle site, we are noticing that the file downloads are really slow. Users are experiencing this particularly when playing SCORM activities whether streaming or downloading them (by right clicking). The server is quite powerful with RAID 5 SSD drives, and 80+GB of RAM and the users average around 100 concurrent users.


This issue seems to be occurring onsite on he same network where the server is and remotely. 

Average of ratings: -
In reply to Sali Kaceli

Re: Slow file download and playing SCORM videos

by Sali Kaceli -

Does anyone have a copy of optimized settings for Apache 2.4 config. file that would like to share. I think the issue with the slow downloads is something related to Apache config. Obviously, having SSL enabled slows down the downloads but LAN downoads should not be around 100K/s. 

In reply to Sali Kaceli

Re: Slow file download and playing SCORM videos

by Nelson Moller -

I had an awful performance in a new site and the change that improved the performance was to fallback to php 5.6.31. I'm running version (2017051502.05) of moodle 3.3(3.3.2+ (Build: 20170928)).

My stack was running in kubernetes on php 7.0.21, after searching the causes of that long download time, I change the php version and the change was incredible. 

From ~50s to less than 5s! I'm still shocked about and I would like to know what caused that.

Average of ratings: Useful (1)
In reply to Nelson Moller

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Nelson,
that's strange: moving to PHP 7.x will increase the performances.

Are you able to share a test env with your configuration by sharing the k8s manifest or a "simple" docker compose file describing all the bits and pieces of your containerized env?

TIA,
Matteo

In reply to Matteo Scaramuccia

Re: Slow file download and playing SCORM videos

by Nelson Moller -

Hi Matteo,

Yes, I had the same reaction sad

You can get the pieces of it at :

https://github.com/nmoller/scorm-issue

In my local station the difference goes from ~3s to ~13s, it is still considerable.

Average of ratings: Useful (1)
In reply to Nelson Moller

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Nelson,
will try by myself, spare time permitted.
Some quick notes:

  1. try to not map Moodle data as a volume, just to keep the system within the volatile context... obviously JUST for testing purposes
  2. memcached 1.5 looks broken for Moodle purposes, see MDL-60037
  3. add https://moodle.org/plugins/report_benchmark to your image|container to evaluate the differences in the numbers (https://moodle.org/mod/forum/discuss.php?d=335357)

HTH,
Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: Slow file download and playing SCORM videos

by Nelson Moller -

Thanks Matteo,

I used my local disk because I was lazy smile .... one installation and all the tests could be done.

That behaviour was identified with memcached:1.4.37, I used the latest in the composer file again because of my laziness.

To arrive there, I've tried to use different db placements, different FS, apache an php confs...

 

In reply to Nelson Moller

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Nelson,

That behaviour was identified with memcached:1.4.37, I used the latest in the composer file again because of my laziness.

beware, I'm talking about daemon version here not the extension: https://github.com/php-memcached-dev/php-memcached/issues/367.

HTH,
Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: Slow file download and playing SCORM videos

by Nelson Moller -

I know,

I use pecl extension memcached 3.0.3 for php7.0.2 and 2.0.2 for php5.6.31, we were talking about the memcached server.

Thanks for the link, I was not aware of that.


N

In reply to Nelson Moller

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Nelson,
I've just given a try to your test case by summing up the two webserver into the same compose (same dataroot and same database) and by using 56 and 70 suffix, memcached included, on a CentOS7 based docker engine (17.09.0-ce) running on SSD.

While running the benchmark provides the expected results (lower is better):

  • 56 => Benchmark Score : 283 points
  • 70 => Benchmark Score : 262 points

the scorm module is actually too slow in starting on "70".
Will look to (try to) understand the reasons: could you share how the webserver image has been built?

TIA,
Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: Slow file download and playing SCORM videos

by Nelson Moller -

Thanks Matteo,

You are really kind. My Dockerfiles are available at

https://gist.github.com/nmoller

I build the final bundle in Jenkins (my building blocks are at https://github.com/nmoller/jenkins-dsl/blob/master/test1 ).

There is nothing special in my apache or php configs and they are the same.

Have a nice day.


N

In reply to Nelson Moller

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Nelson,
if you look at the Network traffic using e.g. Chrome (<F12>) you'll see that the TTFB for JS Trivant files is the culprit while HTML pages are served as fast as expected.
Not sure about the reasons right now, still a couple of possible cleanup:

HTH,
Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: Slow file download and playing SCORM videos

by Nelson Moller -

I've seen the chrome network cascade but I didnot get why that happens in php7.0. That's the elusive question smile

For the Dockerfile, if I did not put opcache explicitly, I did not get it to work. 

The entry point usually contains another command to build the fresh css when the container start (It was due to the time it take to build the css from the scss in the first 3.3 moodle), so I erased that but I did not do it completely... sorry. 

In reply to Sali Kaceli

Re: Slow file download and playing SCORM videos

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Sal,
performances in serving files from Moodle are strictly related to the database too.

What could be a win-win solution when serving files through Moodle, given that web server and db server settings are fine, is to use the X-Sendfile option at web server side, https://docs.moodle.org/33/en/Performance_recommendations#X-Sendfile (Worth reading the whole page): this will off-load PHP from serving the static file.

To get an idea of the areas to be improved in terms of performances, give https://moodle.org/plugins/report_benchmark a try: https://moodle.org/mod/forum/discuss.php?d=335357. This will give you pure numbers to be compared both with other systems and during the tuning path of your environment.

HTH,
Matteo