Can you tell us a something about your Moodle configuration? The version would be a good start
Moodle2.7.1+ (Build: 20140731)
This was a problem with 2.5.1+ (Build: 20130712) as well, I ran that version last year.
Moodle running on a semi-dedicated server. There are a few other things running on it but it's pretty much 90% one large moodle instance. PHP 5.6 with OPCache enabled. 32GB of memory, 2 Xenon processors and an SSD drive. It's not a server issue. As I said, had the same problem with 2.5.1+ (Build: 20130712) on completely different (but similar speced) server last year.
I reinstall a clean version of Moodle every year on a clean database, so there isn't some legacy data/crud left over from last year, nor crud from in place upgrades from previous versions.
What's your operating system?
As you don't know what the problem is how are you so sure it's not an operating system problem?
Where are you writing your 'moodledata' files? Is it all to your SSD drive or to somewhere else?
Centos 6.5
The operating system doesn't create 200+ database connections in the space of a few seconds.
Everything is on the SSD.
Last year: No SSD, different server, different ISP, different data center, Centos 6.4 -- Same problem.
Have you checked the full pluginfile url when this happens? Like I think Tim alludes to, what is is doing? This might give you a clue what is going on here.
That's a good question. The answer is no.
It's possible that there is a single file causing this but, if so, I have no idea which one. The system is used by thousands of students and teachers across the US. So I have no way of knowing who is clicking on what at any given time.
Is there a way to have the system log all pluginfile.php accesses to a file or database, with the URL and a date/time stamp?
Maybe?
tail -f /var/log/httpd/access_log |grep pluginfile.php
could add a redirect to a .log file
could import that .log file to something local for inspecting further - ie, the links become clickable.
'spirit of sharing', Ken
Hi Ken,
Thanks for the suggestion. I grep'ed the access_log file but no mention of pluginfile.php in there.
Then you're looking at the wrong access log or logging is disabled in some way in your web server.
Huh? Well, there's a bunch in all servers that I help manage!!!
Example: (with info changed):
70.x.x.x - - [12/Sep/2014:05:27:53 -0500] "GET /somemoodle/pluginfile.php/2/course/section/2/owl-binocs.jpg HTTP/1.1" 304 - "http://someserver/moodle26/" "Mozilla/5.0 (Macintosh; Intel Mac blah, blah,blah"
Also goes for any images shown on the front page.
What does:
fgrep LogFormat /etc/httpd/conf/httpd.conf
show ya?
Think searching logs for your site might show some info that would enable you to investigate courses/files that are being hit at that time of day (ie, your spikes).
'spirit of sharing', Ken
pluginfile.php is the script that is responsible for letting users download any uploaded file.
This mightinclude any large documents or videos there might be on your site.
So, if lots of people are downloading big files, and the average page-load time goes to 30 seconds, then that is not necessarily a problem.
Nor does it fully explain the spike in pluginfile.php.
Let's say that I have a big file (and I have some big but not huge files, no video but some 20Meg PPTs). And 50 people all decide to download it. Unless all 50 are synchronizing their click to download that file, it would not explain the spike. That would cause lots of usage over a few minutes time. Not a single spike over a few seconds of time. It takes longer than that to download a 20Meg file.
Hi Phil,
you should track down even the full PATH_INFO data since pluginfile.php does some generic job then relies on a per-module logic to do few/many extra checks, usually on the database: this will help in identifying the module and the related DB queries. Besides you should log the slow queries to help the troubleshooting.
Spikes on the database could be also related to concurrency and slow connections in the front-end (your web server): here you could improve the performances by using X-Sendfile.
HTH,
Matteo
We have experienced similar problems with pluginfile.php. In our case, these problems appeared to be related to the use of browser download accelerators. We reproduced the problem with one named "DownThemAll!".
What happens in these scenarios is that the browser extension sends many simultaneous requests for chunks of the file. Requests for session locks then start piling up. As many hundreds of requests pile up, bad things start to happen in various places.
Until we can find a better solution, we have set $CFG->disablebyteserving = true;
Hi Phil,
Did you find a solution to the problem? We have a similar problem.
We altso have these spikes which results in a very slow moodle site. The spikes takes 4-5 minuts. We're able to reproduce the error when we try to install a plugin via Siteadministration. When install is pressed it is like the sites times out and after 4-5 minuts it runs fine again. When we upgrade the database (like the wizard tells us to do) the site slows down once again, 4-5 minuts.
We run Moodle 2.7.4+ on a CentOS 6.4. PHP version is 5.4.37.
The error occured after a upgrade - where we upgraded Moodle and PHP. We have a similar server, only where the PHP version is different (v5.3.X) and on this server there are no problem when installing plugins or spikes.
Im very keen to know if you solved your issue and how you did it.
Thank you,
-- Troels
Try something, and let us know what happens...
In your config.php file, add the line...
$CFG->disablebyteserving = true;
Site is still very slow when applying this fix and installing / removing a plugin. I did restart the http service.
Troels
Qualifications for this response ... support several entities with CentOS/Moodle servers remotely.
Some additional shared information might be needed to help.
1. Are the servers remotely hosted (RackSpace/Other) or locally hosted?
2. Are the CentOS/Moodle servers in a virtualized environment ... ie, VMWare/other?
Comment as to why the questions above: CentOS 6.4 is .2 versions behind. PHP could be higher as well.
The Virtualized environment adds a layer above CentOS/Moodle which could be the culprit. What does the hardware look like especially memory allocated? (note: when one looks at "top" for example, one is seeing the memory allocated by the hosting environment ... ie, the VMWare config of the server instance.
Sounds like the DB server is on the same box as the Apache. That true?
What is version of MySQL and what does it's configuration look like ... bottle neck might be the DB server and config of that.
Do you have mysqltuner.pl installed on the troubled server? IF the issue is NOT VMWare (it could be) then one needs to check for 'bottlenecks' that cause the slowness/un-responsiveness. One of those bottle necks might be the DB config itself - many factors could be involved.
Is Apache configured to use mod or fastcgi or other?
Have you checked the server logs messages/secure/access_log/error_log for clues?
With this sort of issue, the bottle neck/issues of un-responsiveness could be Virtual OS, Config of Apache/PHP/MySQL either individually or in combination with one another. If DB server is dedicated and Apache/Moodle on another server, could be a networking issue (network comes before application in the ISO model). In other words, there is no one simple answer. :| Case in point: spent 6 months with an entity going over the config of everything on a Virtualized (VMWare) CentOS 6.x/Moodle 2.x/PHP 5.5/MySQL 5.5.25 box that every hour at 32 minutes to 35 mintues after the hour would be slow ... even un-responsive. Turned out to be a VMWare issue ... another guest OS on that physical server had been given unlimited usage of the physical resources and when it did whatever it was to do, ALL other guest OS's on that server, would be un-responsive. Guest OS's ... still up, still running, guest OS not reporting any issues internally - but access to it, choked by the other system using up all resources.
'spirit of sharing', Ken
Hi Ken,
1. Servers are locally hosted...
2. ... in a VMWare environment.
We have a test server running exactly the same CentOS and Moodle version which has no performance issues when installing or removing plugins - The test server is also running in our virtual environment.
The DB server is hosted on its own VM. Actually I just found out that our test DB server is running a newer version of Mysql so I will try to update the production server to see if the Mysql version is the bottleneck. I will get back to you when the updates have been applied.
Thank you for your help,
Troels