The quizzes seem to be the issue.. that is making Moodle almost useless LOL
Since Sept 12th (or maybe earlier) a minimum of nine instructors have reported
specifically on the Moodle quiz slowdown issue.
If they each have 20 students in their room during a quiz, that is 180 students affected by this. “Affected” could mean either frustrated and worried because their quiz completed *very* slowly (minutes per question, instead of seconds), or some of them are not able to complete at all due to a session time-out.
Here are my .cnf settings.. The DB Server has 64 GB or ram and CPU still hits 70%
How do I try and narrow down the problems with quizzes and where to start?
# this is read by the standalone daemon and embedded servers
default-character-set = utf8mb4
default-character-set = utf8mb4
innodb_file_format = Barracuda
innodb_file_per_table = 1
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
# this is only for embedded server
About 7 years ago I had Moodle performance issues, mainly with quizzes, on a very old server.
After a lot of digging into it, it was clear that the bottleneck was the CPU processing PHP. I got opcache (which caches the interpreted PHP) working properly and the difference was instantaneous and huge. Now Moodle recommends opcache. And I think that is the first thing you should check
I already have it installed but do I need to "activate" it?
PHP 7.1.8 (cli) (built: Aug 8 2017 09:01:08) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.1.8, Copyright (c) 1999-2017, by Zend Technologies
It needs to be enabled, yes. And it can be tweaked and tuned. If you search for "opcache" in the Moodle PHP info page, at
you will be able to see if its enabled and running. Just be sure that your server threading model actually makes this meaningful. Basically if your server is running CPanel then probably it won't be. And its a losing battle to try and get it working.
This is a little one page wonder that graphs the status of the cache. It or something like it can be helpful ....
I am not really that expert on this. There will be people here better than me .....
Yes it is running
|Opcode Caching||Up and Running|
|Shared memory model||mmap|
|Interned Strings Used memory||8253472|
|Interned Strings Free memory||135136|
|Hash keys restarts||0|
Let us know how you get on, but you'll be lucky if this is your issue.
Opcache tends to reduce the overall load on the system but I've not seen it make a great deal of difference to page load times.
Make sure you have caching (Moodle's caching) properly set up. Make sure you have properly tuned your database (lots of resources online).
I may want to tweak my .cnf settings as 51GB maybe overkill.
Also found something interesting... one of our users were using the sharing cart plugin to copy quizzes and sometimes it fails. I think the user does this during exam times and may have bogged down the application.. that may be the cause.
I will also clean up the foreign key violations as their are 66 of them and one of the has about 300,000 records that violate it... cleaning those up should help optimize the db I think
Yes.. I did that as well its rearing its ugly head again.. makes moodle unusable.
Our stack is solid with Redhat linux, and separated the app server (32 GB) and db server (64 GB). I wonder if because we are using the "community" edition that it may hit a limit of some sort but I was told no.
Are you able to check webserver logs, or browser developer tools network logs for the resources loading when accessing the quiz?
If there are any externally hosted resources that are failing/slow to load it could be slowing page loads dramatically.
Failing that there have been some other suggestions in this thread that may be useful
I think this is a bigger issue that needs to be investigated but most of the forum threads regarding quiz performance try to point to it being a server/DB issue. We have the same issue when we have several classrooms full of students doing quizzes. Moodle becomes completely unusable. No pages will load.
The server architecture / DB is not the issue. We are using Moodle 3.5.1 on a Windows Server 2012 R2 (64bit) with 47GB RAM, PHP 7.2.10 and MySQL 5.6.35. We regularly optimise the MySQL tables and we have followed performance recommendations including using OpCache. Our IT team have monitored server performance and the CPU / Memory / DB are not hitting any detectable limits during peak quiz times and yet Moodle still becomes unusable.
A few things stick out in your post.
You dont mention MUC.
Moodle is slow but your servers aren't hitting limits - this suggests your ar architecture is not configured to make full use of the resources available.
You might consider asking a Moodle Partner to help tune your system although you might have difficulty finding one who will help you with windows based architecture.
I know, the quizzes usually do not need much bandwidth. But one never knows. There may be other limiting factors in the network. More that once I've encountered network cabling from last century which were not only of old standards, but also rusted! And nowadays the wireless access brings peculiarities of its own.
One other possibility to consider...since you did not mention hard drives. What type of drives are you using? When it comes to a quiz, it is pretty easy to hit the read/write speed limits of a spinning drive for your data and make Moodle appear to slow to a crawl. A solid state drive helps a LOT.
Thanks for the suggestions. There are options we haven't explored such as having a MemCache server. We will continue to work to improve the performance our side.
We are pro Moodle, we just thought if a few institutions are having a similar problem it might be worth looking into how quizzes are handled in the back-end.
I'm not sure i understand things well, but it seems that your problem is (only ?) when (lot of ?) students takes quizzes.
From your message, i don't really understand if it's 20 students or 180 students taking quizzes.
What we try to do, when we have lot of people that must take quizzes "at the same time" :
- Avoid all big operations (like database dump, backups...)
- Don't have all questions on one page, but several pages with less questions
- Don't have all student validating quizzes at the same time (with a defined time limit), but rather try to make them begin with 5 minutes delay
So do you stop the cron or just certain tasks? We run cron every minute
We have classes and they take their quizzes and up to about 40 people take a quiz per class so if there are say 10 classes that amounts to about 400 people taking their own quiz at same time
no clustering or load balancing but our db server has quite a lot of resources already.. 64 gb of ram
I take it you have mysqltuner.pl installed and have run it for checking performance/config of DB server.
What does tuner say for this:
InnoDB buffer pool / data size
How did you determine the values for things like: innodb_buffer_pool_dump_pct=75
What does tuner say for "Slow queries" ... it gives a % and also some numbers like 0/3M
What does 'top' look like on the DB server?
'spirit of sharing', Ken
We already had a chat via the messenger here. We experienced the same problems as you describe here and the biggest issues we had was with the network firewalls and load-balancer (F5). We had long query response times and found out with monitoring and researching Splunk logging that a timeout on the load-balancer was causing our long response times in Moodle.
We also had a url in the content that wasn't able to go outside (on the internet), and that finally caused one of our performance problems. Just a few ideas to troubleshoot. With a separated web and database server you sometimes have to dig in these kind of problems (network, caching or database). I suppose you're also using the Redis (or other) caching server on the webserver? Don't use Memcache with php7.1, it isn't supported anymore by PHP.
Very interesting. I wish, I knew the tools you mention: F5, https://www.f5.com/products/get-f5, is software, right? And Splunk, https://www.splunk.com/en_us/software/splunk-next.html, is this a "cloud" in the sense of CloudFlare, etc.?
And when you talk collectively, you mean both you and jerry share the cause?
Interestingly we will use F5 staring next week and have just bought Splunk … Redis caching we don't have expertise on...
Another challenge to get Moodle running and performing well
Working in a large organisation myself, the challenge is to find the commitment and experience of other collegues to get things working smooth. Coorperation is the most important when you're dealing with issues like performance or configuration problems.
The performance was fine on the older computing resources actually LOL .. one server for both app and db and only 16 gb or ram
no caching on server side except opcache...
Good news folks.
After much troubleshooting, I have found that the platform is solid (should be) and what was happening is that unoconv (or soffice.bin) is causing the lockup.
I've also added more cpu (from 2 to 4) as I noticed that ram was rarely utilized and mostly on cpu and have brought it down to normal stage now.
thanks to all who responded to my inquiry
might just write a script to kill the process or prevent it from running at startup/reboot..
Just to clarify that unconv is not the problem per se but soffice.bin that we installed to get unoconv was the cause...
next steps.. to improve even further.. use pp-fpm and trim standard log...can't use Nginx as we're not allowed LOL
also I may experiment with migrating MySQL to postgres… maybe
Hey Jerry, I think I know you. ;)
When we moved over to load balanced system a couple of years ago, using glusterfs, our system slowed to a halt so we implemented a Memcached server for application cache and performance improved considerably.
Note: there are some issues with Memcached documented in the forums here (mostly with PHP 7) and apparently Redis is a better solution so if you are looking into adding a cache server then you may want to try Redis first.
I've increased the cpu to 4 core from 2 also just in case...
Yes I have been looking to try and setup Redis but mgmt may have other plans... also will look at passthrough drives even
Merry Christmas and all the best in the new year!
Yes, back at you!! If you ever want to chat with our linux guy let me know and I'll forward you his info. Just today he came to me with another idea he wanted to try for our moodle environment to get away from gusterfs! We work together on most things but he is essentially the expert on all the back end stuff.
thanks Susan.. could you email me his contact details at work?
Not a dumb question at all! The only reason the new load balanced system slowed down in the first place was due to the shared file system which we didn't have previously. Hence having to implement a caching solution.
Server resources increased as well as our knowledge around fine tuning and configuration so our new environment is definitely much faster.
However with that said, if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.
> if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.
Yes. People have exaggerated estimations of the load on their Moodle and start with clusters. Then their fears get confirmed, yes the cluster is slow. Then they look for bigger clusters.
Have shown more than once that a properly configured single Linux server does better - to the disappointment of all the parties involved.