Quizzes are rending Moodle useless :-(

Quizzes are rending Moodle useless :-(

by Jerry Lau -
Number of replies: 37

Running redhat enterprise 7.1 64 bit (32 GB), PHP 7.1.x Moodle 3.4.7+, and MySQL 5.7.21 community edition (64 GB). All on VMWare. App & database server.

The quizzes seem to be the issue.. that is making Moodle almost useless LOL

Since Sept 12th (or maybe earlier) a minimum of nine instructors have reported specifically on the Moodle quiz slowdown issue.

If they each have 20 students in their room during a quiz, that is 180 students affected by this. “Affected” could mean either frustrated and worried because their quiz completed *very* slowly (minutes per question, instead of seconds), or some of them are not able to complete at all due to a session time-out.

Here are my .cnf settings.. The DB Server has 64 GB or ram and CPU still hits 70%

How do I try and narrow down the problems with quizzes and where to start?

# this is read by the standalone daemon and embedded servers
[server]

[client]
default-character-set = utf8mb4
max_allowed_packet=3G


[mysql]
default-character-set = utf8mb4

[mysqld]
datadir=/ourdb/ourmysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/opt/rh/rh-mysql57/log/mysql/mysqld.log
pid-file=/var/run/rh-mysql57-mysqld/mysqld.pid

slow_query_log=1
slow_query_log_file=/var/opt/rh/rh-mysql57/log/mysql57-slow.log
long_query_time=1

innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_large_prefix

character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-character-set-client-handshake

innodb_buffer_pool_size=51G
max_allowed_packet=3G
innodb_log_buffer_size=32M
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_buffer_pool_instances=8
innodb_thread_concurrency=8
innodb_buffer_pool_dump_pct=75
innodb_adaptive_hash_index_parts=16
innodb_checksum_algorithm=crc32
innodb_page_cleaners=8
innodb_log_file_size=6G

# this is only for embedded server
[embedded]



Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

About 7 years ago I had Moodle performance issues, mainly with quizzes, on a very old server. 

After a lot of digging into it, it was clear that the bottleneck was the CPU processing PHP. I got opcache (which caches the interpreted PHP)  working properly and the difference was instantaneous and huge. Now Moodle recommends opcache. And I think that is the first thing you should check

https://docs.moodle.org/35/en/OPcache

Average of ratings: Useful (2)
In reply to Justin Hunt

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Hi Justin.

I already have it installed but do I need to "activate" it?

PHP 7.1.8 (cli) (built: Aug  8 2017 09:01:08) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.8, Copyright (c) 1999-2017, by Zend Technologies

Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers

It needs to be enabled, yes. And it can be tweaked and tuned. If you search for "opcache" in the Moodle PHP info page, at

[your site]/admin/phpinfo.php

you will be able to see if its enabled and running. Just be sure that your server threading model actually makes this meaningful. Basically if your server is running CPanel then probably it won't be.  And its a losing battle to try and get it working. 

This is a little one page wonder that graphs the status of the cache. It or something like it can be helpful ....

https://github.com/rlerdorf/opcache-status/blob/master/opcache.php

I am not really that expert on this. There will be people here better than me .....

Average of ratings: Useful (1)
In reply to Justin Hunt

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Yes it is running

Zend OPcache

Opcode Caching Up and Running
Optimization Enabled
SHM Cache Enabled
File Cache Disabled
Startup OK
Shared memory model mmap
Cache hits 384528
Cache misses 2694
Used memory 81659032
Free memory 52558696
Wasted memory 0
Interned Strings Used memory 8253472
Interned Strings Free memory 135136
Cached scripts 2643
Cached keys 3085
Max keys 7963
OOM restarts 0
Hash keys restarts 0
Manual restarts 10

Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Let us know how you get on, but you'll be lucky if this is your issue. 

Opcache tends to reduce the overall load on the system but I've not seen it make a great deal of difference to page load times. 

Make sure you have caching (Moodle's caching) properly set up. Make sure you have properly tuned your database (lots of resources online). 

Average of ratings: -
In reply to Howard Miller

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

@Howard.

I may want to tweak my .cnf settings as 51GB maybe overkill.

Also found something interesting... one of our users were using the sharing cart plugin to copy quizzes and sometimes it fails. I think the user does this during exam times and may have bogged down the application.. that may be the cause.

I will also clean up the foreign key violations as their are 66 of them and one of the has about 300,000 records that violate it... cleaning those up should help optimize the db I think


Average of ratings: -
In reply to Justin Hunt

Re: Quizzes are rending Moodle useless :-(

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Advice on Moodle OpCache tuning is here: https://docs.moodle.org/35/en/OPcache

Average of ratings: -
In reply to Tim Hunt

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Yes.. I did that as well its rearing its ugly head again.. makes moodle unusable.

Our stack is solid with Redhat linux, and separated the app server (32 GB) and db server (64 GB). I wonder if because we are using the "community" edition that it may hit a limit of some sort but I was told no.


Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Jamie Biddulph -

Are you able to check webserver logs, or browser developer tools network logs for the resources loading when accessing the quiz?

If there are any externally hosted resources that are failing/slow to load it could be slowing page loads dramatically.

Failing that there have been some other suggestions in this thread that may be useful


Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by John Moz -

Hi Guys,

I think this is a bigger issue that needs to be investigated but most of the forum threads regarding quiz performance try to point to it being a server/DB issue. We have the same issue when we have several classrooms full of students doing quizzes. Moodle becomes completely unusable. No pages will load. 

The server architecture / DB is not the issue. We are using Moodle 3.5.1 on a Windows Server 2012 R2 (64bit) with 47GB RAM, PHP 7.2.10 and MySQL 5.6.35. We regularly optimise the MySQL tables and we have followed performance recommendations including using OpCache. Our IT team have monitored server performance and the CPU / Memory / DB are not hitting any detectable limits during peak quiz times and yet Moodle still becomes unusable. 


Average of ratings: -
In reply to John Moz

Re: Quizzes are rending Moodle useless :-(

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

A few things stick out in your post.

Windows.

You dont mention MUC.

Moodle is slow but your servers aren't hitting limits - this suggests your ar architecture is not configured to make full use of the resources available.

You might consider asking a Moodle Partner to help tune your system although you might have difficulty finding one who will help you with windows based architecture.

Average of ratings: Useful (1)
In reply to John Moz

Re: Quizzes are rending Moodle useless :-(

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
If it is not the server, then it is the network. Did your IT team do a full server to client network benchmarking?

I know, the quizzes usually do not need much bandwidth. But one never knows. There may be other limiting factors in the network. More that once I've encountered network cabling from last century which were not only of old standards, but also rusted! And nowadays the wireless access brings peculiarities of its own.
Average of ratings: -
In reply to John Moz

Re: Quizzes are rending Moodle useless :-(

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Duane,

One other possibility to consider...since you did not mention hard drives. What type of drives are you using? When it comes to a quiz, it is pretty easy to hit the read/write speed limits of a spinning drive for your data and make Moodle appear to slow to a crawl. A solid state drive helps a LOT.

Average of ratings: Useful (2)
In reply to John Moz

Re: Quizzes are rending Moodle useless :-(

by John Moz -

Thanks for the suggestions. There are options we haven't explored such as having a MemCache server. We will continue to work to improve the performance our side. 

We are pro Moodle, we just thought if a few institutions are having a similar problem it might be worth looking into how quizzes are handled in the back-end. 

Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Hi,

I'm not sure i understand things well, but it seems that your problem is (only ?) when (lot of ?) students takes quizzes.

From your message, i don't really understand if it's 20 students or 180 students taking quizzes.

What we try to do, when we have lot of people that must take quizzes "at the same time" :

  • Avoid all big operations (like database dump, backups...)
  • Don't have all questions on one page, but several pages with less questions
  • Don't have all student validating quizzes at the same time (with a defined time limit), but rather try to make them begin with 5 minutes delay

HTH,
Séverin

Average of ratings: Useful (3)
In reply to Séverin Terrier

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

So do you stop the cron or just certain tasks? We run cron every minute

We have classes and they take their quizzes and up to about 40 people take a quiz per class so if there are say 10 classes that amounts to about 400 people taking their own quiz at same time smile

no clustering or load balancing but our db server has quite a lot of resources already.. 64 gb of ram



Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Ken Task -
Picture of Particularly helpful Moodlers

I take it you have mysqltuner.pl installed and have run it for checking performance/config of DB server.

What does tuner say for this:

InnoDB buffer pool / data size

How did you determine the values for things like: innodb_buffer_pool_dump_pct=75

What does tuner say for "Slow queries" ... it gives a % and also some numbers like 0/3M

What does 'top' look like on the DB server?

'spirit of sharing', Ken


Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Alain Raap -
Picture of Particularly helpful Moodlers

Hi Jerry,

We already had a chat via the messenger here. We experienced the same problems as you describe here and the biggest issues we had was with the network firewalls and load-balancer (F5). We had long query response times and found out with monitoring and researching Splunk logging that a timeout on the load-balancer was causing our long response times in Moodle.

We also had a url in the content that wasn't able to go outside (on the internet), and that finally caused one of our performance problems. Just a few ideas to troubleshoot. With a separated web and database server you sometimes have to dig in these kind of problems (network, caching or database). I suppose you're also using the Redis (or other) caching server on the webserver? Don't use Memcache with php7.1, it isn't supported anymore by PHP.

Average of ratings: Useful (2)
In reply to Alain Raap

Re: Quizzes are rending Moodle useless :-(

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Alain

Very interesting. I wish, I knew the tools you mention: F5, https://www.f5.com/products/get-f5, is software, right? And Splunk, https://www.splunk.com/en_us/software/splunk-next.html, is this a "cloud" in the sense of CloudFlare, etc.?

And when you talk collectively, you mean both you and jerry share the cause?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Quizzes are rending Moodle useless :-(

by Alain Raap -
Picture of Particularly helpful Moodlers

Hi Visvanath,

F5 and Splunk are both software indeed.

I share with Jerry the same hardware and software configuration (Redhat 7 Linux), so I try to help to find the cause of his performance problems

Average of ratings: Useful (1)
In reply to Alain Raap

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Interestingly we will use F5 staring next week and have just bought Splunk … Redis caching we don't have expertise on...



Average of ratings: -
In reply to Jerry Lau

Re: Quizzes are rending Moodle useless :-(

by Alain Raap -
Picture of Particularly helpful Moodlers

Another challenge to get Moodle running and performing well wink

Working in a large organisation myself, the challenge is to find the commitment and experience of other collegues to get things working smooth. Coorperation is the most important when you're dealing with issues like performance or configuration problems.

Average of ratings: Useful (1)
In reply to Alain Raap

[OT] More resources, less performance :-(

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Yes, yes. Large organization. The high-flying technical manager throws resources, that nobody has expertise on, killing the performance. Main thing they sound impressive!
sad
Average of ratings: -
In reply to Alain Raap

Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

The performance was fine on the older computing resources actually LOL .. one server for both app and db and only 16 gb or ram


no caching on server side except opcache...


Average of ratings: -
In reply to Jerry Lau

Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Good news folks.

After much troubleshooting, I have found that the platform is solid (should be) and what was happening is that unoconv (or soffice.bin) is causing the lockup.

I've also added more cpu  (from 2 to 4) as I noticed that ram was rarely utilized and mostly on cpu and have brought it down to normal stage now.

thanks to all who responded to my inquiry



Average of ratings: Useful (8)
In reply to Jerry Lau

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Well done for getting the problem solved, and thank you for reporting back here when you identified the problem. That will help other people who have a similar problem and find this thread from Google. smile

Average of ratings: -
In reply to Tim Hunt

Bad news - soffice.bin 100% cpu returned again - Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

https://moodle.org/mod/forum/discuss.php?d=378306

Nope..spoke too soon sad

Came back again



Average of ratings: -
In reply to Jerry Lau

Re: Bad news - soffice.bin 100% cpu returned again - Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

might just write a script to kill the process or prevent it from running at startup/reboot..

Average of ratings: -
In reply to Jerry Lau

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

Just to clarify that unconv is not the problem per se but soffice.bin that we installed to get unoconv was the cause...


next steps.. to improve even further.. use pp-fpm and trim standard log...can't use Nginx as we're not allowed LOL

also I may experiment with migrating MySQL to postgres… maybe

Average of ratings: -
In reply to Jerry Lau

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Susan Mangan -

Hey Jerry, I think I know you. ;) 

When we moved over to load balanced system a couple of years ago, using glusterfs, our system slowed to a halt so we implemented a Memcached server for application cache and performance improved considerably.

Note: there are some issues with Memcached documented in the forums here (mostly with PHP 7) and apparently Redis is a better solution so if you are looking into adding a cache server then you may want to try Redis first.

Average of ratings: -
In reply to Susan Mangan

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

You betcha..

I've increased the cpu to 4 core from 2 also just in case...

Yes I have been looking to try and setup Redis but mgmt may have other plans... also will look at passthrough drives even smile

Merry Christmas and all the best in the new year!


Average of ratings: -
In reply to Jerry Lau

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Susan Mangan -

Yes, back at you!!  If you ever want to chat with our linux guy let me know and I'll forward you his info. Just today he came to me with another idea he wanted to try for our moodle environment to get away from gusterfs!  We work together on most things but he is essentially the expert on all the back end stuff.

Average of ratings: -
In reply to Susan Mangan

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Jerry Lau -

thanks Susan.. could you email me his contact details at work?



Average of ratings: -
In reply to Susan Mangan

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Susan

Dumb question: You said that the load balanced cluster of web servers was much slower than the previous plain server. Memcached improved it. Is it still slower than the original plain server?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Susan Mangan -

Not a dumb question at all!  The only reason the new load balanced system slowed down in the first place was due to the shared file system which we didn't have previously.  Hence having to implement a caching solution.

Server resources increased as well as our knowledge around fine tuning and configuration so our new environment is definitely much faster.

However with that said, if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.

Average of ratings: Useful (2)
In reply to Susan Mangan

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Susan

You wrote:
> if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.

Yes. People have exaggerated estimations of the load on their Moodle and start with clusters. Then their fears get confirmed, yes the cluster is slow. Then they look for bigger clusters.

Have shown more than once that a properly configured single Linux server does better - to the disappointment of all the parties involved.
wink
Average of ratings: Useful (1)