Moodle in English: Quizzes are rending Moodle useless :-(

Running redhat enterprise 7.1 64 bit (32 GB), PHP 7.1.x Moodle 3.4.7+, and MySQL 5.7.21 community edition (64 GB). All on VMWare. App & database server.

The quizzes seem to be the issue.. that is making Moodle almost useless LOL

Since Sept 12^th (or maybe earlier) a minimum of nine instructors have reported specifically on the Moodle quiz slowdown issue.

If they each have 20 students in their room during a quiz, that is 180 students affected by this. “Affected” could mean either frustrated and worried because their quiz completed *very* slowly (minutes per question, instead of seconds), or some of them are not able to complete at all due to a session time-out.

Here are my .cnf settings.. The DB Server has 64 GB or ram and CPU still hits 70%

How do I try and narrow down the problems with quizzes and where to start?

# this is read by the standalone daemon and embedded servers
[server]

[client]
default-character-set = utf8mb4
max_allowed_packet=3G

[mysql]
default-character-set = utf8mb4

[mysqld]
datadir=/ourdb/ourmysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/opt/rh/rh-mysql57/log/mysql/mysqld.log
pid-file=/var/run/rh-mysql57-mysqld/mysqld.pid

slow_query_log=1
slow_query_log_file=/var/opt/rh/rh-mysql57/log/mysql57-slow.log
long_query_time=1

innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_large_prefix

character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-character-set-client-handshake

innodb_buffer_pool_size=51G
max_allowed_packet=3G
innodb_log_buffer_size=32M
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_buffer_pool_instances=8
innodb_thread_concurrency=8
innodb_buffer_pool_dump_pct=75
innodb_adaptive_hash_index_parts=16
innodb_checksum_algorithm=crc32
innodb_page_cleaners=8
innodb_log_file_size=6G

# this is only for embedded server
[embedded]

平均分： -

Re: Quizzes are rending Moodle useless :-(

Justin Hunt - 2018年09月27日星期四 13:38

About 7 years ago I had Moodle performance issues, mainly with quizzes, on a very old server.

After a lot of digging into it, it was clear that the bottleneck was the CPU processing PHP. I got opcache (which caches the interpreted PHP) working properly and the difference was instantaneous and huge. Now Moodle recommends opcache. And I think that is the first thing you should check

https://docs.moodle.org/35/en/OPcache

平均分：Useful (2)

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年09月28日星期五 02:05

Hi Justin.

I already have it installed but do I need to "activate" it?

PHP 7.1.8 (cli) (built: Aug 8 2017 09:01:08) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.1.8, Copyright (c) 1999-2017, by Zend Technologies

平均分： -

Re: Quizzes are rending Moodle useless :-(

Justin Hunt - 2018年09月28日星期五 13:33

It needs to be enabled, yes. And it can be tweaked and tuned. If you search for "opcache" in the Moodle PHP info page, at

[your site]/admin/phpinfo.php

you will be able to see if its enabled and running. Just be sure that your server threading model actually makes this meaningful. Basically if your server is running CPanel then probably it won't be. And its a losing battle to try and get it working.

This is a little one page wonder that graphs the status of the cache. It or something like it can be helpful ....

https://github.com/rlerdorf/opcache-status/blob/master/opcache.php

I am not really that expert on this. There will be people here better than me .....

平均分：Useful (1)

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年09月29日星期六 08:08

Yes it is running

Zend OPcache

Opcode Caching	Up and Running
Optimization	Enabled
SHM Cache	Enabled
File Cache	Disabled
Startup	OK
Shared memory model	mmap
Cache hits	384528
Cache misses	2694
Used memory	81659032
Free memory	52558696
Wasted memory	0
Interned Strings Used memory	8253472
Interned Strings Free memory	135136
Cached scripts	2643
Cached keys	3085
Max keys	7963
OOM restarts	0
Hash keys restarts	0
Manual restarts	10

平均分： -

Re: Quizzes are rending Moodle useless :-(

Howard Miller - 2018年09月30日星期日 18:34

Let us know how you get on, but you'll be lucky if this is your issue.

Opcache tends to reduce the overall load on the system but I've not seen it make a great deal of difference to page load times.

Make sure you have caching (Moodle's caching) properly set up. Make sure you have properly tuned your database (lots of resources online).

平均分： -

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年10月1日星期一 03:40

@Howard.

I may want to tweak my .cnf settings as 51GB maybe overkill.

Also found something interesting... one of our users were using the sharing cart plugin to copy quizzes and sometimes it fails. I think the user does this during exam times and may have bogged down the application.. that may be the cause.

I will also clean up the foreign key violations as their are 66 of them and one of the has about 300,000 records that violate it... cleaning those up should help optimize the db I think

平均分： -

Re: Quizzes are rending Moodle useless :-(

Tim Hunt - 2018年10月2日星期二 22:17

Advice on Moodle OpCache tuning is here: https://docs.moodle.org/35/en/OPcache

平均分： -

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年10月11日星期四 01:36

Yes.. I did that as well its rearing its ugly head again.. makes moodle unusable.

Our stack is solid with Redhat linux, and separated the app server (32 GB) and db server (64 GB). I wonder if because we are using the "community" edition that it may hit a limit of some sort but I was told no.

平均分： -

Re: Quizzes are rending Moodle useless :-(

Jamie Biddulph - 2018年10月12日星期五 00:06

Are you able to check webserver logs, or browser developer tools network logs for the resources loading when accessing the quiz?

If there are any externally hosted resources that are failing/slow to load it could be slowing page loads dramatically.

Failing that there have been some other suggestions in this thread that may be useful

平均分： -

Re: Quizzes are rending Moodle useless :-(

John Moz - 2018年10月9日星期二 17:17

Hi Guys,

I think this is a bigger issue that needs to be investigated but most of the forum threads regarding quiz performance try to point to it being a server/DB issue. We have the same issue when we have several classrooms full of students doing quizzes. Moodle becomes completely unusable. No pages will load.

The server architecture / DB is not the issue. We are using Moodle 3.5.1 on a Windows Server 2012 R2 (64bit) with 47GB RAM, PHP 7.2.10 and MySQL 5.6.35. We regularly optimise the MySQL tables and we have followed performance recommendations including using OpCache. Our IT team have monitored server performance and the CPU / Memory / DB are not hitting any detectable limits during peak quiz times and yet Moodle still becomes unusable.

平均分： -

Re: Quizzes are rending Moodle useless :-(

Dan Marsden - 2018年10月9日星期二 18:06

A few things stick out in your post.

Windows.

You dont mention MUC.

Moodle is slow but your servers aren't hitting limits - this suggests your ar architecture is not configured to make full use of the resources available.

You might consider asking a Moodle Partner to help tune your system although you might have difficulty finding one who will help you with windows based architecture.

平均分：Useful (1)

Re: Quizzes are rending Moodle useless :-(

Visvanath Ratnaweera - 2018年10月9日星期二 19:27

If it is not the server, then it is the network. Did your IT team do a full server to client network benchmarking?

I know, the quizzes usually do not need much bandwidth. But one never knows. There may be other limiting factors in the network. More that once I've encountered network cabling from last century which were not only of old standards, but also rusted! And nowadays the wireless access brings peculiarities of its own.

平均分： -

Re: Quizzes are rending Moodle useless :-(

AL Rachels - 2018年10月10日星期三 02:27

Hi Duane,

One other possibility to consider...since you did not mention hard drives. What type of drives are you using? When it comes to a quiz, it is pretty easy to hit the read/write speed limits of a spinning drive for your data and make Moodle appear to slow to a crawl. A solid state drive helps a LOT.

平均分：Useful (2)

Re: Quizzes are rending Moodle useless :-(

John Moz - 2018年10月10日星期三 20:03

Thanks for the suggestions. There are options we haven't explored such as having a MemCache server. We will continue to work to improve the performance our side.

We are pro Moodle, we just thought if a few institutions are having a similar problem it might be worth looking into how quizzes are handled in the back-end.

平均分： -

Re: Quizzes are rending Moodle useless :-(

Séverin TERRIER - 2018年10月11日星期四 16:25

Hi,

I'm not sure i understand things well, but it seems that your problem is (only ?) when (lot of ?) students takes quizzes.

From your message, i don't really understand if it's 20 students or 180 students taking quizzes.

What we try to do, when we have lot of people that must take quizzes "at the same time" :

Avoid all big operations (like database dump, backups...)
Don't have all questions on one page, but several pages with less questions
Don't have all student validating quizzes at the same time (with a defined time limit), but rather try to make them begin with 5 minutes delay

HTH,
Séverin

平均分：Useful (3)

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年10月13日星期六 06:44

So do you stop the cron or just certain tasks? We run cron every minute

We have classes and they take their quizzes and up to about 40 people take a quiz per class so if there are say 10 classes that amounts to about 400 people taking their own quiz at same time

no clustering or load balancing but our db server has quite a lot of resources already.. 64 gb of ram

平均分： -

Re: Quizzes are rending Moodle useless :-(

Ken Task - 2018年10月13日星期六 21:12

I take it you have mysqltuner.pl installed and have run it for checking performance/config of DB server.

What does tuner say for this:

InnoDB buffer pool / data size

How did you determine the values for things like: innodb_buffer_pool_dump_pct=75

What does tuner say for "Slow queries" ... it gives a % and also some numbers like 0/3M

What does 'top' look like on the DB server?

'spirit of sharing', Ken

平均分： -

Re: Quizzes are rending Moodle useless :-(

Alain Raap - 2018年10月15日星期一 00:55

Hi Jerry,

We already had a chat via the messenger here. We experienced the same problems as you describe here and the biggest issues we had was with the network firewalls and load-balancer (F5). We had long query response times and found out with monitoring and researching Splunk logging that a timeout on the load-balancer was causing our long response times in Moodle.

We also had a url in the content that wasn't able to go outside (on the internet), and that finally caused one of our performance problems. Just a few ideas to troubleshoot. With a separated web and database server you sometimes have to dig in these kind of problems (network, caching or database). I suppose you're also using the Redis (or other) caching server on the webserver? Don't use Memcache with php7.1, it isn't supported anymore by PHP.

平均分：Useful (2)

Re: Quizzes are rending Moodle useless :-(

Visvanath Ratnaweera - 2018年10月15日星期一 03:40

Hi Alain

Very interesting. I wish, I knew the tools you mention: F5, https://www.f5.com/products/get-f5, is software, right? And Splunk, https://www.splunk.com/en_us/software/splunk-next.html, is this a "cloud" in the sense of CloudFlare, etc.?

And when you talk collectively, you mean both you and jerry share the cause?

平均分： -

Re: Quizzes are rending Moodle useless :-(

Alain Raap - 2018年10月15日星期一 04:43

Hi Visvanath,

F5 and Splunk are both software indeed.

I share with Jerry the same hardware and software configuration (Redhat 7 Linux), so I try to help to find the cause of his performance problems

平均分：Useful (1)

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年10月15日星期一 05:08

Interestingly we will use F5 staring next week and have just bought Splunk … Redis caching we don't have expertise on...

平均分： -

Re: Quizzes are rending Moodle useless :-(

Alain Raap - 2018年10月15日星期一 14:44

Another challenge to get Moodle running and performing well

Working in a large organisation myself, the challenge is to find the commitment and experience of other collegues to get things working smooth. Coorperation is the most important when you're dealing with issues like performance or configuration problems.

平均分：Useful (1)

[OT] More resources, less performance :-(

Visvanath Ratnaweera - 2018年10月15日星期一 14:51

Yes, yes. Large organization. The high-flying technical manager throws resources, that nobody has expertise on, killing the performance. Main thing they sound impressive!

平均分： -

Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年10月15日星期一 04:16

The performance was fine on the older computing resources actually LOL .. one server for both app and db and only 16 gb or ram

no caching on server side except opcache...

平均分： -

Re: Quizzes are rending Moodle useless :-(

Alain Raap - 2018年10月15日星期一 04:46

Try to install and configure redis caching, if necessary I can share my experiences

https://moodle.org/mod/forum/discuss.php?d=366556

平均分：Useful (1)

Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月3日星期六 06:55

Good news folks.

After much troubleshooting, I have found that the platform is solid (should be) and what was happening is that unoconv (or soffice.bin) is causing the lockup.

I've also added more cpu (from 2 to 4) as I noticed that ram was rarely utilized and mostly on cpu and have brought it down to normal stage now.

thanks to all who responded to my inquiry

平均分：Useful (8)

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Tim Hunt - 2018年11月5日星期一 16:22

Well done for getting the problem solved, and thank you for reporting back here when you identified the problem. That will help other people who have a similar problem and find this thread from Google.

平均分： -

Bad news - soffice.bin 100% cpu returned again - Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月8日星期四 02:33

https://moodle.org/mod/forum/discuss.php?d=378306

Nope..spoke too soon

Came back again

平均分： -

Re: Bad news - soffice.bin 100% cpu returned again - Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月9日星期五 03:24

might just write a script to kill the process or prevent it from running at startup/reboot..

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月7日星期三 03:08

Just to clarify that unconv is not the problem per se but soffice.bin that we installed to get unoconv was the cause...

next steps.. to improve even further.. use pp-fpm and trim standard log...can't use Nginx as we're not allowed LOL

also I may experiment with migrating MySQL to postgres… maybe

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Susan Mangan - 2018年11月21日星期三 09:25

Hey Jerry, I think I know you. ;)

When we moved over to load balanced system a couple of years ago, using glusterfs, our system slowed to a halt so we implemented a Memcached server for application cache and performance improved considerably.

Note: there are some issues with Memcached documented in the forums here (mostly with PHP 7) and apparently Redis is a better solution so if you are looking into adding a cache server then you may want to try Redis first.

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月22日星期四 02:01

You betcha..

I've increased the cpu to 4 core from 2 also just in case...

Yes I have been looking to try and setup Redis but mgmt may have other plans... also will look at passthrough drives even

Merry Christmas and all the best in the new year!

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Susan Mangan - 2018年11月22日星期四 09:27

Yes, back at you!! If you ever want to chat with our linux guy let me know and I'll forward you his info. Just today he came to me with another idea he wanted to try for our moodle environment to get away from gusterfs! We work together on most things but he is essentially the expert on all the back end stuff.

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Jerry Lau - 2018年11月25日星期日 09:32

thanks Susan.. could you email me his contact details at work?

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Visvanath Ratnaweera - 2018年11月22日星期四 02:41

Hi Susan

Dumb question: You said that the load balanced cluster of web servers was much slower than the previous plain server. Memcached improved it. Is it still slower than the original plain server?

平均分： -

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Susan Mangan - 2018年11月22日星期四 09:38

Not a dumb question at all! The only reason the new load balanced system slowed down in the first place was due to the shared file system which we didn't have previously. Hence having to implement a caching solution.

Server resources increased as well as our knowledge around fine tuning and configuration so our new environment is definitely much faster.

However with that said, if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.

平均分：Useful (2)

Re: Solved! - Re: Quizzes are rending Moodle useless :-(

Visvanath Ratnaweera - 2018年11月23日星期五 18:57

Hi Susan

You wrote:
> if I had to wager a guess I would say that we could set up a single plain server environment with increased resources and be able to achieve the same performance as we do now with 'the farm'.

Yes. People have exaggerated estimations of the load on their Moodle and start with clusters. Then their fears get confirmed, yes the cluster is slow. Then they look for bigger clusters.

Have shown more than once that a properly configured single Linux server does better - to the disappointment of all the parties involved.

平均分：Useful (1)