Moodle For 10000 concurrency especially for Quiz

Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -
Number of replies: 12

Hi All,

Moodle 3.1, Apache 2.4 and MySql DB.

We have a load balancer with,

2 app servers each 128 GB ram and 12 core CPU's

1 DB server with 128 GB ram and  12 core CPU's

Moodle Data is on NFS. MemCache, OPCache and APC are enabled.

For 1000 threads and 600 ram-up , while testing Moodle application with JMeter the average responsive time is close to 7 secs. Anything higher than 1000 threads is timing out or waiting for 30 secs to get server response.

Basically we are trying to make the above server config to work for 10,000 concurrency with total users around 100,000. But for some reason we are not able to achieve that concurrency.

Any help in this regard would be appreciated.

Thank you.

Average of ratings: -
In reply to Shyam Jadala

Re: Moodle For 10000 concurrency especially for Quiz

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

How did you work out you needed 10000 concurrency?   That's *huge* number and, with respect, is so unlikely you really need to justify it - if not to us, to yourself. 

Get that wrong by orders of magnitude and you're going to spend huge sums of money you didn't need to or it just won't work. 

Is Moodle's caching definitely configured properly? Presumably you are monitoring your servers. What are they doing when you get these long waits? What resources are being used up?

Average of ratings: -
In reply to Howard Miller

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Thank you for the Reply Howard.

Client conducts almost 1000 batches at different locations and each batch is of 15 users. So in total almost 15000 users using Moodle and taking their quiz at the same.

APC and OPCache are configured properly and also allocated 10GB for APC. And we are able to see the hits on APC too. And coming to monitoring servers, we observed very low CPU usage and LOW RAM usage. Users are timedout after 20 secs.


Since Apache 2.4 is not allowing us to configure mpm_worker module, we configured mpm_prefork with below settings, we need someone to validate these "prefork" setting.

<IfModule mpm_prefork_module>     

   ServerLimit      2400     

  StartServers      5     

  MinSpareServers   75     

  MaxSpareServers   800     

  MaxRequestWorkers 2400     

  MaxRequestsPerChild  0 

</IfModule>


The error we are receiving while configuring mpm_worker module is given below, any idea on how to fix this issue may also help us to some extent.

AH00526: Syntax error on line 31 of /etc/httpd/conf.d/php.conf:

Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration


We are about to make session persistence / sticky session with round robin on load balancer so that a user hits the same app server till session timeouts. And also moving cache to local app servers from NFS and will keep only "filedir" and "session" directories on NFS as is. 


Hope this helps in understanding our current configuration. 

Average of ratings: -
In reply to Shyam Jadala

Re: Moodle For 10000 concurrency especially for Quiz

by Chris Fryer -

The error you get when using the Worker module is because worker has no knowledge of PHP, and the php_value directive is for PHP.  Prefork understands that because PHP is loaded as a module.

Don't use prefork if you are interested in performance.

https://2bits.com/articles/high-performance-drupal-with-apache-mpm-worker-threaded-server-and-php-fpm.html

Average of ratings: -
In reply to Chris Fryer

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Chris, Thanks for the advice. 

Sure, will check the link you shared and see what we can do.

Average of ratings: -
In reply to Chris Fryer

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Chris,

We are on centOS 7 and Apache 2.4 and PHP 5.4.

Will that be a constraint to enable mpm_worker. When we are trying to enable, thats when its throwing that error shared earlier.

Average of ratings: -
In reply to Shyam Jadala

Re: Moodle For 10000 concurrency especially for Quiz

by Chris Fryer -

You should edit  /etc/httpd/conf.d/php.conf  and ensure that everything is wrapped in these tags:

<IfModule php5_module>
  php_value blah
</IfModule>

This ensures the directives are only read when you load PHP as an Apache module.  It will make the errors go away when you enable the worker module.

The following guide for CentOS is nearly 4 years old, but should help you:

https://www.garron.me/en/linux/apache-mpm-worker-php-fpm-mysql-centos.html

Average of ratings: -
In reply to Chris Fryer

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Chris,

CentOS 7 was not allowing us to install yum in it so could not complete the steps from the link you shared. But our sys admin found another solution (given solution link below) that worked for us.

https://www.server-world.info/en/note?os=CentOS_7&p=httpd&f=19

Hope the above link would help someone looking for similar issue.

Will keep posting the updates on this going forward.

Average of ratings: -
In reply to Shyam Jadala

Re: Moodle For 10000 concurrency especially for Quiz

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

No... I'm talking about Moodle's Cache. Site administration > Plugins > Cacheing. 

If you are using NFS then Moodle will try to cache to NFS by default and this is usually a very bad thing performance wise. Set up something like Memcached. This is nothing to do with OpCache. 

I'm going to suggest that a system that can start 15,000 quiz attempts at the same time and not fall over would be massive. When you quote for it the client will laugh. I administer some systems much larger than yours and we start to worry when quiz attempts number the low 100s. 

It's only fair to tell you that you are attempting something that - if it works exactly as - you describe will be incredibly difficult/expensive. On the other hand, don't confuse "marketing" numbers with reality. They are rarely the same. 

Average of ratings: -
In reply to Howard Miller

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Howard,  

Please find the screenshot of caching status.

Moodle Cache


As I mentioned earlier we are moving all cache folders from NFS to local app server(s) and will keep filedir and trashdir in NFS only.  And also enabling sticky session on loadbalancer.

Average of ratings: -
In reply to Shyam Jadala

Re: Moodle For 10000 concurrency especially for Quiz

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

You can't do that... many of the cache's have to be on a shared cache. Only a few will work with local cache folders. 

Unless you know your NFS to be fast (and it won't be) this is why you will be seeing slow page loads currently. 

When Moodle is load balanced you have to get the caching right or Moodle runs very badly. 

Average of ratings: Useful (1)
In reply to Howard Miller

Re: Moodle For 10000 concurrency especially for Quiz

by Shyam Jadala -

Thank you for your suggestion Howard.

We haven't yet moved all cache folders to local app servers yet.

Currently, "localcache" is the only cache folder which is mapped to local app server rest are on NFS only.

Average of ratings: -