Performance with 2500 concurrent users

Performance with 2500 concurrent users

by Stephen Tod -
Number of replies: 5

Hello everyone, 


Our university has a Moodle 3.2.5 installation with MSSQL as a DB (our only choice). 

We have currently 60 000 registered users and a daily average of 2500 concurrent ones. 

We have on premise hosting at the campus with: 

   

  • VMWare virtual host with Red Hat 7.5
  • Prozessors 8
  • Memory 8GB
  • Disk Space 64GB - application
  • Apache version is 2.4.27
  • PHP version 7.1.8
  • Microsoft SQL Server 2016 with 20GB RAM


Because of the number of cohorts and completions the system is having a bad time with the performance. Especially the scheduled tasks take a lot of time to complete. Another example is loading of the user list - it takes about 6 seconds to load the page. 


The system architecture is pretty monolithic. We are expecting the system to get another 50 000 users from another university and online programs. So at the end we will end with >100 000 users. 


Our target would be to setup everything in two phases: 


  1. Stabilization: It seems, that the base of the hosting architecture is not enough for the current users.  What would be the first step from your point of view? (I alredy read https://moodle.org/mod/forum/discuss.php?d=161040, but it confuses me a bit)
  2. Further development of the solution: Is there a way to make everything more scalable? Or best practices for doing it, as there is a ton of info in the web, but everything is different. 


Our first suggestion was a partner, but sadly we must do it with internal forces, because of the lack of budget. So any feedback will be helpfull. 


BR, 


Stephen. 

Average of ratings: Useful (1)
In reply to Stephen Tod

Re: Performance with 2500 concurrent users

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

I have to admit it's really weird to see that you are embracing linux with Red Hat and Apache - but still trying to force it through MS SQL! smile

I haven't seen an MS SQL based solution scale with that level of concurrency before - the biggest issue I've had in the past (it's been some years since I've been forced to do this) is with the MS SQL driver for PHP being really slow. You can manage if you're just using it to connect to an external db enrolment or sync process, but running Moodle on MS SQL for that number of users is something you should really avoid.

If you have some really good internal PHP Developers/Dev ops staff - they may be able to improve the performance of the MS SQL driver (there are a lot of different ways to connect PHP to MS SQL), but you're likely going to need a lot more time from them than it would just spinning up postgres or mysql on your red hat server.

You also make no mention of MUC - make sure you set up a redis cache - with the number of users you are working with you will need this no matter what DB type you use:
https://docs.moodle.org/36/en/Redis_cache_store

Also - please scare the staff that are responsible for keeping you on 3.2 and tell them that your students can exploit a number of well known public security vulnerabilities including privilege escalation (allowing student accounts to gain access to admin accounts) and Remote Code Execution vulnerabilities that allow for your servers to be completely compromised. There really is no excuse for patching web applications with publicised security vulnerabilities.

Good luck!

Average of ratings: Useful (3)
In reply to Stephen Tod

Re: Performance with 2500 concurrent users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

My immediate thought was, you pull such a big load with Windows? With Apache and MS SQL? Only then realized that the web server is Linux. What about MS SQL? Is it on a separate Windows Server or you run the Linux version of MS SQL on the same Linux VM, which is a 8 vCPU / 8 GB? If that is so, all speaks for the power of Linux! What is the virtualization technology, BTW: VMware? What OS? What is the host disk system?

Anyway, I see that you have received great advice specific to your situation. (Your "only choice", which means "no choice".) About the general questions you asked: I have noticed that you joined the forum just now. Please take some time to get the background of what I am talking about. Your initial pointers are in the entry page to this forum https://moodle.org/mod/forum/post.php?forum=94. The thread you have mentioned is outdated. Moodle has come a long way since the year 2010! You will find more applicable threads through entry page mentioned earlier.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Performance with 2500 concurrent users

by Stephen Tod -

Hi Dan and Visvanath, 


Than you both for the answers. 


@Dan: 

Sadly, we have to go further with MS SQL. I will take your recommendation and look into the performance optimization of the MS SQL driver for PHP and also look into MUC. 

Regarding the version, the first thing on our roadmap is the update to the newest available Moodle stable version. 

@Visvanath:

MSSQL is on separate server. So we have: 

  • Webserver - VMWare with 8GB RAM, REDHat 7.5
  • MSSQL server 20GB RAM


I will keep it for the sence of documentation here updated, in order to help further for other Moodle users. 


BR

Average of ratings: -
In reply to Stephen Tod

Re: Performance with 2500 concurrent users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Stephen

Here are some more pieces of information that could be useful to the others:
- Virtualization host(s)
-- OS? You said VMware, but is it ESX? Version?
-- Hardware? Specially the disc system. Are they fast drives or even SSD?
-- Are both VMs on the same virtulization host?
-- Any other VMs running on it?

- Database server
-- vCPUs of the database server?
-- OS of the database server? You mentioned SQL Server 2016, so perhaps Windows Server 2016. Which release/version?
-- Is it shared? I mean, does it serve applications other than this Moodle?

As you may have noticed, Windows is not my world.
wink
Average of ratings: -
In reply to Stephen Tod

Re: Performance with 2500 concurrent users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm just going to throw this in here as we hear it quite a lot... "our local policies / my manager say that we have to use xyz". Insert your choice of sub-optimal system. 

Moodle is mostly developed and mostly implemented on 'popular' Linux distros (effectively, Ubuntu and CentOS), using Apache and NginX and using MySQL or Postgres for a database. Although other possibilities are available and technically supported they will not be as good.

Your easiest technical option is always to use a an "optimal" configuration. Before you do anything else. Or if you prefer, "use what everybody else uses".

Average of ratings: Useful (4)