Performance Tuning and Load Testing

Performance Tuning and Load Testing

by Indresh Vishwakarma -
Number of replies: 18

Hi,

This is my first post here, Please execuse if i am writting at wrong place and point me to correct forum or post.

 

I am working on performance tuning and load testing on one of our installation and trying to achieve access of 200 to 500 concurrent user using LoadRunner tool.

I have done some changes in Apache config file and enabled opCache in moodle and mod_deflate in apache to compress the response. But it seems that nothing is working. My average cpu utilization in above 70% and response time is also too high.

My average cpu utilization in above 70% and response time is also too high.

Probably i am missing something which i am unable to figure out. 

The system i am using is:

Moodle Version - 3.0.2, Apache - 2.4, PHP -  5.6.8

Server Configuration :

OS - Windows Server 2008 R2, Processor - Intel Xeon 2.30Ghz (4cpu,) Ram - 16BG,

Please anyone help me to put proper configuration settings to achieve 200 to 500 concurrent user access. 

 

Thanks..

 

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Moving to Hardware and performance forum...

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Can you explain what your '200 to 500 concurrent user access' looks like? How are you measuring that?

500 users actively using Moodle at almost exactly the same time (i.e. that many Apache processes at the same time) is huge and you almost certainly don't have enough hardware. Even worse as it's Windows. Even worse again as you are using Apache rather than IIS - I'm not a Windows user myself but it seems to be generally accepted that you are better using the native software.

Usman will be along in a minute to give a more comprehensive answer wink

Average of ratings: -
In reply to Howard Miller

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Hi Miller,

Thanks for your reply and input.

Our testing team have written a script in LoadRunner tool for 200 users accessing different activities E.g quiz, downloading some pdf files, accessing scorm etc. In this tool we are analysing CPU utilization, response time and others details like error and all.

would want to know that is it possible to reach 200 concurrent users having above mentioned hardware configuration ?

 And thanks for pointing that we have worse hardware configration :/ .

Please help in providing some more input and will be waiting for some windows user.

 

Thanks..

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Usman Asar -
Picture of Plugin developers Picture of Testers

Indresh, your problem is evident.

Running Apache on Windows Platform. either run Linux + Apache or Windows + IIS, if properly tuned (Both hardware & software), @ 500 user load, your CPU will barely exceed 40% of continued utilization.

BUT

It's not just always CPU + Memory that defines your performance for system, let me know your hard drive configuration as well, also if you can, exact model of CPU, as there had been many generations of XEON CPU's and every new has slightly better performance.

If, you'll be able to add another SSD for database + tmp directory, expect 100% boost in performance, and SSD's aren't that expensive anyway.

Also, is your environment based on Virtual Setup or its a dedicated hardware?

If you want, I can remotely have a look into your server and may will setup from scratch should it need be with IIS acting as web server.

Average of ratings: -
In reply to Usman Asar

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Hi Usman,

Thanks for the information.

Actually it is an existing setup on windows using XAMPP.

I will provide other details you asked for once will be in office tommorrow.


I would want to share the screen and lean from you but my company will not allow because I have to follow the procedure. Apology ..!!


Thanks ..

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

DON'T use Xampp for a production server. It's only designed for development and testing. It's not secure or tuned for live use. That's going to be another big problem. 

Average of ratings: Useful (1)
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Grant Mucha -

I agree completely with Howard.

You are already running 3.0.6 you likely should make the move to 3.1.x

You are using PHP 5.6.x you likely should make the move to PHP 7.0.x (performance boost)

You CPU utilization is likely due to IO wait time and the fact your not using SSD on the backend. I'm not at all familiar with Windows so you will have to find your own tool to monitor. However, I have seen this on more than one occasion on linux servers lacking the proper hard-drives.

@Usman Asar

Mentions using "another" SSD, however I suspect you may not even be using one currently. You really need to get an SSD optimized hard-drive configuration to handle the IOPS required by Moodle.

XAMPP may also be a source of your problems as that is not really intended for production.



Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Usman Asar -
Picture of Plugin developers Picture of Testers

Indresh, ok no issues with that, you MAY have read  documentation for XAMPP, and like Howard mentioned, it's made for development environment, and not for production.

Though you already have plenty of advices as mentioned by Howard and Grant, if your whole moodle isnt running on SSD which I am sure it wont be, at least get a smallest SSD for Database, for moodle requires high database throughput, even better if you can put everything on SSD, but even in that case keep D/B separate.

Dont worry about exact CPU version, 16GB of RAM is PLENTY to serve even 2000 concurrent without hiccups, SSD and proper tuning is the key together with use of PHP 7.0 for Moodle 3.0.1 onwards, as PHP 7 really reduces memory footprint and renders pages faster.

If your company cannot give access to server still no issues just follow along with my video tutorial here

http://blog.academictools.org/2015/12/installing-moodle-on-microsoft-windows.html

This though is made on 2012 R2 server, but you wont have any much difference, for IIS started processing PHP as FastCGI since 2008 R2 server, and this is what you need for a performing web server on PHP.

As an additional step, you can as well install WinCache together with Zend OpCache, as under load WinCache really shines up.

you'll be doing exact same things like showed  in tutorial (means not checking anything extra, not even ISAPI and ISAPI filters), just CGI is good enough.

Rest, if any other issues, let us know

Average of ratings: -
In reply to Usman Asar

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Thanks Usman,

I'll check your tutorial and will be doing the same, meanwhile i am getting details from my infra team regarding SSD if we have used and will update the details here once input recieved.

i have enabled zend OpCache and will check winCache.

 

Thanks all for your support smile

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Hello All,

Thank you for your recomndation. we will surely work on this

Also we would like to know your recommendation on our existing setup.

Is there any way out so we can work on existing setup to increase the performance (like tuning on APACHE, PHP and Moodle) ????

Some Information already shared as mentioned below,

We have windows server 2008 r8 with processor Xeon v3 (4 CPU) 2.30 Ghz, 16 GB RAM , VHD, on virtual machine.

On this setup we have MySQL ver. 5.6.32 and Moodle 3.0 with XAMPP having Apache 2.4 and PHP 5.6.24.

Our observations on Load testing: We scaled up till 100 users without errors but with high 90th percentile response time(>40sec) and CPU utilization > 70%.

 

 Thanks in advance.

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Usman Asar -
Picture of Plugin developers Picture of Testers

OK it's a Virtual machine, means there is nothing you an do in regards to installing another SSD.

Still no problem, rather than using Apache, use IIS as web server. I had recently encountered a case with a US school district, they were on XAMPP and their moodle was struggling, So i assisted them in setting up IIS and was later forwarded a message from student telling it's incredible speed difference, that was back in April and since hen plenty of quizzes went through where I had monitored some to see if more resources needed allocation, but it wasnt the case and  recently I was told that the moodle is running absolutely fine since then, they never had any performance  issues. 

Average of ratings: -
In reply to Usman Asar

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Ok Usman, Thanks .. !!

I'll discuss with my seniors regarding setting up moodle on IIS to try.

Beside that in apache and php is there any configuration setting which I should apply and try becouse if we Google then we get some setting which suppose to improve performance. E.g. Apache's mpm module, timeout and keep alive time out settings and couple of others.


Thanks ..

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Usman Asar -
Picture of Plugin developers Picture of Testers

Indresh, no matter how much you tune Apache on Windows platform, it will perform low. Despite Apache being available for Windows I have never experienced any production platform with that combination. Apache works beautiful on Linux. Even if you want to proceed with Apache regardless of operating system lying under, my hands unfortunately are bound in recommending Apache's setting to bring out performance.

Time out of keepalive  has nothing to do with performance really, you can however increase the buffer size of database to something near to 4GB if it hasnt been done already. setup opcache and set number of files to 8000 as being moodle 3.x.

Howard, Ken, Visvanath, Emma etc will be able to assist better on Apache tuning.

Average of ratings: -
In reply to Usman Asar

Re: Performance Tuning and Load Testing

by Indresh Vishwakarma -

Usman, your all inputs are very helpfull to understand where I need to look in, and I will sure try Windows+IIS for Moodle so I can also experience the difference in performance.

M am asking about Apache tunning because I need to answer to my senior that we tryied tunning everything but we are unable to achieve 200 user because of wrong setup.I 

I have already enabled OpCache with exactly 8000 file size and will check database buffer size tomorrow and will make.

Will update you if I receive approval to try moodle in IIS.

Hope to get some more inuts from Howard, Ken, Visvanath, Emmad others.

Thanks ....

Average of ratings: -
In reply to Indresh Vishwakarma

Re: Performance Tuning and Load Testing

by Usman Asar -
Picture of Plugin developers Picture of Testers

Indresh, honestly speaking there is nothing really that you can do to tune apache for performance, but surely other things like PHP, Cache's and DataBase, same is the case with IIS, but because the way both web server handles requests slightly differently as Apache opens  up new httpd request for every user while IIS re-uses the worker processes and is tuned to work with Windows operating system, so performance on Windows with IIS is better compared to using Apache on Windows.

For a neck to neck comparison, Apache has to be installed on Linux whilst IIS on Windows, you can simply let your seniors know that Apache + windows is not to be seen in a production environment anywhere, they should understand giving them clear way of either keeping themselves on Windows and setting up IIS on top, or moving to Linux and using Apache instead.

Average of ratings: Useful (1)
In reply to Usman Asar

degrama

by Grant Mucha -

And if moving to Linux I recommend nginx, hhvm, php7 or a combination with upstream php_backend.

Depending on the importance of this project / setup moving away from a virtual machine.

And for simplicity, I would follow Usman's recommendations of testing IIS first.

Basic IOPS Guide

http://thecloudcalculator.com/calculators/disk-raid-and-iops.html

Raid 10 - 32,000 IOPS (SSD) <- performance required by anything database intensive

Raid 10 - 256 IOPS (7k SAS) <- IOwait issues, high cpu usage, poor database performance

Average of ratings: -
In reply to Grant Mucha

Re: degrama

by Indresh Vishwakarma -

Hi,

Thanks Usman, Howard and Grant for your guidance. It really helped me to understand the senario and what i need to try out first.

I'll work on your suggestions for sure and will update improvements.

 

 

 

 

Average of ratings: -