Server Stops with 160 Simultaneous Users

Server Stops with 160 Simultaneous Users

by Don Hinkelman -
Number of replies: 18
Picture of Particularly helpful Moodlers Picture of Plugin developers
Here is a report of the server operation for administering a 50 question m/c language placement test with embedded audio to 160 freshmen on April 5.

Group 1:  160 students  (4 rooms)
- teacher started up IE6 and Moodle site remotely, up to the Moodle login screen on all 160 computers.  perhaps 2-3 minutes to load.
- students logged in (I had pre-enrolled them and prepared login cards for them to copy)  Problems started here.  Long delays. 
- students clicked on the course link and long delays.   10 minutes to get to test page for some.  Most could not get anything.  About 20% got an error screen that said something about a database problem.  No one even got into the test.
- a computer center assistant standing by thought it might be an Apache setting limiting the number of users.   We ran to the server and changed the Apache setting of max clients from a default of 150 up to 1000.  This seemed to help some students a little but still only 2 students could actually get into the quiz after 30 minutes of trying.
- we decided to let the students go, and decided to give them the test on paper the following week.
- we cleared the rooms, and got ready for the next group.

Group 2:  80 students  (2 rooms)
- teacher started up IE6 and Moodle site again remotely, but the remote software did not work in one room, so we tried to do it manually on those computers.
- students logged in with less delay
- 40 students could open the test and began the quiz before the other students.  Some delays in the  audio so we told them to jump ahead to the reading section of the test (no audio).
- second room started the test 10 minutes later and had longer delays in opening the test.
- audio problems started popping up.  Some students could hear only 25 out of 30 audio clips.  Some could hear all, some could hear all but one.  They started sharing headphones to hear the missing clips smile
- on some computers a popup screen appeared asking if the student wanted to enable Flash on their computer.  If the answered Yes, no problem.  The few that answered No, got problems with the Flash player icons appearing.  An odd graphic appeared that could not be played.
- Finally, by going 10 minutes overtime, all 80 students could finish and score their tests.  Success!  big grin
- 11 of the 80 scores however were listed under the ID of the students in the previous group.  All of those were in the room where we could remotely restart all the browsers.  We believe the browsers were not closed and restarted on those 11 students.
- Later that day, my programmer friend found another setting problem...in MySQL the maximun connections were limited to only 100.  He said...

So I changed max_connections from 100 to 1000, and restarted MySQL.
(when you start mysql, you can use some option parameters.
e.g. # /usr/local/mysql/bin/safe_mysqld --user=mysql
--set-variable=max_connections=1000 &)

Server Setup:   Mac OS X 10.3.3 on a Power Mac 933mhz with 768MB Ram
Apache 1.3.29 (standard Mac build), PHP 4.3.2 (no GD working), MySQL 3.23.49
Student Client Setup: PCs running Windows 2000Pro, IE6 browsers, Flash 7 installed.
 
Future Plan:
1. Set Apache max clients=1000
2. Set MySQLmax connections=1000
3. Add a php accelerator  (server logistics version of PHP 4.3.4 includes MMCache)
4. Add memory to over 1GB
5. Upgrade CPU to Mac Dual 1.2ghz
6. Move to Server Logistics Apache2-PHP4.3.4-MySQL4-MMCache package
7. Run the test so that each room starts a few minutes apart (avoid huge jam of requests)
8.  Keep a paper version of the test handy in case of disaster  smile

My Questions
1.  Will the above plan fix the problem?   Will we be able to handle a load of 200 simultaneous users doing the above audio-quiz?
2.  Are other problems, settings that we missed?  How can we fine tune the setup to improve performance?
3.  Is Apache 2 no problem?
4.  Was the embedded audio problem due to the server problems?   Or do we still have some resource permission problem?  Why did only a few audio clips not play?
5.  Any other suggestions?
Average of ratings: Useful (2)
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by Martín Langhoff -
Strong recommendation from an admin with experience in large Apache/PHP/Mysql setups: go the opposite direction with maxclients.

Your server has too many apache/mysql pairs in memory, so it's using swap, and grinding to a halt. Initial steps:

 - maxclients and mysql maxconnections to 100 (_less_ than your students, yes)
 - set mysql's wait_timeout to 15 with --set-variable=wait_timeout=15

Now, this will make things flow better, but probably be short of your needs. The trick is to monitor the memory load on the server as it's getting loaded. If should never start using virtual memory/swap; inclease max_clients slowly, studying how far you can go with it without risk of swapping.

A very log wait_timeout is key here, as it will counterbalance php's pconnect() calls' tendency to hog db connections innecessarily.

RAM is probably your constraint. The other steps you plan to implement are good too.

You can use ab (a command line utility) to simulate dozens of users hitting it a the same time, using the concurrency setting.
Average of ratings: Useful (2)
In reply to Martín Langhoff

Re: Server Stops with 160 Simultaneous Users

by Don Quixote -

Thanks Don and Martin, excellent postings with very valuable information! approve

Andreas

Average of ratings: -
In reply to Martín Langhoff

Re: Server Stops with 160 Simultaneous Users

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Thank you very much, Martin.  I will need a little help though to understand.  mixed
First of all, the problems happened when we did have Apache max clients set to default 150 and when MySQL max connections were set to 100.  Raising the clients up to 1000 seemed to help.  We still don't know the result of raising max connections of the database to 1000. 

It seems what you are saying is that default settings are perfectly fine.   Is that what you mean?
Average of ratings: -
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by Martyn Overy -

Don,

Your evaluation so far  is useful information indeed. Do you know of any other large scale use of Moodle, or are most users still in the evaluation stage?

I am trying to convince the powers that be at our school  that Moodle is the way to go....and to look at stepping up the usage of Moodle from small-scale evaluation levels.  Problem is, I have not as yet a convincing argument for the requirements of Moodle for a school such as ours  (over 1600 students...).

What I am looking for is an answer to the problem:  For n users online , we will require  the following setup......... Then I need to convince them it will work!  It is unlikely that the school will fork out 100's of dollars/pounds a year on a guestimate!

Average of ratings: -
In reply to Martyn Overy

Re: Server Stops with 160 Simultaneous Users

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Actually, Martyn, Moodle is used in large universities with over 10,000 users.  They use sophisticated servers with distributed databases and things I don't understand.   For a simple solution, you can host for as little as $200 per month and have absolute confidence to handle 2000 students using Moodle.com. 

I am a single teacher with a server next to my desk trying to do something rather ambitious-- a placement test with 160 users all pushing the same key and the same time--this is something few people will ever do.  Also I have a simple little machine that I stretched too far.  Again, note from my posting that Moodle was *NOT* the problem, my server was.

Look around the postings in this forum back during the past year and you will see lots of useful information for the systems managers at your school and also look at the list of registered sites http://moodle.org/sites/.  You will see that schools such as Ohio State, Univ of York, Univ of Glasgow and others use it.  I am sure if you contact some of them directly you will get a similar positive response. 

But actually I am afraid your question really doesn't fit here in this topic smile   I am waiting for an answer from another "Martin" about configuring my server.    Try posting your question again as a separate topic (it is a good question!)   "For 1600 users, in xxx number of courses, accessing x times /week, what is the required hardware and software I would need for my own server(s)?  Or what would it cost for a hosting provider to handle this load?"  smile
Average of ratings: -
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Don

Could you clarify the problem?

I'm just curious whether there is a connection to this
http://moodle.org/mod/forum/discuss.php?d=24985 ?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Servers Handle 250 Simultaneous Moodle Users!

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Visvanath,

Yes, we solved the problem. Thank you for asking.

Here is what happened. Despite the stoppage one year ago, my school forgave me and agreed to let us proceed full-scale and do an English placement test (Moodle Quiz Module, ver 1.4.4 with 30 audio questions) this past April 2005 for all 1100 entering freshmen to our university. If you see my report up above, you will notice in April 2004, I used a single slow Mac as a server--and yes, the other thread shows us OS X and MySQL don't love each other so well. This is what we did to make our test a complete success.
  • switched to cheap Linux servers (see specs below)
  • divided the students into 5 sessions of up to 250 students each
  • used four servers (one server per 55 students)
The students and teachers had no problems--speed was fantastic. 3-5 seconds to load the full 50 question test (55 students at the same time on campus). I was off-site in Australia and it still took me only 8-12 seconds loading at the same time the students were doing it! A server technician watched the load in real time and noticed **no** slowdown spike at the beginning of the session. He reckons we only needed two servers, and each can handle 120+ users at the same time.

Here are the server specs and some other notes--each server cost US$1500 back in March 2005.
  • Dell small business server--no OS, no monitor/keyboard
  • Dual Xeon 2.8ghz CPU
  • 2GB Memory
  • 80GB hard drive
  • Internal LAN--servers in another building on campus
  • Fedora Core 1, Apache 2.0.50, PHP 4.3.1, MySQL 4.1.10a
  • Ioncube PHP accelerator for PHP 4.3.0
I am very happy with Moodle for intensive, simultaneous testing. I think this shows that for very low cost you can build a fast, heavy duty server that will handle the highest load situations for Moodle 1.4--long quizzes with audio files. 1.5 should be even faster with the new paged quiz option (all quiz questions load page by page, not all at the beginning).

Average of ratings: -
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by Don Quixote -

Don,

I don't know if you're under timely pressure and it should work as soon as possible (?).

If so, maybe I could help you temporarly to let it run on my server (1,7 GHz, 1024 RAM), since I am only preparing it for other stuff at the moment... Then you could search for your solution more calmly.

 (It depends on how much data you have and what kind of internet connection.)

Just an idea...

Greetings

Andreas

Average of ratings: -
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by Martín Langhoff -
Don wrote:

First of all, the problems happened when we did have Apache max clients set to default 150 and when MySQL max connections were set to 100.  Raising the clients up to 1000 seemed to help.

Defaults are pretty good, except that mysql has a horrendous long wait_timeout default  of 8hs. This kills most web app servers under load.

Mysql's wait_timeout is a major factor, and having _higher_ maxclients/maxconnections will make it worse. In fact, you need apache's maxclients to be matched to mysql's maxconnections, and add a few extra mysql connections. Try maxclients 100 and maxconnections 110; this ensures you have 10 extra available connections to connect to mysql and monitor it.

Try mytop to monitor mysql.

You need apache/mysql roughly paired for max connections, and then assess how much extra memory each pair takes. Unfortunately, this isn't trivial to check, so you can increase the values slowly (10 at a time, say) and monitor memory consumption.

Now, I know you can't be monitoring the sever and teaching at the same time. And when slowdowns happen, you only hear about them  after the fact. I suggest you install sysstat, which collects usage stats every 5 minutes. You can then run 'sar' to see whether the server was using swap.

Your other planned fixes were good, but you must avoid using swap at all costs in the first place.

regards,



martin
Average of ratings: -
In reply to Martín Langhoff

Re: Server Stops with 160 Simultaneous Users

by W Page -
QUOTE
"...
Try mytop to monitor mysql.

Snip...

Now, I know you can't be monitoring the sever and teaching at the same time. And when slowdowns happen, you only hear about them  after the fact. I suggest you install sysstat, which collects usage stats every 5 minutes. You can then run 'sar' to see whether the sever was using swap.
..."  Martin Langhoff

Hello Martin!

Can Mytop and Systat be installed on a shared hosting server?  By that I mean, installed on a website which is on a shared hosting server?

If they cannot, do you know of other programs which can be installed on a website on a shared server which can do the same things as these two programs?

WP1
Average of ratings: -
In reply to Martín Langhoff

Re: Server Stops with 160 Simultaneous Users

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Martin,
Thanks for very detailed help. We checked error logs and can report the following...

1. On the day of the placement test overload, there were no wait_timeout error. We only had below; " [Mon Apr 5 14:56:40 2004] server reached MaxClients setting, consider raising the MaxClients setting"

2. We did a simple ab test and read the file "english-memusage-ab.txt"
The result is with 30 connections (n=30) at the same time and that was looped for 300
connections (c=300). CPU usage result was "77.6% user, 22.4% sys" and "0.0% idle". It means that our englishforum server was overflowing. Also memory usage was very tight. We will also use a Web Application Stress Tool from Microsoft later and report results to you.

3. LAN usage was only 1Mbps maximum load at any one time, well under the 10Mbps capacity. This was measured by MRTG tool.

So from this information, it seems we need to increase MaxClients, but test it slowly and incrementally (+10) as you suggest to avoid overusage of memory. We also need to increase physical memory. My system engineer friend recommends to go up to the limit of 2GB. That will allow us to increase MaxClients. Is that correct? We will also investigate mytop to monitor mysql.  And we will "LOG OUT" during heavy stress times, to reduce that 22.4% system overhead.   Thanks again!
Average of ratings: -
In reply to Don Hinkelman

Re: Server Stops with 160 Simultaneous Users

by W Page -
Hello!

What is an "ab test"?

WP1
Average of ratings: -
In reply to W Page

Re: Server Stops with 160 Simultaneous Users

by Freddy Montilla -

Hi,

ab is "Apache Benchmark" , it will help you have an idea of the expected performance of your site.

This is a tool that comes with the Apache Web Server and can be used to do an stress test of Apache.  So it will tell you an aproximation of how many requests per second can serve your web server. 

ab simulates the concurrent users with one command line, for example:

$ ab -n 100 -c 50  www.mysite.com/report.php

this command simulates 50 concurrent users, requesting this script "report.php" 100 times.

The command will give you a summary of the performance of Apache for this test, somethinh like:

Server Software:        Apache/1.3.26                                     
Server Hostname:        www.mysite.com
Server Port:            80

Document Path:          /report.php
Document Length:        3320 bytes

Time taken for tests:   217.244 seconds
Complete requests:      100
Failed requests:        0
Broken pipe errors:     0
Total transferred:      3667665 bytes
HTML transferred:       3657803 bytes
Requests per second:    18.55 [#/sec] (mean)
Time per request:       264.25 [ms] (mean)
Time per request:       50.22 [ms] (mean, across all concurrent requests)
Transfer rate:          64.90 [Kbytes/sec] received

                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Freddy Montilla
                                        
Re: Server Stops with 160 Simultaneous Users

                                        

                                            by Arne Gylseth - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        
Very interesting. I did not know the ab test. Running two remoote logons on the server and then "ab" in the one window and "top" in the other window gave, I think, a lot of interesting information. (Enough to start concitering a new server ...)
I also tried to change between a html page and a php page. I'm surpriced how much less server load that were the result of calling the html page compared with the php page. (Even though I knew it should be a difference.) A general rule when it comes to server load, seems to be something like "allways replace a php page with a html page when it is possible" (!?)
I think that the ab test apaer to be a very usable function. By the way, at my RedHat Linux there is also a "man ab"

                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

        

                

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Freddy Montilla
                                        
Re: Server Stops with 160 Simultaneous Users

                                        

                                            by W Page - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        Hi Freddy, 

Thank you  for  the explanation.

Can this also work on a shared server?

WP1


                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

        

                

        

                

        

                

        

                

        

                

        

                

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Martín Langhoff
                                        
Re: Server Stops with 160 Simultaneous Users

                                        

                                            by rick shepherd - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        Er... Why not move onto PosGres?



Rick
                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to rick shepherd
                                        
Re: Server Stops with 160 Simultaneous Users

                                        

                                            by Genner Cerna - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        
Im having the same problem with my server when concurrent use, the server dies... where is maxclient be locate? is it on the apache conf or mysql conf? 3 to 5 times our server dies when accessing the same time... why? what can be done?

                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  - 

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

        

                

        

                

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Martín Langhoff
                                        
Re: Server Stops with 160 Simultaneous Users

                                        

                                            by Bryce Thornton - 
                        

                                

                
                                        

                                        

                            

                
                            

                
                                

                                    

                                        For anybody that's having trouble setting the "wait_timeout" option, you may want to try setting "interactive_timeout".  "wait_timout" seems to inherit from this value. 



Bryce
                                    

                
                
                
                                        
                
                
                                            

                                                    
Average of ratings:  -