Windows server: Apache OVERLOAD

Windows server: Apache OVERLOAD

by Hamid Takesh -
Number of replies: 16

Hello all.

Directly to the problem.

I am running a pretty good server machine with W2K3 + Apache + PHP + MySQL + Moodle.

I have tried all versions of apache from 2.0 - 2.2.3, all latest versions of php up to 5.2.0.

And nometter what I do, my apache is getting overloaded when many users go online.

Now I am running XAMPP full package. It seems to be stable... But even so, apache goes down every time alot of hits are made. And XAMPP is not for a production server I know that, but I can't configure the httpd.conf so it works! Nometter what I try, I get the same result. It happens of course only when a PHP page is used. There are no problems with normal HTML.

Can anyone share their system configurations, or advice what packages do you use if any? Any help would be appriciated.

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Ken Wilson -

Hi Hamid

Here are a couple of ideas for you to try out:

1. Install a PHP accelerator. The ones recommended on these forums are Xcache and APC. Installation instructions for APC on W2K3 are here. You should get a noticeable drop in processor utilisation even with a small cache (48M seems to be enough for moodle).

2. Check that MaxRequestsPerChild is set to zero in your apache2/conf/httpd.conf file. This is the recommended setting for Windows systems.

3. How much RAM is installed in the system? Take a look at some of the posts in this forum and on the hardware and performance one - more memory nearly always makes a big difference as it stops disk swapping.

Once you've done everything possible to optimise the software and hardware you've got (don't forget MySQL optimisation), you need to look at what users are doing in Moodle. How many concurrent logins do you expect, and will they be machine-intensive users (quizzes and chat are known to be machine intensive wink).

Let us know how things go!

Ken

Average of ratings: -
In reply to Ken Wilson

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

Hey, Ken.

1) Will do. Thanx. But is it better then Zend Optimizer? Couse it did slow down the website, not actually optimized it.

2) Yea, it iz 0

3) I've got 1 GB RAM. My my.cnf is here:

[mysqld]

basedir="C:/slms/xampp/mysql"

tmpdir="C:/slms/xampp/tmp"

datadir="C:/slms/xampp/mysql/data"

port = 3306

socket = mysql

skip-innodb

max_connections = 500

key_buffer = 16M

myisam_sort_buffer_size = 64M

join_buffer_size = 1M

read_buffer_size = 1M

sort_buffer_size = 2M

table_cache = 1024

thread_cache_size = 64

wait_timeout = 1800

connect_timeout = 10

max_allowed_packet = 16M

max_connect_errors = 10

query_cache_limit = 1M

query_cache_size = 32M

query_cache_type = 1

[mysqld_safe]

err-log=/var/log/mysqld.log

open_files_limit = 8192

[mysqldump]

quick

max_allowed_packet = 16M

[myisamchk]

key_buffer = 64M

sort_buffer = 64M

read_buffer = 16M

write_buffer = 16M

[mysql.server]

user=mysql

I expect up to 1k concurent logins. Iz that possible for moodle on a Xeon 3.2, 1GB RAM?

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Ken Wilson -

Hi Hamid

  1. Zend Optimiser optimises but APC caches - you cannot use both at the same time, as they conflict - choose one or the other. The performance improvement with APC will be greater.

  2. It's difficult to judge if the my.cnf settings are OK - they look like the query_cache_size could be worth increasing. However, the best way to tune mysql is to start with the system variables (mysql> SHOW VARIABLES;) and then move to the status variables (mysql> SHOW STATUS;).

  3. There is so much available on mysql tuning - some of it contradictory, and it all takes time to read and absorb - it's also a *very* steep learning curve. Here are a couple of tricks:

    (a) To start with, make sure you use the correct my.cnf as your template for your server. The one to use depends on how much free RAM you have in task manager, e.g. my-large.cnf is for a server with 512Mb free. 

    (b) After you have got the right ini settings, you're ready to start tuning. You've already enabled the query cache which is a good start. Remember though that some ini settings will multiply per connection and use up your RAM, e.g. key_buffer - so if you get 100 connections that's 6400Mb of RAM they'll take up and your 1Gb server will start to swap processes to disk sad. 

    (c) Turn on your slow query log with these commands
    log-slow-queries = mysql-slow.log
    long_query_time = 5
    This will keep you informed of any potential mysql bottlenecks for queries taking longer than 5 seconds. Decrease this time if you want to.

    (d) To get tuning done quickly, save on reading time and bypass the learning curve smile, try to use mysqlreport from www.hackmysql.com which provides detailed analysis on your mysql database performance and combines all of the mysql variables together into a meaningful form. On W2K3, you'll need to install activestate perl and use the perl package manager to install additional modules that mysqlreport needs first. Also run your system for over a week to collect reasonable data to analyse.

    (e) The documentation that is on the hackmysql website has to be a must read for anyone who has a serious moodle install. So spend some time studying this!

  4. Our system is a dual 2GHz Xeon with 2Gb RAM. It coped easily with 500 concurrent users. A way to test your system capacity is to use something like JMeter. For response/connection times, use ab.exe in your apache/bin folder.

    A good way of test system capacity with JMeter is to look at loading the moodle front page moodle/index.php, which fires off between 15-17 mysql queries. Keep increasing the number of concurrent threads in JMeter until you start to get errors - that'll give a rough guide if your system can cope with 1K users. Concurrency is a difficult idea to simulate - when you say 1K users it is unlikely that they will be accessing the server at exactly the same time. So, for a better picture of how your server will cope with 1K users, use the Gaussian timer in JMeter to model user browsing.

    For us, it was the RAM that did it wink - so look to increasing yours if the system gets slow. Also, take a look if you can enable hyperthreading on your server - it may help.

If the server is still not coping, then start using Performance Counter to look at other bottlenecks or even using load balancing.

Ken

Average of ratings: -
In reply to Ken Wilson

Re: Windows server: Apache OVERLOAD

by Patrick Blaine -
Thank you very much for all this information Ken. We've been trying for a little bit now to squeeze more performance out of our box (while keeping it stable), but sort of stumbling around in the dark. All of this will save us a lot of time in pointing us to some good resources and indicating some preliminary trouble areas to look at. Perhaps we can eventually get all this together and have it added to Moodle Docs. I think it would be of use to a lot of people.

Patrick
Average of ratings: -
In reply to Patrick Blaine

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

Thank you, Ken. This post should go into moodle docs.

I'll post here as soon as I get all of it or some of it done.

Thank you again.

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

OK.

I've just installed APC. The results are trully pretty imressive.

Let's see if the rest of your advices are as good, Ken... wink

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -
Well... it did increase the performance, and the apache process isn't killing the machine no more, but it still goes down when more then 30 students are taking quiz at the same time.
Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Ken Wilson -

Hi Hamid

Quizzes are database intensive. Take a look at this thread, where Martin gives some hints about how really intensive they are. What is needed now is the output from mysqlreport -all so that we can all take a look at what the system limitations are. Can you post the output from immediately after the time when your users are attempting the quiz and the system is heavily loaded?

Also, what is your disk configuration? SATA/SCSI? RAID level?

Ken

Average of ratings: -
In reply to Ken Wilson

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

Hi Ken.

The problem is - I didn't set up the server myself. So i don't really know much about it. I also don't have fisical access to it - only remote desctop.

I know it's RAID SATA.

I'll try to post the DB results today.

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

There you go.

SHOW STATUS and SHOW VARIABLES

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

Forgot to tell you. The stats are with 30 students online taking a quiz.

The system didn't crash today, but will it handle 100 students doing the same? Or 200? The plan is to get all the subjects for grades 9-12 into our moodle installation. So it WILL have to handle 1k students at the same time.

Now, when I am testing the server with apache 'ab' with 100-200 concurent connections arround half of requests fail. I am requesting index.php that is.

I tried to crash the server myself by immitating alot of concurent requests. It went slow... almost dead. RAM usage went up to 800MB, but it didn't die. Once I stopped the requests, the server recovered, RAM usage dropped to normal and it went fast again.

So now the question is how to avoid the 'going slow' part? We will be haveing an extra GB of RAM and may be another xeon. Wil this solve the problem?

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Ken Wilson -

Hi Hamid

I tend to find that apache ab.exe may sometimes give false failures, so try to use another load testing utility if you can.

Your point about quizzes is a recurring one. If you enable your mysql general.log file, you'll see how many database queries are being done when a page is accessed. Just as a test, a one question multiple-choice generates 144 queries from start to finish. Interestingly, the same quiz done in HotPotatoes generates 61 queries - so this could be something worth investigating if you want to reduce db load. The level of SELECT queries remains the same at roughly 70% for both types.

Taking a look at your mysql data, the effectiveness and stability of the Query Cache is good with a utilisation of 59% and no prunes. However, here are a few changes that might help:

  • Change key_buffer_size to 8M (to improve key buffer utilisation) 
  • Change query_cache_min_res_unit to 2M (to reduce the high QCache fragmentation ratio)
  • Change table_cache to 512
  • Change thread_cache_size to 600 (to reduce possible thread thrashing)

Another area to look at is your httpd.conf. General apache performance tuning suggests to remove as many modules as you can get away with, so as to reduce the footprint of the httpd process. The lowest I've ever been able to reduce the modules used on Windows is to rem out all the Load_modules statements except:

  • mod_authz_default, mod_authz_groupfile, mod_authz_host
  • mod_dir
  • mod_log_config
  • mod_mime

So, all the others are remmed-out with a hash (except the php module wink). Also, remember to set ThreadsPerChild to 1000 if you anticipate 1k users. Another tip to reduce processor cycles is to turn off compression (either in php.ini - zlib.output.compression = Off - or in apache).

I think that the extra RAM and processor will definitely help out - make sure to turn on hyperthreading in the BIOS as this will give you 4 virtual processors cool.

As for knowing if the system will hold up, I've attached a JMeter file which might be useful. It will simulate users logging in and browsing to the moodle home page of your site. Use it as follows:

  1. Click on the Moodle Home Page icon and set the server name or ip address of your server.
  2. Click on the Login icon and set the server name /ip as well as the username and password to login as.
  3. Click on the HTTP Thread Group icon and set the Number of Threads (users) to the number of users to simulate.
  4. Click on Run -> Start and switch to the Summary report to see what's happening.
  5. A successful simulation shows zero Error %.

Keep increasing the number of users until you start getting errors - you'll then know the absolute maximum your system can cope with - but note that this is dependent on the number of threads your own PC can create, so if you want to simulate over 200 users it's best to run it on 2/3 PCs at the same time.

In the end, you can only do the best with what you have, after which the next step is load balancing and splitting the database and apache onto different servers - for which a Moodle Partner will probably be ideal!

Let us know how things go.

Ken

Average of ratings: -
In reply to Ken Wilson

Re: Windows server: Apache OVERLOAD

by Hamid Takesh -

Hi, Ken.

Thank you so much. I really appriciate your help.

All of this really helped. It not only is not crashing, but became faster. Both Apache and MySQL.

Stick this topic on top. Really helpfull information that is hard to find all in one place.

Thanx again.

Average of ratings: -
In reply to Hamid Takesh

Re: Windows server: Apache OVERLOAD

by Gary Anderson -
While it would be best for Apache never to go down, you can keep this from affecting your users by making sure that you set the Apache 2 service recovery (a tab on the service section of Administrative tools) to restart the service on the first and second failure and restart the computer after that (I have never had it restart the computer). Click the checkbox to restart service after 0 minutes so you don't get a 1 minute downtime. Check the logs for failure from time to time.

Whether this is a bug in php, running out of some type of memory, or who knows what, this this does happen, but the restart does fix the problem and users will still have their sessions retained, etc.

I found that xcache and APC helped this problem (I might have one service restart a week on a busy high school server, vs. 3 or 4 a day). But with the above fix, the server is never seen as going down.
Average of ratings: -
In reply to Gary Anderson

Re: Windows server: Apache OVERLOAD

by Antoine Elearn -
Hi everybody,

I currently try to evaluate the performances of my Moodle server with JMeter;
but i know some problems... In fact, I manage to connect threads to the platform using HTTP request and parameters but as of the second request, threads are redirected to the login page!

I want to know how to preserve a session at the sequence of the requests?

Thanks in advance for your help.

Cordialy

Average of ratings: -
In reply to Gary Anderson

Re: Windows server: Apache OVERLOAD

by Mark Hilliard -
Gary,
I have been trying to fine tune a production Moodle Server (using Win 2003, PHP 5x, Apache 2.x.x., and MySQL) and I have some questions for you...

You mentioned using the one of the tabs in the Win 2003 Admin-Services to automatically restart the Apache server if the "the service fails"
I think I found the Service Configuration Screen you mentioned, but how did you test out how this works? (see attached PDF)

Why did you start using the re-start process, did you have a Win2k/Apache/Moodle server that would lockup and not respond???

How does the Windows 2K3 server KNOW to restart the Apache service?
Does it read an error log somewhere or get notified by some part of the Op. Sys that the Apache Service has/is failing?

As we have been seeing some failed connections to the Moodle-Apache server under loads, I would like to use this Auto-restart Apache Service Tool, but I would like to know how it works, or more about your experience before I try it out.

Any info you can share would be very helpful.

Also: FYI Like you we have been seeing a "memory leak" in the Apache web servers with PHP.

This appears to be a known problem going back to 2002 with no fix. One of the two Apache Processes (either Apache.exe 1 and 2 (or in newer versions Htppd 1 and 2) starts to grow and consume memory until the web server stops responding. IT can be triggered and recreated on purpose with PHP scripts. This might be the cause of many slow or non-repsonding Windows/Apache/PHP web servers.
There are a few tweaks to help but I don't know of a real solution I have seen yet.

Mark H.

Average of ratings: -