Dear all,
This is our second time to deploy Moodle. The first time was a test run comprising of 2500 users and about 50 concurrent users (later increased to 150). Initially all was fine, but then the site started slowing down. Our hardware specifications in that test are as follows:
CPU: Mac G5 Xserve 2.0GHz (single core)
RAM: 1GB DDR2-400 -> later increased to 3GB
Hard drive: 80GB HDD
Connection: T1 line
Operating System: Mac OS X 10.4
In a few months, we'll be going to production but on a larger scale. We anticipate 4000 users this year and by 2010, 10000 users. I don't think our server can handle this, so we're thinking of going with 2 servers and 2 load balancers (for redundancy). Right now, I have no idea how much processing power is required, so I'll just list what's on the top of my head.
2 Servers
-----------
CPU: Intel Xeon 5140 2.33GHz
RAM: 4GB DDR2-667
Hard drive: 2 x 160GB SATA-II HDD (RAID 1)
Connection: T1 line
Operating System: Debian or Ubuntu Server
Load Balancers (2) - don't know about this, shouldn't take much processing power, right?
I appreciate any comments you guys have. Thanks!
Jerico
How many simultaneous users you looking at? You will also need a way to share the moodle data and database. I'm not sure if you have planned that for your setup or not.
Much as I am big fan of Apple's products, your old Xserve isn't going to cut it.
It *really* depends on the likely number of simultaneous users, but I'm going to suggest that those two machines, one as the web server and one as the database server will probably do the job for you in short to middle term (if not the long term). Don't bother with load balancing. For that you are really talking about
* a database server
* a file server
* multiple web servers
...which is rather more than 2 machines
It *really* depends on the likely number of simultaneous users, but I'm going to suggest that those two machines, one as the web server and one as the database server will probably do the job for you in short to middle term (if not the long term). Don't bother with load balancing. For that you are really talking about
* a database server
* a file server
* multiple web servers
...which is rather more than 2 machines
Thanks for your replies. The Xserve was actually just a gift from higher administration officials and that being the most powerful we had at the time, we decided to go with it. We don't know a lot about Mac OS X Server even though we have Macs at the office.
Anyway, we're looking at about 200 simultaneous users and that's probably on an average day. On a congested day, that will probably jump to 600. However, I don't know if 2 servers will cut it or we need more.
We're currently planning on having the following:
1 database server
X web servers (we don't know yet, but we'll start with 2)
1 file server
2 load balancers (one for backup)
That makes at least 6 computers for now.
Anyway, we're looking at about 200 simultaneous users and that's probably on an average day. On a congested day, that will probably jump to 600. However, I don't know if 2 servers will cut it or we need more.
We're currently planning on having the following:
1 database server
X web servers (we don't know yet, but we'll start with 2)
1 file server
2 load balancers (one for backup)
That makes at least 6 computers for now.
You have definitely too little RAM in your Mac X Server for 50 concurrent users. Try to double it. A single core machine is also a tad low-end for such a task. You also likely need to tune some php settings so mysql is not choking you. This was discussed on this forum a few times.
Two machines will likely not be enough for 200 simultaneous users spiking to 600. You need to have enough machines in your server farm that if one falls out, the remaining ones can still handle the average load. There are results of online quiz taking by a few hundred Moodle users, a true concurrent usage, posted somewhere in this forum, albeit for an older version of Moodle.
Two machines will likely not be enough for 200 simultaneous users spiking to 600. You need to have enough machines in your server farm that if one falls out, the remaining ones can still handle the average load. There are results of online quiz taking by a few hundred Moodle users, a true concurrent usage, posted somewhere in this forum, albeit for an older version of Moodle.
The "rule of thumb" is about 1GB of RAM per 50 simultaneous users. So your single web server will cut it (just) for 200 users with no spare capacity. It's going to fall over with 600!! I would think in terms of 4 such web servers as being a bit nearer the mark.
In my experience, OSX needs more RAM than your rule of thumb, which I believe is a good estimation for Linux systems. RAM is quite cheap these days, so no reason to be skimpy.
One of our Moodle instances has over 10000 users. The highest concurrent user load seen was 1200. We use one Dell 2850 Sever with 2 Dual Core CPUs and 8GB of memory running RHEL 4. That server hosts 3 Moodle instances. We have off-loaded the Database to a identically configured Server. We found that the most profound performance improvements were gained by adding memory. However, how that memory is used also impacts performance. The web server software ( Apache in our case ) has to be carefully tuned. Also OS parameters have to be tuned. Even the buffer size for your NICs have to be tuned. Don't forget file system tuning (noatime).
Moving the database to another server alone gave us about 40% less resource utilization on the Moodle Server. Which translates to a better user experience.
Simultaneous logins and the resulting authentication load is a crucial factor in performance. Our installation uses LDAP authentication. We now use 2 LDAP servers via a "round robin" DNS entry. Getting a user logged in and ready to Moodle seems to be a resource intensive process.
Also the selection of themes for Moodle has a very significant impact on performance. As do the choices course developers make about course materials. Bandwidth should be considered when selecting the type and number of videos, images, and documents used within a course.
The exact optimization details depend on the operating system and web server your system is running on. There are some very good how-to articles on the web about tuning the apache web server and Linux and Mac operating systems and the MySql database. Give them a try.
Our installation was performance tested, ( I should say torture tested) for about 3 weeks. Early in our testing the configuration went "belly up" at about 150 concurrent users then later at 500. After tweaking some kernel parameters (max open files - nfiles and a few others) and tuning Apache memcache the system ran acceptably at 1200 users. That is where we said "Ok, we can live with that".
There is still some tweaking to be done on the MySql server and on the connection between the Moodle and DB server. But that will have to wait till school is out this summer. We are eager to see what effect changing the IP packet size or MPU will have between the two.
Should there be a forum for "Moodle server tuning tips and tricks"?
Moving the database to another server alone gave us about 40% less resource utilization on the Moodle Server. Which translates to a better user experience.
Simultaneous logins and the resulting authentication load is a crucial factor in performance. Our installation uses LDAP authentication. We now use 2 LDAP servers via a "round robin" DNS entry. Getting a user logged in and ready to Moodle seems to be a resource intensive process.
Also the selection of themes for Moodle has a very significant impact on performance. As do the choices course developers make about course materials. Bandwidth should be considered when selecting the type and number of videos, images, and documents used within a course.
The exact optimization details depend on the operating system and web server your system is running on. There are some very good how-to articles on the web about tuning the apache web server and Linux and Mac operating systems and the MySql database. Give them a try.
Our installation was performance tested, ( I should say torture tested) for about 3 weeks. Early in our testing the configuration went "belly up" at about 150 concurrent users then later at 500. After tweaking some kernel parameters (max open files - nfiles and a few others) and tuning Apache memcache the system ran acceptably at 1200 users. That is where we said "Ok, we can live with that".
There is still some tweaking to be done on the MySql server and on the connection between the Moodle and DB server. But that will have to wait till school is out this summer. We are eager to see what effect changing the IP packet size or MPU will have between the two.
Should there be a forum for "Moodle server tuning tips and tricks"?
Darn, Alex! That's impressive!!!! Congratulations! and
have a great evening!
have a great evening!
Useful information about tuning should be in the docs probably - in fact there's some here:
http://docs.moodle.org/en/Performance
...more info and specific tuning information wouldn't hurt at all
http://docs.moodle.org/en/Performance
...more info and specific tuning information wouldn't hurt at all
Yes, please add Moodle server tuning tips and tricks to Moodle Docs, either in the performance documentation or by creating a new page. In the meantime, I've added a link to this discussion so that people can easily find it.
Sorry I forgot to mention that. The information in the Performance link is where I started. It is a very good summary. The information is dead on and concise. Thanks to all the folks who helped put it together.
I'll see what I can do to contribute some platform specific tuning tips.
I'll see what I can do to contribute some platform specific tuning tips.
Hi Alex!
How do you measure concurrency? IOWs how did you reach the 1200 number?
How do the numbers look if you count only session files that have been created/modified in the last 5 minutes? How about in the last 5 seconds? (Or session records if you are using DB-based sessions).
Does the 5s measure match a multiple of apache processes? I'd expect it to have between 2x and 3x apache processes to that count.
I normally count last 5s, last 5m, last 60m, last 24hs. The 5s measure is the only one that means concurrency
and is important to understand peak load. In many cases people count session files in the sessions directory - which is the last 4hs, too long to give you a good idea of actual concurrency.
How do you measure concurrency? IOWs how did you reach the 1200 number?
How do the numbers look if you count only session files that have been created/modified in the last 5 minutes? How about in the last 5 seconds? (Or session records if you are using DB-based sessions).
Does the 5s measure match a multiple of apache processes? I'd expect it to have between 2x and 3x apache processes to that count.
I normally count last 5s, last 5m, last 60m, last 24hs. The 5s measure is the only one that means concurrency
First we should distinguish between simultaneous logins and concurrent users.
Simultaneous logins are the number of users hitting the login button at the same time. Concurrent users are how many users are taking the course at a given time.
Our test was conducted using 3 workstations and one server running WebLoad testing software. The numbers for concurrent users came from the number of sessions reported by the Webload software. We took great pains to create multiple recorded sessions to simulate user load. In all we had about 30 different models. (Believe me when I say there are better things to do with your weekends.) We loaded the models on the test systems and pointed them at the system under test. We actually started 1200 user sessions with the WebLoad software. Each session actually went though a target course using models that simulated watching the videos, reading the course text material with different user reading speeds, and taking the tests. Some models also simulated review of previous material and also simulated users starting the course logging off then returning to the course and completing it. Since the numbers generated internally leave some room for doubt we thought that the numbers and performance stats (screen paint times) generated from the WebLoad software was a more accurate way of estimating user experience.
Run after run we kept increasing the number of concurrent users until we were satisfied. Each run took between 30 and 45 minutes. That is the time it would take the average participant to complete the course. BTW: If you create models don't forget to turn of the question randomization feature. If you don't your recorded sessions will never match and will abnormally terminate.
We tested against a copy of production course that had the quiz question randomization turned off.
Simultaneous logins created a problem for our testing since the Webload software can spawn hundreds of sessions in just a second or two. We had to tune the testing software to more realistically model real user behavior. We set up the test software on each workstation to login about 10 and 15 users per second.
The 3 workstations (3GHz pentiums w/Gbit NICs) were inside the firewall to give us bandwidth numbers also. We created a boat load of test accounts on our LDAP server to get end to end performance stats. We constantly monitored with mytop/apachetop/bandwidthD and htop. Atop was running in the background to record the system stats.
We tested parts of the test suite during the week and then came into the shop on two Saturdays to run the full suite of tests. (Hence the weekend comment above) On the last Saturday we were mentally drained from trying to keep up with all the different parameters. You can imagine our relief when we started up the last test run and got it up to 1200 users and let it run for 20 minutes with no problems. As soon as we saw that result we decided it was time to go home. Our job was done.
The production course went into service the following week.
Simultaneous logins are the number of users hitting the login button at the same time. Concurrent users are how many users are taking the course at a given time.
Our test was conducted using 3 workstations and one server running WebLoad testing software. The numbers for concurrent users came from the number of sessions reported by the Webload software. We took great pains to create multiple recorded sessions to simulate user load. In all we had about 30 different models. (Believe me when I say there are better things to do with your weekends.) We loaded the models on the test systems and pointed them at the system under test. We actually started 1200 user sessions with the WebLoad software. Each session actually went though a target course using models that simulated watching the videos, reading the course text material with different user reading speeds, and taking the tests. Some models also simulated review of previous material and also simulated users starting the course logging off then returning to the course and completing it. Since the numbers generated internally leave some room for doubt we thought that the numbers and performance stats (screen paint times) generated from the WebLoad software was a more accurate way of estimating user experience.
Run after run we kept increasing the number of concurrent users until we were satisfied. Each run took between 30 and 45 minutes. That is the time it would take the average participant to complete the course. BTW: If you create models don't forget to turn of the question randomization feature. If you don't your recorded sessions will never match and will abnormally terminate.
We tested against a copy of production course that had the quiz question randomization turned off.
Simultaneous logins created a problem for our testing since the Webload software can spawn hundreds of sessions in just a second or two. We had to tune the testing software to more realistically model real user behavior. We set up the test software on each workstation to login about 10 and 15 users per second.
The 3 workstations (3GHz pentiums w/Gbit NICs) were inside the firewall to give us bandwidth numbers also. We created a boat load of test accounts on our LDAP server to get end to end performance stats. We constantly monitored with mytop/apachetop/bandwidthD and htop. Atop was running in the background to record the system stats.
We tested parts of the test suite during the week and then came into the shop on two Saturdays to run the full suite of tests. (Hence the weekend comment above) On the last Saturday we were mentally drained from trying to keep up with all the different parameters. You can imagine our relief when we started up the last test run and got it up to 1200 users and let it run for 20 minutes with no problems. As soon as we saw that result we decided it was time to go home. Our job was done.
The production course went into service the following week.
It seems that you are really squeezing every drop of juice out of your servers, but 1200 simultaneous Apache child processes in 8 Gb would leave only around 6 megs per process which seems very low. I haven't seen a lot of Moodle processes under 10 megs, usually they are around 15 megs (+ some shared megs). Maybe the WebLoad is not giving a true spectrum of the process types? Too much sharing/caching? Or have you been able to fine tune Apache (dropping mods etc) really to run Moodle in 6 megs per child process?
The webload software reports on currently running sessions. The session is a simulated user taking an actual course in the normal time it takes to take a course. The distribution over time looked like stair steps until it reached an average of 1200 users.
If your interested in some of Apache setting we used:
KeepAliveTimeout 3
<IfModule prefork.c>
StartServers 20
MinSpareServers 100
MaxSpareServers 120
ServerLimit 4000
MaxClients 4000
MaxRequestsPerChild 4000
</IfModule>
<IfModule mod_mem_cache.c>
CacheEnable mem /
MCacheSize 400000000
MCacheMaxObjectCount 13001
MCacheMinObjectSize 1000
MCacheMaxObjectSize 6400000
MCacheRemovalAlgorithm LRU
MCacheMaxStreamingBuffer 65536
</IfModule>
Yes unnecessary modules were turned off.
If your interested in some of Apache setting we used:
KeepAliveTimeout 3
<IfModule prefork.c>
StartServers 20
MinSpareServers 100
MaxSpareServers 120
ServerLimit 4000
MaxClients 4000
MaxRequestsPerChild 4000
</IfModule>
<IfModule mod_mem_cache.c>
CacheEnable mem /
MCacheSize 400000000
MCacheMaxObjectCount 13001
MCacheMinObjectSize 1000
MCacheMaxObjectSize 6400000
MCacheRemovalAlgorithm LRU
MCacheMaxStreamingBuffer 65536
</IfModule>
Yes unnecessary modules were turned off.
Martin,
I'd be really interested to know what you do to get some data on that figure. Are you running something that keeps a record of concurrency over a period of time?
I guess it's important to make sure you really have seen the peak and useful to know what the distribution over time looks like.
I'd be really interested to know what you do to get some data on that figure. Are you running something that keeps a record of concurrency over a period of time?
I guess it's important to make sure you really have seen the peak and useful to know what the distribution over time looks like.