News instalation (20k registered users / 1k concurrent users)

News instalation (20k registered users / 1k concurrent users)

by Renato Jr -
Number of replies: 14
Hello,

I need to set up moodle (1.9.4) for this scenario:
- 20.000 registered users
- 1.000 concurrent users
- All courses in format SCORM, using flash resourcing (SWF)

My questions are:

1) Operation System: It is appropriate to use the Windows Server 2003 or is essential to use Linux?

2) Database: Can MySQL support it?

3) Concurrent users: "The general rule of thumb is that Moodle can support 50 concurrent users for every 1GB of RAM" (http://docs.moodle.org/en/Installing_Moodle#Hardware). Can the number of concurrent users may increase if database is on another server?


4) Hardware: Can this settings support my moodle instalation?
- web server: Intel Xeon 2.40GHz com 8Gb RAM
- database server: Intel Xeon 2.40GHz com 4Gb RAM

Other details: I'll use Apache 2.2 / PHP 5.2 / MySQL 5.0

Any help appreciated. Thanks.
Average of ratings: -
In reply to Renato Jr

Re: News instalation (20k registered users / 1k concurrent users)

by Greg Lund-Chaix -
OS: Personally, I'd never consider using Windows for a server. Too much overhead, both software-wise as well as sysadmin time.

Database: Can MySQL support 20k users? Yes. I have 30,000 user accounts on my MySQL server (2x quad-core Xeon, 16GB RAM). It is handling the load with absolutely no trouble at all, although our concurrency is not as high as your requirements specify.

Concurrent users: I think the rule of thumb assumes "for every 1GB of RAM *dedicated to Apache*". For a system that large, I wouldn't even consider running the database on the same server.

Hardware: How many cores on those Xeons? An opcode cache (APC, eAccelerator, etc.) is essential, so make sure to plan for its memory use as well. 8GB of RAM seems a bit small to me. RAM is cheap, there's no reason not to load the server up with as much RAM as you can. You may also want to look at an alternate/lightweight alternative to Apache (like nginx or lighttpd) that has a lower memory footprint and can fee up more RAM for cache and additional user sessions.
In reply to Greg Lund-Chaix

Re: News instalation (20k registered users / 1k concurrent users)

by James McLean -
I'll echo exactly what Greg has said, mostly your requirements sound fine.

However I would suggest running a newer version of Moodle, there have been a number of important improvements and security fixes recently (in the last week!), and moving up to the current version from your existing version shouldn't be too much of a headache.
In reply to Renato Jr

Re: News instalation (20k registered users / 1k concurrent users)

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

> I need to set up moodle (1.9.4) for this scenario:
> - 20.000 registered users

That number alone doesn't say much.

> - 1.000 concurrent users

Yes, that is on the high side, depending on how you define "concurrent users" http://docs.moodle.org/en/Performance_FAQ#How_do_you_define_.22concurrent_users.22.3F

> - All courses in format SCORM, using flash resourcing (SWF)

That has more of an impact on the bandwidtch (and the client CPU 'course). If all your 1 k users are really concurrent, the theoritical maximum per user through a 1 Gbit/s network interface is only 1 Mbit/s!

> 1) Operation System: It is appropriate to use the Windows Server 2003 or is essential to use Linux?

As you must have noticed, Linux is considered more efficient by most of the contributors of this forum, see http://docs.moodle.org/en/Performance#Operating_System . People argue that if you master Windows, you can achieve the same. Well, if you do, you won't be asking, right?

BTW, it is operating system. Operations are more serious things smile

> 3) Concurrent users: "The general rule of thumb is that Moodle can support 50 concurrent users for every 1GB of RAM" (http://docs.moodle.org/en/Installing_Moodle#Hardware).

Means you need 20 GB RAM?

> Can the number of concurrent users may increase if database is on another server?

Generally yes, because they share the work. But it won't double. For one, they won't share the load equally, secondly, network latency comes in to the picture. Still, this is a simple solution (apart from more RAM) compared to real multiserver techniques.
In reply to Visvanath Ratnaweera

Re: News instalation (20k registered users / 1k concurrent users)

by Renato Jr -
Thanks for answering so fast!

Greg and James:

Operation System: I'm thinking about using Linux (Ubuntu).
Database: I'm going to use MySQL, thanks.
About concurrent users: How many concurrent users can I support having 16gb RAM at the webserver and 16gb RAM at the database server? (I'm guessing 800)
Hardware: My processor is a 2x Intel Xeon 5405 2.0 GHz Quad Core


Visvanath:
Your answer was really interesting. All my 1k users are really concurrent (so it means I need 20gb RAM, right?).

As I said, I'm looking for a Linux version (maybe Ubuntu).


***************************
My hardware set up

Web server
2x Intel Xeon 5405 2.0 GHz Quad Core
250 GB IDE/SATA HDD
16 GB RAM

Database server
2x Intel Xeon 5405 2.0 GHz Quad Core
250 GB IDE/SATA HDD
16 GB RAM
****************************

Thank you.
Renato



In reply to Renato Jr

Re: News instalation (20k registered users / 1k concurrent users)

by Renato Jr -
My infrastructure goes attached below. How many concurrent users can it support?


Webservers config (x3):
250 GB IDE/SATA HDD
Second Intel Xeon 5405 2.0 GHz Quad Core Processor
8 GB RAM
Windows Server 2003 - Enterprise - 64 Bit



Database config:
Second Intel Xeon 5405 2.0 GHz Quad Core Processor
146 GB 15K RPM SCSI/SAS HDD
16 GB RAM
Windows Server 2003 - Enterprise - 64 Bit


Thanks again.
Attachment load-balancer_EN.jpg
In reply to Renato Jr

Re: News instalation (20k registered users / 1k concurrent users)

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Renato

You are welcome!

Two things, before there is a misunderstanding:

> All my 1k users are really concurrent

There is another factor: One has to know what there 1 k users are doing. Requesting a normal course page is one thing, submitting an online-test (Quiz module) is something different. Or, those 1 k chatting through the Chat-module would be a headache.

> (so it means I need 20gb RAM, right?).

That questionmark is mine ;) meaning I don't know the answer.
In reply to Visvanath Ratnaweera

Re: News instalation (20k registered users / 1k concurrent users)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
On the subject of quantifying what we mean by a 'concurrent users', my colleague sam recently came up with an interesting measure: Number of rows written to the mdl_log table per minute.

Just thought I would throw that idea out into this forum. Do people think that is a useful number?
In reply to Tim Hunt

Re: News instalation (20k registered users / 1k concurrent users)

by Bryan Williams -
Tim, I think I like that idea. I'm wondering if that value might be something that could be added to Performance or perhaps Session Handling in Admin. Could this act as a throttle on the Moodle site in terms of limiting the number of concurrent (rows written) load? Very useful if so.
In reply to Tim Hunt

(re)defining 'concurrent users'

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Tim

> Number of rows written to the mdl_log table per minute

Bit too Moodle-specific for me. And, why per _minute_?

I'm thinking in the direction of "number of http-requests pending at a given instant" for a pint measure or "number of http-requests server during a minute (or second or hour)" if one needs an average.
In reply to Visvanath Ratnaweera

Re: (re)defining 'concurrent users'

by Gary Benner -

HI all,

Sometimes a simple request hides a deceptively complex issue.

Road engineers often "count cars", for traffic managment reasons, but this does not give the information on the impact on the road itself.

The weight or load of the vehicles has much to do with the demand placed on the road surface.

In a similar way, the payload of any http request / response may make a significant difference on the demand placed on the server.

For instance, a simple file request is quite different to a Moodle Report request. The file request places a minimum of demand for authentication etc, and rest is handled quite quickly using low level features of the OS. A small file ( eg text outline ) compared to a 10Mb video will also make a difference of course.

The Report however may require a significant amount of resource as the DB searches and retrieves data, and then formatting it.

So the nature of the request may make a HUGE difference on your server operation. Simply "counting cars" is not a good measure.

In my experience, you can anticipate an overall scale, but use it to define the architecture of your server cluster, as already outlined in earlier posts. You must however be prepared to ramp up your resources ( additional web servers, larger DB server etc ) should the demand experienced be found to be higher than initially expected.

Likewise it could be less. <g>

With Moodle it is a case of the users ( and that includes the course creators and teachers ) ultimately determining the demand on your server, and that can change easily overnight. So be prepared. ( or as they say in the movies, "be afraid, very afraid!!" <g>)

Enjoy.

Gary

 

 

 

 

Average of ratings: Useful (1)
In reply to Gary Benner

Re: (re)defining 'concurrent users'

by Bryan Williams -
Gary, that is a fairly appropriate analogy you're using IMO. All cars are not equal in the Moodle server space, that's for sure. Concurrency isn't necessarily a problem, it's what users are doing that matters. Specifically, db intensive tasks such as quiz taking or working on lessons can dramatically lower concurrency, whereas simply reading contextual material or making forum posts isn't that hard on the server. I think Tim's idea of counting rows written to the db for certain actions (quiz, lesson) might be an interesting experiment.
In reply to Bryan Williams

Re: (re)defining 'concurrent users'

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

If we follow the road engineers, what the customer would like to know is how many small cars, how many limousines and how many trucks can pass during a given time intervall.

It would be nice if you could translate that into three Moodle (unit) loads, like page-request, file-request and quiz-request and describe how to measure them seperately. There a few hints in this script http://moodle.org/mod/forum/discuss.php?d=57028
In reply to Tim Hunt

Re: News instalation (20k registered users / 1k concurrent users)

by Pablo Piquemilh -

Hi, every one I have the next issue. We have installed Moodle 1.9.5+ I have 6000 users and approximately 1000 concurrent users. In one server I have the data base Mysql and in the other one my web application Appache.

And when I have more than 500 concurrent user with more than 300 hundred requests, the Web Application, the server goes down.

I share we what we have installed and all the tunnigs we have done, Any advice or help will be appreciated.

The server A with the Web Application is:

- Procesador Intel Quad Core 3 ghz x 2 ( two procesors Quad Core )

- At the begening Ram 4 GB , now 8 GB

- Hard Disk 114 gb SATA

- Software: Fedora 12, Apache 2.2.13, PHP 5.3.1

The Server B with the Data Base is:

- Intel Quad Core 3 ghz

- Ram 8 GB

- HD 114 gb SATA

- Software: Fedora 12, Apache 2.2.13, MySQL 5.1.40

250 Users consum 1.2 GB Ram from de Application Server I had free 3 GB. But even that the application fell Here is a brief summary with stadistics which describe what it's happening.

This was the result when I had 300 concurrent users in my production environment.

Here I don't have problem

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 13:01:56 CST

Restart Time: Saturday, 20-Feb-2010 13:00:28 CST

Parent Server Generation: 0

Server uptime: 1 minute 27 seconds

Total accesses: 1172 - Total Traffic: 24.3 MB

CPU Usage: u14.32 s1.73 cu0 cs0 - 18.4% CPU load

13.5 requests/sec - 286.0 kB/second - 21.2 kB/request

53 requests currently being processed, 61 idle workers

BUT THEN we simulate over the 300 concurrent users we had on that moment other 300 hundred concurrent users, givig 2 click p/s.

And the result is that thet the CPU load never reach 90% or 100% just fell down sudenñy at the 50%

Project and Scenario Comments, Operator Test Setup Test Type: TIME (run test for 1 minutes) User Simulation: 300 simultaneous users - 2 seconds between clicks Logging Period:Log every 5 seconds

1Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:42:22 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 42 seconds

Total accesses: 1215 - Total Traffic: 15.8 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 42.3% CPU load == Here is normal

28.9 requests/sec - 384.3 kB/second - 13.3 kB/request

347 requests currently being processed, 66 idle workers === normal

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:42:37 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 57 seconds

Total accesses: 1217 - Total Traffic: 15.9 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 31.2% CPU load === But here fell down

21.4 requests/sec - 284.8 kB/second - 13.3 kB/request

376 requests currently being processed, 47 idle workers == Mean while The number of requests increses

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:42:48 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 1 minute 8 seconds

Total accesses: 1220 - Total Traffic: 16.2 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 26.1% CPU load

17.9 requests/sec - 243.3 kB/second - 13.6 kB/request

391 requests currently being processed, 51 idle workers

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:43:00 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 1 minute 20 seconds

Total accesses: 1222 - Total Traffic: 16.4 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 22.2% CPU load

15.3 requests/sec - 209.5 kB/second - 13.7 kB/request

417 requests currently being processed, 47 idle workers

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:43:16 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 1 minute 36 seconds

Total accesses: 1237 - Total Traffic: 16.8 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 18.5% CPU load

12.9 requests/sec - 179.5 kB/second - 13.9 kB/request

456 requests currently being processed, 51 idle workers

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:43:32 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 1 minute 52 seconds

Total accesses: 1238 - Total Traffic: 16.9 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 15.9% CPU load

11.1 requests/sec - 155.0 kB/second - 14.0 kB/request

491 requests currently being processed, 57 idle workers

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:43:45 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 2 minutes 5 seconds

Total accesses: 1239 - Total Traffic: 17.1 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 14.2% CPU load

9.91 requests/sec - 139.9 kB/second - 14.1 kB/request

538 requests currently being processed, 47 idle workers

Apache Server Status for 10.36.193.7

Server Version: Apache/2.2.13 (Unix) DAV/2 PHP/5.3.1

Server Built: Oct 27 2009 15:15:20


Current Time: Saturday, 20-Feb-2010 12:44:05 CST

Restart Time: Saturday, 20-Feb-2010 12:41:40 CST

Parent Server Generation: 0

Server uptime: 2 minutes 25 seconds

Total accesses: 1241 - Total Traffic: 17.4 MB

CPU Usage: u15.8 s1.96 cu0 cs0 - 12.2% CPU load

8.56 requests/sec - 122.7 kB/second - 14.3 kB/request

583 requests currently being processed, 48 idle workers

FOLLOWING THE NEXT RULE

MaxClients = Total available memory * 80% / Max memory usage of apache

WE APPLIED THE TUNNINGS OVER APACHE

FEB 11

- Timeout 300

- MaxKeepAliveRequests 100

- KeepAliveTimeout 15

<IfModule prefork.c>

StartServers 12

MinSpareServers 12

MaxSpareServers 30

ServerLimit 500

MaxClients 500

MaxRequestsPerChild 10

</IfModule>

<IfModule worker.c>

StartServers 12

MaxClients 500

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 10

</IfModule>

FEB 12

- Timeout 120

- MaxKeepAliveRequests 800

- KeepAliveTimeout 5

<IfModule prefork.c>

StartServers 8

MinSpareServers 5

MaxSpareServers 20

ServerLimit 256

MaxClients 256

MaxRequestsPerChild 10

</IfModule>

<IfModule worker.c>

StartServers 2

MaxClients 150

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 0

</IfModule>

FEB 15

- Timeout 120

- MaxKeepAliveRequests 500

- KeepAliveTimeout 5

<IfModule prefork.c>

StartServers 45

MinSpareServers 45

MaxSpareServers 30

ServerLimit 1000

MaxClients 1000

MaxRequestsPerChild 10

</IfModule>

<IfModule worker.c>

StartServers 12

MaxClients 500

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 10

</IfModule>

FEB 18

- Timeout 120

- MaxKeepAliveRequests 500

- KeepAliveTimeout 5

<IfModule prefork.c>

StartServers 45

MinSpareServers 45

MaxSpareServers 60

ServerLimit 1000

MaxClients 1000

MaxRequestsPerChild 10

</IfModule>

<IfModule worker.c>

StartServers 45

MaxClients 1000

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 10

</IfModule>

FEB 20

- Timeout 60

- MaxKeepAliveRequests 100

- KeepAliveTimeout 2

<IfModule prefork.c>

StartServers 3

MinSpareServers 3

MaxSpareServers 5

ServerLimit 1000

MaxClients 1000

MaxRequestsPerChild 1000

</IfModule>

<IfModule worker.c>

StartServers 3

MaxClients 2400

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 1000

</IfModule>

FEB 22

- Timeout 60

- MaxKeepAliveRequests 500

- KeepAliveTimeout 15

<IfModule prefork.c>

StartServers 2

MinSpareServers 2

MaxSpareServers 5

ServerLimit 500

MaxClients 500

MaxRequestsPerChild 0

</IfModule>

<IfModule worker.c>

StartServers 2

MaxClients 2400

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 25

MaxRequestsPerChild 0

</IfModule>

The last set up was

MaxClients 2400

MaxRequestperChild 30

Keep a live time out is between 2 and 5

Lighttpd performance Always the memory keeps consuming 1 GB of the 4 GB in the average, now I have 8 GB and still fell down the application server.

In MySQL the next value maximum number of connections was set in 200.

(I don't have data base errors) and the CPU load on the data base is stablish under the 20%-30%.

The log life time setted is 7 days

In PHP memory was setted for  PHP 5.2.1 in  128 MB.

And we have  eAccelerator v v0.9.6-svn358-dev

Any way the problem stills happening. We don't know what else we need to do, if I need to have increase on the Server Application more memory up to 16 GB or what to do. 

PLEASE HELP!! or advise.

Thanks...!!! smile

In reply to Pablo Piquemilh

Re: News instalation (20k registered users / 1k concurrent users)

by Valery Fremaux -

Hi Pablo,

could you give also settings of the MySQL engine ? usually the default tuning of MySQL is quite low for high load handling. Maybe the crash comes from blocking calls to the database API that comes unanswered due to logical limitations in settings...

We ran a sucessfull test for 1750 simultaneous users on a 3 x 1 x QuadCore culstered plant and we had to retune Mysql to accept 5000 input connections (in our case, Moodle virtualisation solution we made require an extra connection per page).

It may only be an hypothesis....

Hope it can help.

cheers.