HI,
We are going to make a new installation for 3000 simultaneous users, I have no doubt about the number of simultaneous users.
I am thinking that following hardware is needed:
1- One Coyote load balancer
2- 6 web server ( 2 proccessors and 4 GB RAM each server)
3- 1 DB server ( MS SQL is mandatory) with 2 processors and 8 GB RAM
Do you need that this is enough?
How can I solve the problem of MoodeData folder? We have no experience with clusters.
I am wondering if your servers webservers will be windows or linux based. I was also wondering if you would use apache on windows or IIS. What version of PHP and will you use any sort of php caching(eg. eAccelerator).
I would do multiple DB/Moodle data servers so you can have complete fail-over. With 3000 simultaneous downtime can be bad. I would also add a hot spare for the load balancer. But the setup i have listed is for maximum availability.
The coyote load balancer should work for you depending on which model you get so that you can handle the appropriate throughput. I'm not sure what the average throughput is and that also depends entirely on what sort of content the Moodle site will have.
If you wouldn't mind, more details please.
I would do multiple DB/Moodle data servers so you can have complete fail-over. With 3000 simultaneous downtime can be bad. I would also add a hot spare for the load balancer. But the setup i have listed is for maximum availability.
The coyote load balancer should work for you depending on which model you get so that you can handle the appropriate throughput. I'm not sure what the average throughput is and that also depends entirely on what sort of content the Moodle site will have.
If you wouldn't mind, more details please.
Thanks Greg, the webserver will be Windows based using IIS. the PHP version will be 5.2.2. Its possible to use eAccelerator if needed ¿Do you advise me to use this php caching system?
About the DataBase, Do you think that one server will be enough? There will be a back up of the database but we cannot use multible database servers.
If crashes load balancer crashes, we have another load balancer with the same configuration loaded.
There will be only one course, it will be stored in any web servers. The course is dhtml + flash based.
Thanks again
I would use the php cacheing. Thats good that you have the extra load balancer.
One course with 3000 users at once? Thats a major course, what is it? The flash could pull some bandwidth but it depends on what the flash content is.
Your moodle data can be served from the same server as the DB, I recommend getting some SCSI drives 10k or 15k rpm in raid 5 or 10. That will give you data protection on the server as well as a performance increase. If you are servering the files and the DB off the same server you will want that. The 8GB of ram should be fine for the DB server. What are the specs of the processors your looking at?
Do you have any planning as to where/how you are going to host these servers and where are your users going to be accessing them from(local or external)? I ask this to make sure you will be able to provide enough bandwidth to your users.
Which coyote load balancer are you considering? if you are only using a 100Mbit port the 33Kbits/sec of bandwidth could show some lag for your users, especially if your flash content is videos.
Well thats enough for now. I'll see what I can do to help you after your next response.
One course with 3000 users at once? Thats a major course, what is it? The flash could pull some bandwidth but it depends on what the flash content is.
Your moodle data can be served from the same server as the DB, I recommend getting some SCSI drives 10k or 15k rpm in raid 5 or 10. That will give you data protection on the server as well as a performance increase. If you are servering the files and the DB off the same server you will want that. The 8GB of ram should be fine for the DB server. What are the specs of the processors your looking at?
Do you have any planning as to where/how you are going to host these servers and where are your users going to be accessing them from(local or external)? I ask this to make sure you will be able to provide enough bandwidth to your users.
Which coyote load balancer are you considering? if you are only using a 100Mbit port the 33Kbits/sec of bandwidth could show some lag for your users, especially if your flash content is videos.
Well thats enough for now. I'll see what I can do to help you after your next response.
OK, I will use php cacheing.
This course will be used by 3000 users because it´s about a new product and our customer needs to teach it to all their employees. the content is basically JPG images compresed as SWF.
When you say:"Your moodle data can be served from the same server as the DB":
I am not sure if I have understood you, I think that it´s better that the course (Scorm course is stored at moodledata folder) is served from 6 diferent web servers than served by only one server, because this will increase the performance. You say that Moodledata folder can be stored at DDBB server and can be shared by all moodle servers...This could be intererting, How can I do it? but... what about the performance (3000 users launching the same file at the same server)? If I have 6 webservers, Why should I use only one server to serve the Scorm course?
I Will use Coyote E350 with 1GB
Thats where i said you should have load balanced moodledata/db servers. since you are using windows you should be able to create some sort of distrbuted/clustered file system that you can share to all 6 servers. the windows OS should handle the load balancing.
mssql 2k5 can be clustered for high availibility/redundancy. again, windows should handle the load balancing. i am not completely SureHow windows handles the load balanceing but i know t is designed to do that an email or phone cal to thm might answerthat. Again, if you ae worriEd about the server load for the DB server i would drop in 2-3 of the to balace the IO load.
mssql 2k5 can be clustered for high availibility/redundancy. again, windows should handle the load balancing. i am not completely SureHow windows handles the load balanceing but i know t is designed to do that an email or phone cal to thm might answerthat. Again, if you ae worriEd about the server load for the DB server i would drop in 2-3 of the to balace the IO load.
Another option you may want to look into is a SAN. that could solve your moodle data load issues. another thing that i have heard of people doing is using a squid cacheing server to keep content loaded and save load times. truth is there are several options here to set this up, it all just depends on what your up for.
So would guys recommend using a server with a RAID 10 array or several servers to serve this content. What kind of a drive setup would you recommend? SCSI? SAS? I would go wityh 10k to 15k rpm drives. largest cache possible. I just don't know the comparisons between the SCSI and SAS. Also, If multiples were used, how would the file system setup go? And how would you share it to the web servers?
Nice to meet you Martin Langhoff.
It would be great if you can share some of experence in mulitple web servers case.
We are looking for a solution for implement moodle on 2 web server which running on Windows 2008 server + apache + php, it was no problem when I install the first moodle server, but how I can install moodle on second server which point to same same MSSQL 2008 database, the installation process shows the table have been created and updated.
Thank you once agian and hope that you can give me some hints.
It might be a bit late to make a reply, but for the sake of future reference, I would like to just share what I have done. I have 2 windows 2008 servers running apache web server and 1 oracle as database server.
Connection to the database server is pretty straight forward like how you would connect PHP to any of the oracle database server. Make sure both, your web servers are capable to connect to the database server.
As for the web servers, here are the setups.
1. Implement network load balancing on both windows 2008 servers (built in feature in windows 2008).
2. Once network load balancing is up, you can copy your moodle files into both server (make sure they are identical).
3. The crux of the problem is your moodledata. In my case, I use one of the server as the source for moodledata (you can also use an independent file server as well). What I did is to make the moodledata as a share folder. The server where the moodledata resides locally can access the file locally while the remote server (i.e. the server where moodledata file not residing in it) access it via the share folder.
Here you have to pay extreme attention on your apache configuration for the remote server. Make sure they are run as a domain user instead of local system. Make sure the domain user has read and write access.
In your remote server, here is how you access to the share folder
$CFG->dataroot = '\\\\fileserver\MoodleFiles\moodledata';
Notice the extra slash (fileserver is the name of the server where the moodledata resides. MoodleFiles is the share folder where moodledata resides).
3. Once everything is setup properly (i.e. moodle in both servers working), you just have to let the windows 2008 network load balancing does its job.
Hope this helps.
The trouble is then, if you have moodledata on each machine, how do you keep then synchronized? I don't think you can.
I would have one machine available to all the webservers - connected by whatever is fast in the windows world. That being the same machine as the database is just a judgement call really. It strikes me as a little inflexible - I'm guessing the budget is sizable for this one
I would have one machine available to all the webservers - connected by whatever is fast in the windows world. That being the same machine as the database is just a judgement call really. It strikes me as a little inflexible - I'm guessing the budget is sizable for this one
The Moodledata *does* change!!
It isn't only the stuff you upload that lives there. Moodle uses it for all sorts of other things that may not be immediately obvious. For example, session data is stored there (yes, I know this can go in the database but I don't know that it's a very good idea).
All your sites have to see the same moodledata - it will break if not.
Are you sure you don't want to get a Moodle Partner to do this?
It isn't only the stuff you upload that lives there. Moodle uses it for all sorts of other things that may not be immediately obvious. For example, session data is stored there (yes, I know this can go in the database but I don't know that it's a very good idea).
All your sites have to see the same moodledata - it will break if not.
Are you sure you don't want to get a Moodle Partner to do this?
That's interesting Howard, could you explain a bit more about session storage. I'm currently storing sessions in my database, however I find that queries to mdl_sessions are locking the database and causing significant performance issues. I'm considering storing the sessions on my file server and having all web servers read from it. What are your thoughts?
Setup
moodle 1.8.3
4 Web servers load balanced with layer 7 switch
1 Active DB server (1 passive on the way)
1 file server to house moodledata
Setup
moodle 1.8.3
4 Web servers load balanced with layer 7 switch
1 Active DB server (1 passive on the way)
1 file server to house moodledata
Just to address my own issue. I decided to try hosting the moodle sessions on my shared filesystem instead of my dbase. The results were outstanding for me. My site is now lightening fast and dbase concurrent connections are down to a minimum. This along with memcached has dropped my database load from an average of 13 to 1.
I can store session files in Moodle data, there is no problem with this.
I can set virtual cluster in the load balancer, whe a user begins a new session, the load balancer will do that the same server asnwers all the requests from this user in this session, so there is no problem with moodledata folder to store sessions.
> wondering if you would use apache on windows or IIS
or the new kid Lighttpd http://moodle.org/mod/forum/discuss.php?d=83722
or the new kid Lighttpd http://moodle.org/mod/forum/discuss.php?d=83722
I don't.....
If you believe the Moodle Installation notes (and I say no reason not to) it says simultaneous users = 50 * RAM (GB). So, if you do the sums it equates to 1,800 - well away from your figure. Then you go and make it worse by using Windows :-;
Also, have you considered where the data files are going to live? You don't say how many users and/or courses you are talking about but you could be looking at a lot of data. You may need an additional RAID box, hooked up to each web server by NFS (well it would be NFS is you used Unix). You *really* have to watch the performance here. You need Gigabit and fancy switches and again, I wouldn't be happy with Windows to turn in enough performance.
Of course, it all depends on what those 3000 users are actually doing. A little more info might help here if you can talk about it.
At the end of the day, this is serious stuff, you might want to consider getting Catalyst IT to take a look at this. They seem to be the experts in "big" Moodle installs.
EDIT:
This is going to need some careful tuning too. Do you have the experience to make the database work with that number of connections?
Don't mean to be rude, but I would be scared
If you believe the Moodle Installation notes (and I say no reason not to) it says simultaneous users = 50 * RAM (GB). So, if you do the sums it equates to 1,800 - well away from your figure. Then you go and make it worse by using Windows :-;
Also, have you considered where the data files are going to live? You don't say how many users and/or courses you are talking about but you could be looking at a lot of data. You may need an additional RAID box, hooked up to each web server by NFS (well it would be NFS is you used Unix). You *really* have to watch the performance here. You need Gigabit and fancy switches and again, I wouldn't be happy with Windows to turn in enough performance.
Of course, it all depends on what those 3000 users are actually doing. A little more info might help here if you can talk about it.
At the end of the day, this is serious stuff, you might want to consider getting Catalyst IT to take a look at this. They seem to be the experts in "big" Moodle installs.
EDIT:
This is going to need some careful tuning too. Do you have the experience to make the database work with that number of connections?
Don't mean to be rude, but I would be scared
The users cannot upload files, all the content (one course and a few PDFs) will be pre-loaded at the beginning of the project and this will be not changed, so Moodledata is not very important for us.
I am worried about you numbers, 1800 users ...... we can put 8 GB insetad of 4 GB per web server:
8x50=400 usser per 6 webserver= 2400. We will have 2700 simultaneous users
What do you think?
This installation will be done in Portugal, We will contact Catalyst IT ( I am no expert in "big" Moodle installs)
Thanks
I would use 7 web servers if you are worried about the ram. Err on the side of caution. Always be able to accomodate more users than you are expecting to the best of one's abilities. It will be much easier down the road.
Since you will have so many users I would try to setup the course data and database on a high availability system. Even though it will not be changing there will be a high demand for it. I would contact catalyst IT about the setup just the same as I am no expert on these(would like to be and am doing my best but school comes first).
Jose, if you wouldn't mind posting the final setup and load statistics. I would be most appreciative. I am sure other Moodlers would be also.
Greg
Since you will have so many users I would try to setup the course data and database on a high availability system. Even though it will not be changing there will be a high demand for it. I would contact catalyst IT about the setup just the same as I am no expert on these(would like to be and am doing my best but school comes first).
Jose, if you wouldn't mind posting the final setup and load statistics. I would be most appreciative. I am sure other Moodlers would be also.
Greg
Hi Jose
I can't somehow make a clear picture of your project. Let's recollect the requirements:
- 3000 simultaneous users
- only one course, about introducing a new product
- it is already available in SCORM
- it uses dhtml and flash
- users don't _interact_ in any way (uploading files, writing into forums, editing wikis/blogs, chatting, .. none)
- question: do those 3000 users login with 3000 different accounts or will they use a generic account or don't logg-in at all rather use the guest account?
- question: Is this an one-time exercise? For how long? (Can't imagine a company sparing all its 3000 employees for long
)
- web server's OS must be Windows 2003 (see http://moodle.org/mod/forum/discuss.php?d=88921 )
- web server itself could be Apache or IIS (am I correct, did you say somewhere it has to be IIS?)
- database server must be MS SQL Server
- all those servers are in one physical (and logical) network
Looking at the list, the first question that come to my mind is: Is Moodle the correct choice?
And then, it seems like there are higher powers who dictate things like this http://moodle.org/mod/forum/discuss.php?d=88921 If I were you, I'll ask _them_ to finish the job themselves?
Provided that you build a server farm and find software to match, are you sure that your physical medium (100 Mbis/s line?) will deliver the load?
Sorry that I can only bring more questions. I hope it is still better to face them before the exercise than during the exercise. ;-[
I can't somehow make a clear picture of your project. Let's recollect the requirements:
- 3000 simultaneous users
- only one course, about introducing a new product
- it is already available in SCORM
- it uses dhtml and flash
- users don't _interact_ in any way (uploading files, writing into forums, editing wikis/blogs, chatting, .. none)
- question: do those 3000 users login with 3000 different accounts or will they use a generic account or don't logg-in at all rather use the guest account?
- question: Is this an one-time exercise? For how long? (Can't imagine a company sparing all its 3000 employees for long
- web server's OS must be Windows 2003 (see http://moodle.org/mod/forum/discuss.php?d=88921 )
- web server itself could be Apache or IIS (am I correct, did you say somewhere it has to be IIS?)
- database server must be MS SQL Server
- all those servers are in one physical (and logical) network
Looking at the list, the first question that come to my mind is: Is Moodle the correct choice?
And then, it seems like there are higher powers who dictate things like this http://moodle.org/mod/forum/discuss.php?d=88921 If I were you, I'll ask _them_ to finish the job themselves?
Provided that you build a server farm and find software to match, are you sure that your physical medium (100 Mbis/s line?) will deliver the load?
Sorry that I can only bring more questions. I hope it is still better to face them before the exercise than during the exercise. ;-[
Hi Visvanath,
- 3000 simultaneous users
- only one course, about introducing a new product
- it is already available in SCORM
- it uses dhtml and flash
- users don't _interact_ in any way (uploading files, writing into forums, editing wikis/blogs, chatting, .. none)
JOSE>>>All these items are correct.
- question: do those 3000 users login with 3000 different accounts or will they use a generic account or don't logg-in at all rather use the guest account?
JOSE>>>3000 different accounts using LDAP.
- question: Is this an one-time exercise? For how long? (Can't imagine a company sparing all its 3000 employees for long 
)
JOSE>>>No, It´s a short time course (aprox 12 hours) The company has 9000 employees who will use the course by groups)
- web server's OS must be Windows 2003 (see http://moodle.org/mod/forum/discuss.php?d=88921 )
JOSE>>>Yes Its mandatory
- web server itself could be Apache or IIS (am I correct, did you say somewhere it has to be IIS?)
JOSE>>>correct, could be APACHE or IIS, or lighttp
- database server must be MS SQL Server
JOSE>>>Yes Its mandatory
- all those servers are in one physical (and logical) network
JOSE>>>Correct, all those servers are in one physical (and logical) network
Looking at the list, the first question that come to my mind is: Is Moodle the correct choice?
JOSE>>>I dont know if Moodle is the better choice, I am still trying to know it
STOAS and other Moodle partner seem to be sure that Moodle can do it (http://moodle.org/mod/forum/discuss.php?d=88921)
And then, it seems like there are higher powers who dictate things like this http://moodle.org/mod/forum/discuss.php?d=88921 If I were you, I'll ask _them_ to finish the job themselves?
JOSE>>>Perhaps we should contact STOAS directly.
Provided that you build a server farm and find software to match, are you sure that your physical medium (100 Mbis/s line?) will deliver the load?
JOSE>>>I think that this bandwidth could be enough, but our customer sould have to design and provide it. our job is design and install Moodle for 3000 simultaneous users.
Sorry that I can only bring more questions. I hope it is still better to face them before the exercise than during the exercise. ;-[
JOSE>>>You are right
Hello Jose,
interesting issue. I am working at Stoas so let me clarify our experience with Windows setups for our clients:
1. We prefer to use Linux and strongly advice our clients to use the regular Linux Apache postgres/mysql PHP setup with eaccelerator which is working fine for all our clients.
2. Don't use Windows
3. For some clients points 1 and 2 are irrelevant because company policy dictates them to use a certain hosting environment / architecture. Obviously, you are also in that situation. For those cases my 2 cents:
For one of our clients we did some load testing in the precontract track on a LAMP setup in comparison to a Windows2K3,IIS,SQLserver2005,PHP setup. Initially we found an order of magnitude difference between the two setups. Note that the initial setup was with all components on one (virtual) server. Moving the database to a seperate server already showed some improvement in performance but was still nowhere near the LAMP figures. Based on those tests, the client decided to outsource the technical Moodle hosting to an Indian software/hosting/consulting moloch. Another condition of the client was that Moodle is compiled with ioncube (http://www.ioncube.com/). So after our initial testing the Indians took over. I don't have an update yet about their progress as they are implementing the environment as we speak.
Some things to note here are that our client is a huge globally operating company with several hundreds of thousands of users (i.e. enrolled, concurrent I don't know yet). Their Moodle environment will probably be much more heavily used 24/7 (users uploading files, interacting users in forums / chats, users taking quizzes) than your environment which is basically, if I understood correctly, a kind of rich SCORM player.
Summarising, your performance requirements are much smaller than of our client and, moroever, you are not tied to IIS, which is a huge advantage. I think that using your setup in combination with Apache/PHP would deliver enough performance for your client. The short time of testing we did gave me the feeling that the bottleneck in the MS setup was the combination of IIS-PHP. I know this sounds vague but I do not have so much experience with IIS-PHP and Googling the issue showed lots of users with similar experiences, but not much concrete solution strategies. Moreover I didn't have time to dig further into the problem so, unfortunately, I can't give you more useful details.
Hope this helps. Of course, feel free to contact us if you have more questions. Anyway I am very much interested in your experiences with the eventual setup and performance.
interesting issue. I am working at Stoas so let me clarify our experience with Windows setups for our clients:
1. We prefer to use Linux and strongly advice our clients to use the regular Linux Apache postgres/mysql PHP setup with eaccelerator which is working fine for all our clients.
2. Don't use Windows
3. For some clients points 1 and 2 are irrelevant because company policy dictates them to use a certain hosting environment / architecture. Obviously, you are also in that situation. For those cases my 2 cents:
For one of our clients we did some load testing in the precontract track on a LAMP setup in comparison to a Windows2K3,IIS,SQLserver2005,PHP setup. Initially we found an order of magnitude difference between the two setups. Note that the initial setup was with all components on one (virtual) server. Moving the database to a seperate server already showed some improvement in performance but was still nowhere near the LAMP figures. Based on those tests, the client decided to outsource the technical Moodle hosting to an Indian software/hosting/consulting moloch. Another condition of the client was that Moodle is compiled with ioncube (http://www.ioncube.com/). So after our initial testing the Indians took over. I don't have an update yet about their progress as they are implementing the environment as we speak.
Some things to note here are that our client is a huge globally operating company with several hundreds of thousands of users (i.e. enrolled, concurrent I don't know yet). Their Moodle environment will probably be much more heavily used 24/7 (users uploading files, interacting users in forums / chats, users taking quizzes) than your environment which is basically, if I understood correctly, a kind of rich SCORM player.
Summarising, your performance requirements are much smaller than of our client and, moroever, you are not tied to IIS, which is a huge advantage. I think that using your setup in combination with Apache/PHP would deliver enough performance for your client. The short time of testing we did gave me the feeling that the bottleneck in the MS setup was the combination of IIS-PHP. I know this sounds vague but I do not have so much experience with IIS-PHP and Googling the issue showed lots of users with similar experiences, but not much concrete solution strategies. Moreover I didn't have time to dig further into the problem so, unfortunately, I can't give you more useful details.
Hope this helps. Of course, feel free to contact us if you have more questions. Anyway I am very much interested in your experiences with the eventual setup and performance.
Thanks Arjen,
I think that the major problem could be IIS-PHP too.
Do you know if STOAS have completed any large Moodle-Windows installation? Marcel Verberkt says that you have done it (http://moodle.org/mod/forum/discuss.php?d=88921)
Marcel pointed me at your topic. What he meant is that we were involved in a large Moodle-Windows installation in the pre-contract track. As described above, the client decided to go big after seeing the performance issues on Windows. I am expecting test results from the hosting party today or tomorrow but I don't know if I can disclose them to the public yet.
However, we do have plenty of hands on experience and knowledge in setting up and managing load balanced Windows servers for e.g. Blackboard and Questionmark.
However, we do have plenty of hands on experience and knowledge in setting up and managing load balanced Windows servers for e.g. Blackboard and Questionmark.
I think the installation you propose is possible. However, there might be some hidden caveats depending on your particular install, so I would recommend intensive testing with e.g. Apache jMeter.
See also the post about authentication in this thread of Alex de Pena warning for large overhead when using LDAP authentication.
See also the post about authentication in this thread of Alex de Pena warning for large overhead when using LDAP authentication.
In my experience the Authentication overhead with a large number of simultaneous users is very significant. We now use two LDAP Servers to handle the load. We set up a round-robin DNS entry to spread the load.
I don't know anything about Window/IIS tuning, however on our server running RedHat4 and Apache2 after careful tuning and testing we are able to serve 1500-1900 simultaneous users. One Moodle Server 8GB memory, 2 LDAP servers, 1 Database Server.
I don't know anything about Window/IIS tuning, however on our server running RedHat4 and Apache2 after careful tuning and testing we are able to serve 1500-1900 simultaneous users. One Moodle Server 8GB memory, 2 LDAP servers, 1 Database Server.
Hello Alex,
I am interested in getting information about your Moodle setup and your tuning information. Please let me know if you are willing to share it.
We are looking at growing our Moodle site out and I think your numbers seem to be about the size we would anticipate optimizing to.
Thank you,
Tammy
I am interested in getting information about your Moodle setup and your tuning information. Please let me know if you are willing to share it.
We are looking at growing our Moodle site out and I think your numbers seem to be about the size we would anticipate optimizing to.
Thank you,
Tammy
Tammy did you ever get those resources for your moodle setup?
Hello Alex,
I am interested in getting information about your Moodle setup and your tuning information. Please let me know if you are willing to share it.
We are looking at growing our Moodle site out and I think your numbers seem to be about the size we would anticipate optimizing to.
Thank you,
Tammy