My name is Anthony McKenna and I am engineer at Domino's Pizza LLC. My team is working to adapt Moodle to fit our needs for an intranet-based knowledge assessment center for our team members. We're piloting the project now with a limited number of stores and teams.
We're working on estimating the requirements for a production environment. We are using RedHat and MySQL. We have approximately 110,000 users and are anticipating a maximum of 5,000 simultaneous users.
I am finding a wide array of results from different installations with different equipment setups. However, I have not seen very many installations with 100,000+ users. I would like to hear from anyone who has clustered Moodle for a very large number of users and how this system has performed for them.
What setups have worked for you and your large organization?
Welcome to Moodle! - I work for the Moodle Partner Catalyst IT, we have offices in UK/AU/NZ and are very experienced with high-user sites - we can provide a large range of services inlcuding support with performance tuning and scaling of your site. Drop me an e-mail if you have any questions! - For more information on our services see: http://www.catalyst-eu.net/
There is some stuff about the OU environment in, or linked to from, my Boss's blog: http://learn.open.ac.uk/mod/oublog/view.php?user=27287.
I will also note that when we were starting out at the OU, we consulted with Catalyst IT, and they gave us useful advice. Probably several of the other Moodle partners would have similar experience, but I can't speak from experience there.
At LUNS, one of our largest Moodle deployments was to 250,000 users in schools in the UK. The setup included load-balancers to spread the load, a database cluster and file-stores on the back-end, and additional servers for running bespoke software for administrative tasks/cron. You may take a look at my colleague Andrew's presentation about Mahara scalability, where he is talking about many aspects of this setup.
Some general tips from our experience:
- Invest in your database servers and shared file storage. Scaling out web-servers is relatively straight forward and easy to grow with demand. Shared file storage and databases are less easy to achieve.
- Consider your disk layouts carefully when you specify and buy your hardware - particularly for your database servers.
- It should be possible to virtualise most parts of the infrastructure, but I’d highly recommend against virtualising the database components.
- Use Postgresql if you can. We used MySQL (due to migration reasons) and whilst MySQL can achieve this kind of scalability, we would not recommend it and would not deploy it if starting of fresh. MySQL replication is worryingly fragile and high concurrence can really degrade MySQL performance in high usage scenarios. If you need to go for MySQL please ensure you go with innodb (which I believe will become enforced in future releases of Moodle) else your table locks will kill performance.
- Use a PHP op-code cache such as APC, this often gets overlooked but can dramatically decrease CPU demand and reduce hardware requirements. Don’t forget to tune it.
- Carefully consider any third-party modifications - much of Moodle core code gets well tested on high user sites, many third party Modules do not.
- Use solid metrics! Don’t spend time optimising without any metrics to analyse if performance is actually improved.
I'm afraid that installing PHP on a tomcat web server isn't going to get you very far in terms of improving scalability.
Tomcat is a Java Servlet server (see http://tomcat.apache.org/ for more information). Moodle is written in PHP (not Java) so although you could install PHP5 on your Tomcat webserver to actually run Moodle (At least I *think* you can do so), it's unlikely that it would be any faster than using a stock apache2, nginx, or lighttpd server.
I didn't understand this question either - however I think (vaguely remember, could be wrong) there may be a Java implementation of PHP that converts PHP scripts to Java bytecode of some sort. This would probably sit within Tomcat.
If this even exists, it runs a restricted subset of PHP and definitely will not work for Moodle.
(And even if it did work and there was, say, a 100x performance improvement on CPU-bound PHP code, the limiting factor in Moodle installations is generally database performance anyway...)
BTW... my experience is - choosing my words carefully here - that for most people the web server running out of memory is the limiting factor. Nothing that loads more memory or setting up apache.conf properly won't mitigate. Thus shifting the problem to the database.
Thanks for link. Yes that is the one I meant. It doesn't work with Moodle, as you say; I skimmed the list of unsupported functions and spotted a couple just like that, also I think there are some extensions that aren't on their list. Maybe worth watching for the future but...
If you don't think Java is fast on server, you haven't been paying attention. It benchmarks at essentially the same speed as native C code these days, so there is nothing faster. You can of course make slow Java systems if you like (should you want to, there are many frameworks available to help you do just that), but it isn't the language or the runtime's fault.
Java is not scalable any more than PHP is scalable. For instance, there was once lots of Moodle code that did things like 'load entire user table into memory, and then...' - this falls over on large sites and it would fall over just as much on Java as it does on PHP. By the way, this type of code pattern isn't entirely a thing of the distant past - it came up recently with regard to moodle web services...
Regarding your point about other limiting factors - I agree, of course you're right. What I meant is that database performance is a harder factor to work around than front-end CPU usage, for which there are various solutions (use some kind of accelerator, use more than one machine).
At Remote-Learner (US, Canadian, UK) Moodle partner company we are supporting a number of very large installations, like yours. Happy to help if you contact us. You might also be interested in ELIS for Moodle, an enterprise extension we have developed.