I am enjoying the "Teaching with Moodle" MOOC course. This is the first time that I have been in a Moodle course as a student.
What makes this course work so well, from a hardware and software viewpoint? Could the following information, and files be shared so that we can learn from them? I believe that this course illustrates how to set up Moodle well. I am not sure if any of this is confidential, and I know that I am asking for a lot, but any of this information would be appreciated.
1) Server specifications (Hardware, and Apache version)
2) Moodle version
3) Moodle theme, and modifications to the them
4) mysql my.cnf configuration file (uploaded)
5) php.ini file (uploaded)
6) config.php Moodle file (uploaded)
7) Output of info.php (Site Administration|Server|PHP info)
8) httpd.conf file (uploaded)
9) Moodle settings (I am not sure how one shares these. Maybe this is a table in Moodle).
Don't think the MOOC audience was intended to be System Admins/Moodle Admins. While everyone could share what you've listed in their profile right now, think anyone who has never been a true server administrator (shared host/dedicated host) would have to be taught how to 'share' in a safe manner. Nearly all of those config files would have to be edited to remove some items that shouldn't be shared and those that are very specific to the server/hosting plan/company, etc..
Would remotely hosted folks on shared systems really benefit by what would surely be information overload and just as confused?
I certainly would not take an httpd.conf nor a php.ini nor a my.cnf and just drop it in (replacing what I had). Someone desparate for a quick fix might attempt to use a php.ini or an httpd.conf or a my.cnf as is - a VERY BAD idea.
Maybe there needs to be a MOOC designed and targeting the Moodle Server admin who is also the true server admin. Boy, even that would be an undertaking ... how may different Linux distros would have to be covered? Also Windows and MacOSX. how many hosting providers?
Just my 2 cents!
'spirit of sharing', Ken
Rick (Ken) did you mean you wanted Moodle HQ to share how they had set up the Learn Moodle site? That's how I understood it when I read your post. I can't give any help because I am completely non-technical although the theme is a custom one for which the link has been posted somewhere on here and we're running the latest Moodle version. I daresay at a point in time Martin or Matthew our sysadmin will provide more details - at the moment we're just into day 2 /3 and still in the thick of it
Edit: As I was writing this, 3. Sep 2013 9 h UTC, http://learn.moodle.net gave errors. CloudFlare gave Error 502 and 524 telling it can reach Frankfort but not learn.moodle.net.
you can read about the overall picture here: https://moodle.org/mod/forum/discuss.php?d=236151#p1025673.
It lacks of many details - otherwise, too much disclosing of a sensitive infrastructure - but I guess that PostgreSQL/MariaDB Galera Cluster is the DB server while I wonder what could be the web server, Apache or nginx.
The key points are the usage of SSD (probably with $CFG->localcachedir, MDL-40545) as well as - really important, makes the difference - of memcached as the backend for the MUC cache definitions.
Thanks to all of you for your posts and thoughts.
Ken, yep, there could be a danger that some folks might try to use these config files as is. I was hoping to look at some of the settings, and to learn from them. I am one who knows that one must be very careful making changes.
Mary, things are going well with the MOOC. I was interested in performance because the course was to start at a specific time, which meant that there would be a high number of concurrent uses. This question about "concurrent users" and performance occurs in the Moodle.org forums, so I was interested in seeing how HQ's Moodle install would work. By the way, I wasn't logged in when the course started, I logged in later in the day.
Matteo, thanks for the additional information. Looks like a pretty powerful environment. It appears that you are saying the Moodle site is running PostgreSQL, not mySQL...right?
Visvanath, Moodle 2.5.1+?
To all: Which theme is being used? Has it been modified?
yes, powerful and ready to be scaled out: I missed to mention that the other key point is having enough memory to potentially store the DB into memory.
What DB server? Only Matthew Spurrier knows the exact answer : PostgreSQL is the preferred choice of most people - AFAIK moodle.org was running on top of it before being deployed into this new infrastructure - while having 3 server for the DB cluster recalls to me also the option of having MariaDB (more than a MySQL drop-in replacement) deployed with Galera which requires n+1 servers, where n is the numer of the redundacy, and InnoDB, required for Moodle.
Database cluster is running a modified version of mysql (Percona XtraDB Cluster to be exact), which uses Galera for it's innodb replication, and using the xtrabackup system for non-blocking backups and joiner transfers).
Each of the DB servers do have quite a fair amount of RAM, however there is no query caching occurring due to the inherent nature of replicated architecture, it just doesn't work.
That said, our infrastructure is under very low amounts of load due to the configuration and other caching levels on and in front of the application it's self.
The database servers are multi-master, and the load is distributed by haproxy running on the local application servers, this also serves as automated failover which takes a host out of the loop should it fail or report that it is not ready.
The database setup it's self requires a minimum of 3 nodes, either 2 data nodes and an arbitrator, or 3 data nodes, we're using 3 data nodes to reduce the risk of split brain, and allow the sites to remain operational in the event of a db server outage.
I'm not going to provide configuration files, because they've been developed for our infrastructure, and are not plug and play so to speak, plus if you dropped in our mysql config you would more than likely corrupt your own innodb files due to size changes in the ib_logfile's. Plus there's plenty of documentation around if you know where to look.
There is no local application cache being stored on the application nodes, in fact, they can be treated as ephemeral, all application cache is stored in memcache which is accessed by both servers, moodle data is accessed through a gluster cluster with some special configurations to get around temperamental behaviour (disabling overrides in apache to prevent the lookups of .htaccess on each read was a big one, as is nodiratime,noatime on the gluster server mounts).
There are only 2 gluster nodes, as bandwidth requirements grow linearly the more replica nodes you add, as you write to a gluster mount, it writes to all servers at the same time, so write 1MB in 1 second, it requires 3MB/s.
SSD's tend to make a huge difference in request latency for gluster, and also significantly speed up mysql queries (particularly writes).
At the learn moodle launch we had over 500 users on learn moodle, along with another 400 or so on moodle.org all using the same backend infrastructure, the resulting load on the servers was negligible.
Another thing that tends to help a lot is caching of static assets in nginx, specifically those stored in the data directory (which gluster would have served).
For those interested, I run Apache with mod_php, and nginx as a reverse proxy/static asset cache in front, this allows apache to do it's work, and allow nginx to handle external tcp connections freeing up apache's threads for more work.
Hi Matt, I found your reply helpful since I am tuning a site setup with nginx (the free version) configured as a load balancer in front of 2 production servers. I would be ashamed to hijack this thread, but I am looking for confirmation or comments on a solution I might have found to my problem related to nginx.
The problem is that NGINX, has a default timeout of 300 sec, so NGINX returns an error (upstream timed out ,110: Connection timed out), when the apache backend (upstream or production servers or whatever we choose to call them) is taking very long time to process a request, like huge backups or restores or as I have experienced, cache performance tests for 50000.
Rummaging through php documentation to find the function set_time_limit (), and grepping the Moodle code I found (in Moodle 2.6)
# grep -r set_time_limit * |grep -v 'set_time_limit(0)'|less
admin/tool/uploaduser/index.php:@set_time_limit(60*60); // 1 hour should be enough
backup/controller/backup_controller.class.php: set_time_limit(1 * 60 * 60); // 1 hour for 1 course initially granted
lib/filelib.php: @set_time_limit(60*60); //reset time limit to 60 min - should be enough for 1 MB chunk
So in order to cover those absolutely worst case scenarios, NGINX would have to wait for an hour for a reply from the backend. In order to cover this, an NGINX parameter was used:
As shown in this example. Instead I would set it to 3600. What do you think?
Also thanks for the information about tuning the moodle data filemount, I would like to post some more about that.
A minor correction: NGINX, has a default timeout of 60 sec
I'd like to follow up a bit on Rick's questions, if that's OK.
The answers here suggest that the Teaching With Moodle course was just run on the same instance of Moodle that the rest of moodle.org (including these forums?) runs on - is that right?
I'd love to know what process you went through to assure yourselves that the architecture/specification you had was going to be adequate for the number of students you had on the MOOC - did you come up with estimates based on total numbers of students, number of simulataneous logins, nature and number of activities, assignments, etc?