Hardware and performance

Of course, an even cheaper solution is to throw the LB away and just use DNS round-robin. 

As long as you are monitoring your web servers properly it works fine and avoids LB oddness. Just a thought...

I have seen a situation where chucking out the load balancing lead to a dramatic improvement in performance, Not sure why.

Hi Don,

I assume that this is a Linux server, so check the relevant logs (the software in question, and also the kernel logs).

I wonder whether you may be hitting a max open files limit, or something along those lines. You'll see this in standard `dmesg` output, or in your kernel logs. I forget the exact output you get, but it will be something about either max open sockets, or max open files per process, or max open files. Typically it's relatively low on a new system and will need to be tuned up according to the capability of your system. Whenever your load balancer receives a connection it uses a socket for the inbound, and then an additional socket for the outbound connection. Others may also be used if you have intermediary steps.

Have you got any monitoring on your system (e.g. Icinga)? This can be invaluable in helping you to track down such issues.

I've not used Ultramonkey with Pound before -- I've always used haproxy extensively for Moodle loads with no ill-effect. It's a powerful, free, open source, load balancing tool, with low TCO and setup. It's relatively easy to setup, with a good manual. I've run it for pretty large installations (>= 20,000 current users) with no ill effect. The latest versions have SSL capabilities built-in (previously you could not terminate with haproxy), and supports quite a number of features. At Lancaster, we ran a pair of haproxy servers (2vCPU, 1GB RAM) with Keepalived for IP failover between the boxes. We didn't bother with Pacemaker or heartbeat - we deemed it unnecessary for a pair of dumb boxes which just passed stateless traffic.

Hope this helps,

Andrew

How is your load balancer set to handling http traffic? Can you have it do load balancing HTTP traffic with cookie persistence? I think a few things start acting up for users when you have them bouncing between apache nodes while in the same session. I only noticed errors with our load balancer with items using localcache and localtempdir set on each node.

Since documentation on what parts of moodle user localcache and localtempdir its a little hard to say if a user going through Apache node 1 is taking a quiz and when they click next they jump to Apache node 2 but the work done on node 1 has stored files and caches in tempdir and localcachedir that the Apache node 2 is looking for within its own local folders. So your browser is stuck spinning waiting files to to load. 

Hi Don,

Another throught occurs: Where are you storing your sessions?

I would recommend not storing them in moodledata, especially if it is on a shared filesystem. The reason for this is that you may encounter file-locking issues.

I would recommend that sessions are stored either in DB, or if you prefer in memcached.

Best wishes,

Andrew