We have moodledata on nfs & I've often wondered if taking away the session from moodledata might make a large speed difference. An unscientific test I did quickly didn't support that intuition so I'm intrigued at what others do for sessions.
I can vouch for this because, as of now, we do do it.
It causes lots of problems if you get sequential requests on different servers. For example, a request comes to Server A, it updates the session. Almost simultaneously, another request from that user comes to Server B; this uses the old session. Or it tries to change the session as well. Or they both try to change it at once and it actually gets corrupted, causing users to get logged out.
So basically you get rare, hard-to-explain problems. 'I looked at this page and half of the images didn't load' or 'I went to post this forum message and it told me my session is invalid and now I lost it' etc.
We are trying to switch our load balancer to sticky sessions, but there are other difficulties (to do with adequately balancing load in this configuration) that mean we haven't achieved this yet. When we do though, this will remove these sporadic problems; and also we'll be able to put sessions on local disk which should be a significant performance gain (local disk = instantaneous).
memcached might be an improvement also but sticky sessions will give you large benefit for other areas that rely on NFS, otherwise you can get into 'I uploaded this file to the course but now moodle says it isn't there' territory as well. And if you use sticky sessions, you don't really need to use memcached.
My opinions only, I think others here have a slightly different take on it.
We already use sessions successfully with NFS
The reasons we don't use local disk are: a) the hardware loadbalancer we have seems to have limited stickiness timeframe (< 1hr iirc) b) it makes bringing down servers for security updates more inconvenient for users
otherwise you can get into 'I uploaded this file into the course but now moodle says it isn't there territory'
I don't see how you can get into that teritory, surely a write of a file should instantly be availbe for reading on other servers?
The thought behind moving to memcached it reduces the expensive writes to reads for the majority of requests.
The corruption and short-term inconsistency problems definitely occur on our NFS system. I don't really know why - this isn't reproducible (if you do a trivial test like read a file, immediately write it on another server, read it again, it works) so must be some kind of timing issue. Could be something to do with OS version.
By the way, we did test a memcached cluster - I asked about it for you. Couple things:
- When testing it they couldn't get the system to work reliably when one of the memcached servers was down. After that every request had a 15 second delay, which would probably not be acceptable. It must be possible to make it not do this, but the alternative solution came along so they didn't spend more time on it. Consequently, they didn't get around to testing performance (even if it's fast, creating a new single point of failure didn't sound like a good idea).
- To avoid introducing other problems they made it use memcache only for sessions, and not for database queries, which apparently required a hack in moodle setup.php.
Also, I take your point about (b) if servers go down [or are taken down], using local disk will lose current user sessions as those users get directed to other servers. That is certainly a disadvantage.
With GFS, watch out for it's issues with slow access for small-files, like those used in Sessions. For this reason a better option is simply to store the sessions in the database, Moodle has this as a default option. On PGSQL or MySQL this would perform very well. On Oracle it has it's issues due to the CLOB data field, but nothing that can't be managed.
We have been intending to test Memcache sessions, but we have had no real need to actually investigate it as yet as storing the sessions in the Database has been near-perfect.