Moodle sessions timing out

Moodle sessions timing out

by Adam Norwood -
Number of replies: 12

Hi There,

We are using Postgresql for sessions and recently, within the last few weeks (though no updates have happened during this time) we have noticed that there has been a sharp increase in the number of users which are reporting that they're unable to log in as the session wait timeout is being hit. 

What I'm seeing is a bunch of long running processes seem to start piling up in the database just prior to these issues appearing, and my current theory is that these processes are locking the database in some way which stops new sessions being made - But I'm not sure what the queries are doing (I'm far from a DBA), why they're doing it or how to stop it happening. I've seen a few suggestions that I should move to File-based sessions but as this problem only arose recently I'm keen to understand what the problem is and not just work around it as it could be impacting other things

the three most common queries I'm seeing (in order of how commonly i'm seeing them run over 1 minute) are

SELECT * FROM mdl_user WHERE mnethostid = $1 AND id = $2 AND deleted = $3

SELECT uif.*, uic.name AS categoryname , uind.id AS hasuserdata, uind.data, uind.dataformat FROM mdl_user_info_field uif LEFT JOIN mdl_user_info_category uic ON uif.categoryid = uic.id LEFT JOIN mdl_user_info_data uind ON uif.id = uind.fieldid AND uind.userid = $1 ORDER BzqY uic.sortorder ASC, uif.sortorder ASC

SELECT pg_advisory_lock(9910095)


Any help would be appreciated! I am somewhat at a loss

Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Adam Norwood -
Just to add, I'm also seeing the following in my Apache error log

Cannot obtain session lock for sid: XXXXXXXXXXXXXXXXXXXXXXX
Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Personally, I wouldn't use a database for Sessions. I wish I had an explanation but I've never seen it work well.

What does work very well is to use Redis. While you're there, you probably want to be using it for your MUC as well.
Average of ratings: -
In reply to Howard Miller

Re: Moodle sessions timing out

by Adam Norwood -

Hi Howard,

Thank you for the reply, I can certainly agree with those optimisations, though I did have a query or two to follow them up with

For file based sessions - How much disk space am I likely to need with these? I don't have a lot of disposable disk space to play with and it's fairly crucial that the server doesn't go belly up because sessions filled up all it's space. Even a rough ballpark figure would be good

I did try Redis in a test environment, but when I did the cache benchmarking for redis versus the file system, I found that redis fell short on all but one of the results moodle gives and I'm not sure why (Home>Site administration>Plugins>Caching>Test performance), I don't suppose you could offer any tips or an explanation why?


All the best


Adam

Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Thorsten Bartel -
Picture of Core developers

Hey Adam,

sessions really don't need a lot of space. They're usually in the range of 3 to 130 kB (in my experience). We're having about 60k active sessions with an average of 7.4 kB per session. That's roughly 430 MB of usage for a pretty active system.

That being said, depending on how much usage you'll get, I strongly advise against file sessions, as even on high performance RAID systems in a virtual environment, disk I/O eventually becomes a bottleneck. We recently moved from file sessions to memcached and got rid of a whole lot of problems at once.

But then again, those were peak usage times at the beginning of a term with about 1-2k concurrent users (active processes on the webserver).

If your requirements are well below that, you might do well enough with file sessions.


All the best

Thorsten

Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
+1

If you're changing session handlers moving to file-based sessions is not really an improvement.

I would strongly recommend investigating a proper cache. I would use Redis, but memcached will still be loads better.
Average of ratings: -
In reply to Howard Miller

Re: Moodle sessions timing out

by Adam Norwood -
Hi Howard,

I'm looking at both memcached and redis, however I have Redis setup on a test box (which is a carbon copy of the live site, but with less hardware resources) and the cache testing I've done suggests it's considerably worse than file-based in all but SET operations, see attached image

I was not expecting Redis to perform so poorly, and I've tried to follow Moodle's own optimisation tips as closely as possible - I don't suppose you could suggest a reason Redis is performing so poorly?


Attachment rediscachetest.JPG
Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
That's not the interface for Sessions. The 'Session cache' is a different thing entirely.

You need to enable session handling via some lines in config.php (see config-dist.php)
Average of ratings: -
In reply to Howard Miller

Re: Moodle sessions timing out

by Adam Norwood -
Hi Howard,

I think the larger point is that if Redis isn't performing any better than files for caching, it's only job, why would I expect Redis to work any better than Files for Session caching?
Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Thorsten Bartel -
Picture of Core developers
Hey Adam,

"I'm looking at both memcached and redis, however I have Redis setup on a test box (which is a carbon copy of the live site, but with less hardware resources) [...]"

Just to clarify: Have you set up Redis on the same host as the moodle instance you are testing it with?
What are you using as session handler? Whatever it is, it seems to be some orders of magnitude faster than either file cache or Redis when testing session cache speeds.
Average of ratings: -
In reply to Thorsten Bartel

Re: Moodle sessions timing out

by Adam Norwood -
Hi Thorsten,

The current session handler is our Postgresql Database, which I keep being told is one of the slowest methods of storing session information.

And yes Redis is on the same host as the Moodle Instance, I know this isn't optimal but I wasn't expecting an on-server cache to be worse than no cache at all
Average of ratings: -
In reply to Adam Norwood

Re: Moodle sessions timing out

by Thorsten Bartel -
Picture of Core developers
Hi Adam,

first of all, sorry for making you repeat yourself, I see now that I could have gotten the information about your current session handler from your original post.

I'd strongly suggest testing various session handlers (file cache / Redis / memcached) and how well they perform for you with something more than just the cache store performance reporting tool. Actually set them up as the session handler and then compare the line they output under "Session Cache".

Just to give an example... for this setup, I put the same memcached server I use for a session-cache as a test-server for memcached, yet their numbers differ by a factor of 100 (or greater).
I don't know what causes this huge difference, but you'll only know for sure if you actually set it up.

Numbers are with 10k requests.

P.S.: I'd actually say that setting up the Redis / Memcached on the same host is beneficial, since it's way faster than having to connect over the network. This only applies if you're not using a clustered system, though.
Attachment sessioncache.png
Average of ratings: Useful (1)