What to cache with MemCached

What to cache with MemCached

by curt bixel -
Number of replies: 19

I have Memcached up and running on my site, but am not sure if I really need it, if it really helps, or if it needs further configuration.  

I am running moodle 2.8 on a VPS 2000 account at Cloudways.  This gives me 2GB of ram.  The site actually runs fairly well with this, with the only performance issues happening when too many students are taking a quiz at the same time, if the quiz has many images.  

Anyway, I see a "cache administration page" under  site administration > plugins > caching > configuration.

Is there something further there that needs to be set up to take advantage of memcached?  

I don't see much activity on memcached on my server monitoring page at cloudways...

Average of ratings: -
In reply to curt bixel

Re: What to cache with MemCached

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If it doesn't increase your page load times all the time then I wouldn't bother. 2GB is very little, so if memcached is eating into that then I definitely wouldn't bother. 

It makes a (big) difference when reading/writing to memcached is significantly faster than the default cache store which is the disk. If your disk is reasonably fast you may not notice the difference. 

In reply to Howard Miller

Re: What to cache with MemCached

by curt bixel -

Hmmm...  When I look at my free memory, it looks like I rarely even begin to eat into the 2GB of RAM that I have.  I was a bit surprised by that, but it looks like the bottleneck is not there.  

I am interested in trying out caching some things with Memcache to see if it works.  When I go to the site administration > plugins > cache > configuration, I can see where things are mapped to different caches.  

It looks like most of the application cache goes to APC and session cache goes to the "default" cache. 

Can you tell me what and how to change these settings to get the correct things to cache with memcached?  

I am not finding directions for this in the moodle documentation.  


In reply to curt bixel

Re: What to cache with MemCached

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

For a start, have you *definitely* got memcached set up properly? You're looking for a tick in the Ready column (next to memcached) in the top table. You can then create an instance pointing at your memcached server which will appear (again with a tick) in the 'Configured store instances' table. 

If that's all in order, you can go right to the bottom of the page and click 'Edit mappings'. You can then point the Application store at your new memcached store. 

You will get a warning that memcached does not support all cache types but you can ignore that - well, it's an undocumented 'feature' and it's never caused me bother wink

You are right that (for such an important feature) the documentation is sadly lacking. I'm unhappy about that too but nobody who fully understands the MUC cares angry

Average of ratings: Useful (2)
In reply to Howard Miller

Re: What to cache with MemCached

by curt bixel -

Howard,

Thanks for your reply.  I think I do have Memcached set up properly.  I have both the tick marks you mentioned above, and when I test my server using the "test" function, I can see hits registering to memcached on the monitoring statistics on my server.  

Other than running the test function, I was not able to get any activity on memcached.  

I did just go on and point the Application cache to Memcached, but before it was pointed to APC which I have heard might be even a bit faster than Memcached.  Can you tell me anything about that?

Finally, is there any way to point the "session" or "request" caches to memcached?

In reply to curt bixel

Re: What to cache with MemCached

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Paul is right... if you are swapping from one in-memory cache to another I wouldn't expect to see much difference. 

Unfortunately, I have no idea why the memcached store is incompatible with some of the caches. I have asked the question a number of times but (as yet) no response from the devs. 

Average of ratings: Useful (2)
In reply to Howard Miller

Re: What to cache with MemCached

by Paul Verrall -

Howard,

Of the default application caches only 'Event Invalidation' is not supported by Memcached. 

Event Invalidation requires 'Data Guarantee'  which ensures data exists in the cache once it is put there. It is *never* cleaned up to free space or because it has not been recently used. Because (as discussed above/below?) Memcached evicts objects form it's cache it cannot be used Event Invalidation.

The Event Invalidation cache is used to ENSURE events are handled correctly, i.e. the objects exist when they are created and don't only when they are purged, this may not occur if the cache object vanishes unexpectedly.

HTH

Average of ratings: Useful (6)
In reply to Paul Verrall

Re: What to cache with MemCached

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

What would be *really* useful would be....  1. if some outline information about this was added to the MUC end-user docs (Caching) and 2. if a table could be added to that page showing what caches support which (required) facilities. 

I know I seem a bit grumpy about this, but we are currently in a situation where some caches simply don't work for certain cache types completely without explanation.  This is not a good thing wink

Average of ratings: Useful (3)
In reply to Howard Miller

Re: What to cache with MemCached

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Isn't the documentation a wiki that anyone (e.g. you) could improve? Not sure if it is restricted...

I don't think this specific issue you've raised is mainly a documentation problem (although it probably could use improvement). Rather, the actual interface is missing something.

Half the necessary information actually is included on the configuration screen. If you look at the Installed cache stores table, you'll see that it has a Supports column listing the features of each backend. So memcache doesn't list data guarantee while other stores do.

What is missing is that the Known cache definitions table does not have a corresponding Requires column which would then list data guarantee. Seems like it should probably be quite simple to add this - I wonder if there's already a tracker issue? Not volunteering, I'm afraid - I have a lot of work to do lately (I did code a couple of other niceish core Moodle fixes in recent weeks).

With the right information in place it would be much clearer - and the documentation then might also need some improvement, but you wouldn't need to start adding tables in there as well, because it would be covered on-screen.

Given the complexity of the screen, I would suggest that it also might benefit from some brief popup help ? icons on the column headings. For example, this type of issue could be covered on the help for Supports or Requires (well that might be the same help text covering both). There is no popup help at all on that page at present.

--sam

Average of ratings: Useful (2)
In reply to curt bixel

Re: What to cache with MemCached

by Paul Verrall -

Hi Curt,

If you already had an 'in memory cache' active such as APC configured you will perhaps not seem much change in performance. The default cache is to 'file' and both APC and Memcached will blow that out of the water. Basically it sounds like you were already optimised to a degree.

Now there is a *KEY* difference between the way APC and Memcached work that is quite important to understand. Memcached has a concept of 'eviction', if it's running low on space it will remove what it thinks is the least useful object from the cache to make room for the new one. APC does *NOT* do this, instead when it's cache fills it 'purges', deleting all cached objects and starting again. If APC is not configured such as to avoid purges this can lead to intermittent slowdowns as your cache is rebuilt.

Average of ratings: Useful (4)
In reply to Paul Verrall

Re: What to cache with MemCached

by curt bixel -

OK.

So, it sounds like I have a lot of things configured correctly.  I do have a few more questions.


  1. Is it possible to map session cache or request cache to memcached?  They are still mapped to the "default" and I am not really sure where that cache resides.
  2. Is it possible to get moodle to treat more things as static resources?  I have some quizzes that have images in them, and even a few with small videos in them.  (I do try to avoid putting ANY video onto moodle at all, any time it is possible to do so.)
  3. I have heard that you can configure VARNISH to work with moodle.  I have not had any luck, as once I enable varnish, students run into trouble with log in.  They enter their login info, but then are just presented with the log in screen again over and over.
In reply to curt bixel

Re: What to cache with MemCached

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

1. I think Paul's explanation above implies 'no'. Memcached does not work in a suitable manner for this. If I read correctly he is saying that being unable to *guarantee* that memcached will have the record is the issue.

2. Not as such, but if you are using the same resource multiple times then it should be cached by the user's browser. Failing that you might consider some sort of reverse-proxy. 

3. Isn't Varnish a reverse proxy? I'm not familiar with it. If you do a search then there have been some discussions about it. 

In reply to curt bixel

Re: What to cache with MemCached

by Paul Verrall -

I've tinkered with Varnish and Moodle and IMHO there is little point to it. The dynamic nature of moodle content, coupled with the need for permissions checks on almost everything served means that Varnish has very little it would be able to cache.

For example even with sites which work well with Varnish, such as wordpress, you still configure Varnish to not cache in areas where context and permissions are important, like the wp-admin pages. This is the case for almost all content on Moodle.

There are exceptions like content under themes, but you need to balance marginal gains against complexity of the deployment. Remember though, "premature optimization is the root of all evil". That said I have explored this before myself and you can read about that here (beware it is nginx specific).

What you DEFINITELY DO WANT to do is ensure you serving your static content using 'X-Sendfile' (or it's equivalent for you httpd) see Performance_recommendations for more on that.

Specifically regarding sessions the default session cache is defined in your config.php and NOT via the MUC (I know it's confusing having sessions stuff listed there). In the config.php you CAN specify Memcached as your session store and this is not a bad idea...  see config-dist.php there are 'gotchas' to consider though so read the docs.

Average of ratings: Useful (4)
In reply to curt bixel

Re: What to cache with MemCached

by Galin Vassilev -

Hello,

On point 1, yes you can map sessions to memcached. To do that, refere to the "config-dist.php" in your installation. You would find the information you need if you look for this: "Memcached session handler". I would be cautious though. If you reboot the memcached daemon while in production, you will loose all sessions along with all of the information they carry. Memcached does not persist its data to disk.

On point 2, Not easily, beyond what it already does, which is quite a lot. All content created in Moodle gets "Cache-Control" settings which are pretty good and things will get cached at browser level at the client. What I am assuming you are trying to do is accelerate the delivery of those at the server side with Varnish, which while a very good tool, is not something I would recommend for Moodle as I describe below.

On point 3, you are correct that it is possible to use Varnish to work with Moodle, but as mentioned in the forum, it would be in rather limited fashion based on the dynamic nature of Moodle content. In addition, Varnish will not handle SSL content at all, so you have to keep that in mind as well. You can get similar effect at far less "complexity" cost with just using your web server's caching capabilities. For vast majority of cases that would be more than enough, including what I would venture a wild guess is your use case. I have had success, in using Nginx to cache all static content as defined by Moodle itself. Usually this means, style sheets, java scripts images with proper "Cache-Control" headers. It is possible as well to specify exactly which type of content to cache and even set headers if not already specified. Same can be done with Apache as well.

Hope this helps.

Average of ratings: Useful (3)
In reply to curt bixel

Re: What to cache with MemCached

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Just another thing about sessions - people already answered this but to make it explicit, the 'Session cache' store simply uses the PHP session data mechanism. You can configure that PHP session data mechanism via Moodle settings in config.php but this is completely separate from MUC, as per the information others provided.

Our system is configured to store sessions in memcache, which is good with multiple front-end servers. We use a different memcache server for sessions than for MUC which may be a good idea as it means you can (manually) purge/restart those servers, or they can crash or whatever, without losing the other type of data.

In MUC, probably everything that uses session cache should be left there (?) - it's basically stuff that should be cached specific to an individual user for the duration of the session.

Incidentally, leaving aside the sessions and going back to MUC, the MUC memcache configuration that works for us with multiple web servers is to have a local memcache server that runs on each web server. We have then set the system up so that when it stores data to a cache, it stores it to all the different ones, while when it reads it, it reads from localhost. As the caches are read a lot more frequently than written, this can be more efficient - using memcache over a network means you essentially have to wait for a network ping (maybe 0.5ms) for each cache read, whereas localhost is close to zero. (I think basically all the caches that allow it are configured to use memcache in this way.)

--sam


Average of ratings: Useful (4)
In reply to sam marshall

Re: What to cache with MemCached

by curt bixel -

Just a quick thank you to everyone out there who responded to my posts.  My site is running quite well now.  My students have noticed that it is faster and have commented upon that.  They feel that the site is really exceptionally fast, which is what I was shooting for, as they have enough to think about without the frustration of a slow site.  smile

In reply to sam marshall

Re: What to cache with MemCached

by Denis FOULON -

Hi Sam,

could you tell us how to separate read/write on cache in your configuration? Can we do that on moodle 2.6?

In reply to Denis FOULON

Re: What to cache with MemCached

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

In recent Moodle versions (since 2.8) this was built into standard Moodle. You just had to configure the memcache store that way.

In earlier Moodle, there was a separate plugin: memcache_cluster_store.

In reply to sam marshall

Re: What to cache with MemCached

by Nadav Kavalerchik -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Sam,

I am curious... we have a similar cluster setup of 4 front end Moodle app servers that uses clustered memcached and we have a lot of (10K+)  memcached connections on each web server. 

Are you familiar with the phenomena? is it normal/healthy? 

Nadav