Moodle in English: Proposed changes to cache store in Moodle 2.9

Skip to main content

Future major features

Proposed changes to cache store in Moodle 2.9

This discussion has been locked because a year has elapsed since the last post. Please start a new discussion topic.

Hello everyone,

It's my last day at Moodle HQ today and while cleaning things up for my departure I have organised a proposal for changes to the cache stores in Moodle 2.9. This just involved consolidating what was already out there and fixing a few bugs in the process. I'm purposing the introduction of new stores as well as the removal of less used/desirable stores.

I have created MDL-48226 for this, let me summarise what I am purposing:

MDL-37069 Add an XCache store.
MDL-39117 Add an APC store.
MDL-38987 Add a WinCache store.
MDL-48468 Add aRedis store.
MDL-48469 Remove the MongoDB store.
MDL-48505 Remove the Memcache store.

To summarise my view: I really like XCache and APC, nice fast opcode caches, not multi-site safe but in situations where you've only one web server these are ideal. Redis is fast, really fast for an external service and right up there with memcache. Truthfully I haven't even published this plugin in the plugins database yet (I will do of course). MongoDB is slow, really slow, I needed it when writing the Cache API to test functionality and it shows what a NoSQL store would look like. That being said I doubt anyone really uses it and I don't believe it needs to be in core. Memcache, well this is an interesting one, its removal is based really on the fact that it is not multi-application safe, and cannot be made to be so. I have created MDL-48506 to make Memcached multi-application safe as that is possible. WinCache I've there for completeness, I've truthfully not looked at it nor run it in a long time.

So my votes: +1: Add XCache, add APC, remove memcache, remove mongodb +0: Add Redis -1: Add WinCache

That would see 2 added, 2 removed, 1 left to public vote/time, and 1 rejected.

I'd really appreciate any thoughts, feedback, votes if you can muster the effort.

Many thanks Sam

Average of ratings: Useful (7)

Permalink | Reply

Your leaving? I did not know.

Thanks for the help with the themes...and for the new cache store!

You'll be missed...

Cheers

Mary

Byeeeee - All the best...

Average of ratings: -

Permalink | Show parent | Reply

Thanks Mary, but don't worry you won't have seen the last of me, I am pursuing new adventures but they are related to Moodle. I will still be around, just not as much and not as a Moodle HQ employee.

Average of ratings: -

Permalink | Show parent | Reply

Sam,

Thanks for taking the time to work on this. Over the last few years I've used the XCACHE, and memcache stores in both my test and production environments. (My production server is running MS Server 2008r2 and currently hosts a few Moodle instances ) Here are a few things I found:

In a single server environment running on MS windows the Xcache store worked generally worked well depending on the type of data stored in it. If I remember the only issue I had with it was storing course group information, where it the data being cached was not always simple keys (CONTRIB-4318)

When I upgraded PHP to 5.5 I started using optcache in place of XCACHE since the Moodle environment check recommends it. Since optcache cannot work as a cache store, I am using memcache to as the cache store for the bulk of the high traffic data on my site. (This is because I had issues getting memcached to work properly under windows) If redis is a viable alternative to memcache for those of us on a single server install, then I would support moving it into core. I currently have a couple instances of memcache running (one per install) and it seems to be working well.

All I ask is that you don't force those of us on Windows to have to use the file cache for everything since it is rather slow. And before someone comments that we are free to install a different cache store from the plugin directory, While cache stores can be installed, they cannot be removed (MDL-45498)

So my vote would be:

+1 for xcache (assuming CONTRIB-4318 is addressed)
+1 for Redis
-1 for Memcache (It works well in a single server / application environment)

-Steve

Average of ratings: Useful (3)

Permalink | Show parent | Reply

Hi Steve,

For the most part I agree with you, but I don't think it unreasonable to ask people to install a couple of plugins from the plugins database as per their individual requirements. I definitely think that we should remove the memcache plugin - if you look at the technical details behind it, it sucks. Using the memcached plugin instead is a much better solution.

All of these plugins require installation of a php extension, and installation and configuration of server/daemon.

Andrew

Average of ratings: -

Permalink | Show parent | Reply

Hi Andrew,

I also don't think it is unreasonable to ask people to install additional plugins from the database for their specific environment either. My only concern is that certain types of plugins cannot be removed from a Moodle instance once installed (MDL-38259). Cache stores are one of those. When I migrated from xcache to opcache the only methods I could find to remove the cache store was to either reinstall the site from scratch and restore all of the course backups one at a time or to hack the database to try to remove all references to the plugin. (On the three Moodle sites I currently maintain, I ended up rebuilding two of them while the other I attempted to remove all references to that plugin)

If the decision to encourage users to install cache stores from the plugins database, could part of that process also involve improving the machinery around the installation and removal of cache stores so they can be removed like most of the other plugins?

-Steve

Average of ratings: -

Permalink | Show parent | Reply

Hi Sam,

Thanks for all the hard work and I look forward to continuing to work with you in your future endeavours.

I've just peer reviewed your Redis plugin, which I must say looks spot on. I've been looking at Redis for several years now and when I last looked at writing a plugin the php support was a little too immature so I'm really keen to see this now.

As I mentioned in my review, I'd give this a -1 for inclusion in core, and I do so for the following primary reason:

Redis requires both a server and a php extension which are non-standard in a typical installation. I don't think it's unreasonable to ask that an administrator installs the Redis plugin from the plugins database too.

I also agree that we should remove the memcache plugin for the reasons that you list (also because it just completely sucks because of the memcache extension itself), and we should remove the mongodb plugin. If nothing else, having some of these plugins in core legitimises their use and, as we've found, they are not really ideal for use in core for several of the reasons you mentioned.

With regards APC and XCache, I'm less convinced. OPCache is now standard in PHP 5.5, and I think we should be looking at prioritising caches which are standard and do not require a large amount of installation, and configuration. That said, they are useful caches, for older installations which you find with a lot of shared hosting providers. Similarly, Wincache is useful for those windows installations which people insist on running. Providing Wincache in core again legitimises use of Windows as a valid server environment. For anyone interested in running on a windows server, I don't think it's unreasonable to ask them to install the Wincache plugin.

So in summary, my votes:

APC addition: +- 0
XCache addition: +-0
Wincache addition: -1
Redis addition: -1
memcache removal +1
mongodb removal +1

Andrew

Average of ratings: -

Permalink | Show parent | Reply

Andrew,

One thought about APC, XCache and WinCache. While I understand your concern about supporting competing caches, all of these are actively maintained and provide functionality not available with PHP's built in OPCache, specifically the ability to cache user data and be used as a cache store. When OPCache became part of PHP, it looks like the user cache portion of APC was broken out into it's own PHP extension called APCu. From what I've read, it looks like APCu can be installed in parallel with PHP's OPCache and is fully backward compatible withe the earlier APC functions. This would mean that if the APC store were to be added to Moodle core, it would support both the APC and APCu PECL extensions, which I feel would be a useful to smaller single server sites. Since Moodle core ships with a store for large sites (memcached) it would be nice to support a cache for small sites (APCu)

There is some useful information about APCu here:

-Steve

Average of ratings: -

Permalink | Show parent | Reply

My thoughts are:

We should have a small set of backend choices with some helpful advice for what suits each Moodle deployment scenario. By sticking to a small set in core, we support each well.
It looks like an opcode cache solution is a gap in our small set and it would be good to have one in core for single-server sites.

So:

1. I would like see only one 'winner' chosen between xcache/apc etc to go into core. Lets have one and focus our 'core' energy on that working well.

2. It sounds sensible to remove mongo, I don't expect it to affect many of our users (based on issues reported).

3. I suspect memcache removal might affect more users (because we gave too much choice in the first place), but memcached is the alternative and it is justifiable to me to remove the confusing choice between the two.

4. Outside of core, the community can develop any promising alternatives, but lets make some simple choices. A problem we have with MUC is making it too flexible and so hard for people to choose their configuration.

Average of ratings: -

Permalink | Show parent | Reply

We (the OU) currently use memcache rather than memcached. This is probably something to do with which extension is supported in our server platform - which is currently rather old. I think we would like to investigate a move to redis and/or opcache when we upgrade our server operating system.

One thing I'd like to clear up - am I correct in thinking that removing something from core means that HQ are dropping all support for it and will not maintain it in future? In other words if memcache is removed from core that doesn't just mean we have to install a plugin, which in itself is fine - that also means it will probably stop working at some point and we will have to fix it ourselves (unless somebody else volunteers as maintainer). Or is it going to remain supported by HQ for a certain number of versions, like when code is deprecated?

And while messing about with the list of included caches - I don't think memcache(d)_clustered was integrated in core yet. I said we use memcache but I think we actually use memcache_clustered... Would this be a good time to incoporate that plugin as part of the core memcached handler, perhaps? It's basically just adding some extra optional configuration settings. It makes a large difference to performance on multi-server systems because it lets you use local memcache servers; having to go across the network for every MUC read is a substantial performance penalty.

--sam

Average of ratings: Useful (3)

Permalink | Show parent | Reply

Good luck in your new position Sam.

Average of ratings: -

Permalink | Show parent | Reply

Why is "everybody" leaving Moodle HQ?

First Petr Skoda. Now Sam... Who next? (Is Totara pinching all the Moodle HQ developers?)

Average of ratings: -

Permalink | Show parent | Reply

This is probably insulting to the 30 or so of us working at HQ!

Sam and Petr had personal reasons for wanting to work in Wellington. And anyway, we still see them all the time working on Moodle in the tracker, so all good.

Average of ratings: Useful (3)

Permalink | Show parent | Reply

Two people is hardly everyone, Moodle HQ has a great team and that certainly didn't change with our departures.
My new role with Totara came about after the decision my wife and myself made to move to Wellington, it puts me back into an office and I still get to work with Moodle HQ and contribute back to the project.
While my reasons for the change are my own I can assure everyone that there was no pinching going on at all, and that I still love Moodle and the crowd at Moodle HQ.

Average of ratings: Useful (1)

Permalink | Show parent | Reply

Pedro

I can assure you we didn't and wouldn't "pinch" or poach. That's not what happened. Also as Sam points out, their joining the Totara team helps us in our ongoing commitment to contributing to the Moodle project.

Cheers

Richard

Average of ratings: -

Permalink | Show parent | Reply

-1 the memcache store removal.

Users that have RHEL for their servers will have some difficulty with getting the memcached php extension currently as I have not seen a new release software collection with it.

I am nearly done building a LAMP cluster that is using memcache and it works just fine. Memcache was not my first choice but using a third party repo to get php memcached installed was out of the question.

It would be make upgrading to moodle 2.9 VERY difficult to do if memcache is no longer supported (depending how memcache acts being a add on instead of part of core moodle code).

PS: How does this change impact memcache if it is a add on plugin when it comes to session handling?

Average of ratings: Useful (1)

Permalink | Show parent | Reply