Future major features

 
 
Picture of David Mudrák
Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators

Hello everybody

In this thread, I would like to discuss a new feature for Moodle - so called "Automatic updates deployment". This is a follow-up project on the recently introduced "Available updates notification" feature. In short, Moodle administrators should be able to download and deploy (install) updates purely via the web interface. We believe that this feature would encourage more admins to keep their sites up-to-date which is considered as the key protection against security attacks.

In the first phase, we plan to focus on updating and installing contributed extensions from the Plugins directory. Later, we plan to add a support for updating (and even installing!) Moodle core itself via this new tool.

The use cases and a rough implementation plan is described at Automatic updates deployment project page. I admit it's a bit late for significant changes in the spec for the first version of the tool (planned for 2.4 release). However, any idea, comment and feedback will be still useful as it can be considered in later phases of the development.

 
Average of ratings:Useful (7)
Picture of Tomasz Muras
Re: Automatic updates deployment
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

From the wiki:

A standalone tool (script) that is able to fetch a ZIP package from a given URL, compare its MD5 content hash with a given value, unpack it, and deploy the content to a given location inside wwwroot.

Isn't there something like this for the lang downloads? It seems like that would be a similar funcitonality - maybe we can re-use other code or handle those two together?

I agree that the downloader/installer code needs to be separate code as handling includes will be very problematic.

We should probably learn a bit from other projects - have you seen drush?

cheers,
Tomek

 
Average of ratings: -
Picture of David Mudrák
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators
Yes, the language installer does a very similar job. Just note that the language installer has a lot of underlying Moodle environment ready to be used - starting with parameters handling from displaying the output. In this case, we must not include anything from Moodle. Thence direct re-using of any code is a bit tricky...
 
Average of ratings: -
Just wondering . . .
Re: Automatic updates deployment
Group Particularly helpful Moodlers

I'm interested how hosted, very secure installations will cope with this.

We currently have to go through code check etc before thuings are installed, often quite detailed.  Can this be made secure enough for the providers to be happy with?

I think it is a great idea.  Regularly it is a small fix that needs to be sorted.  I'll head off to the docs to see if this new feature involves working for themes and course layouts as well.

Cheers

-Derek

 
Average of ratings: -
Picture of David Mudrák
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators
I am not sure I understand what you mean by "be made secure enough". Strict environments will actually want to have this feature locked to prevent accidental installation of a plugin that has not been reviewed by their own.
 
Average of ratings: -
Just wondering . . .
Re: Automatic updates deployment
Group Particularly helpful Moodlers

OK, that's the answer to my question: it's not secure enough for strict environments.  So it's a speed thing for those who host things themselves, and a speed thing for hosted environments to do it for us.  But no advantage to give more power to users in a hosted enviroment.  OK, I can live with that.

-Derek

 
Average of ratings:Useful (1)
Picture of Hubert Chathi
Re: Automatic updates deployment
 

In your plan, you indicate that the downloaded zip file's MD5 hash is compared with an "expected value".

  • You don't indicate how the expected MD5 is obtained.  In order to have any security value, the expected MD5 must be obtained via a separate channel from the zip file -- otherwise, anyone who can tamper with the zip file can tamper with the MD5.  I would suggest using a public key digital signature scheme, instead of just a simple hash.  Alternatively, it can ensure that the zip file is downloaded via HTTPS (if you trust the SSL certificate authorities).
  • MD5 is no longer considered secure.  Is there any reason for not using SHA-512?
 
Average of ratings: -
Picture of David Mudrák
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators
Good point Hubert. Yes, we are aware of this. We are planning to use HTTPS for obtaining the MD5 hash and maybe even for downloading the ZIP. I am also aware of MD5's weakness from the cryptography point of view. But in this case, it actually serves as a control checksum, not a cypher. Of course, hackers might be able to reconstruct the original contents of the ZIP file knowing the MD5 hash - but I don't think they would waste their time on it really smile And I am not aware of any simple way how to produce infected ZIP with malicious contents having the same MD5 hash and reasonable size. So, to paraphrase yourself - is there a reason for using SHA-512?
 
Average of ratings: -
Dan at desk in Moodle HQ, Perth
Re: Automatic updates deployment
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers

DISCLAIMER: know nothing about cryptography, I just read Bruce Schneier. (In fact, is this even under the term 'cryptography'?)

So, to paraphrase yourself - is there a reason for using SHA-512?

Well, MD5 has had collision exploits, see: Microsoft Technet: Flame malware collision attack explained. So its not beyond the realms of possibility that such a zip could be created, (though I agree, this wouldn't keep me up at night).

These cryptography algorithms seemingly an arms race, SHA-512 is the best runner at the moment and widely available, so, why not?

(Lets not mention using two different algorithms to generate the signature to help protect against weakness in one).

 
Average of ratings: -
Picture of Hubert Chathi
Re: Automatic updates deployment
 

Why use SHA-512?  Because it is the currently generally-believed-to-be most secure hash algorithm.

IMHO, when designing a new protocol/system/etc., one of the goals of the design should be to build for the future as much as possible and as much as practical.  That means, for example, making things extensible, and avoiding having to change things in the future.  Even though there may not be any currently known attack for generating a malicious Zip file, the known weaknesses in MD5 indicate that it can't be trusted for much longer.  In fact, US-CERT says that you "should avoid using the MD5 algorithm in any capacity." (emphasis mine)

Also, regarding building for the future, even though you say that its intended  purpose is as a control checksum rather than a cipher, unless you are absolutely certain that nobody will think that it's a good idea to also have a cipher, if you're choosing to use a cryptographic hash function anyways, why not use a stronger one that can be used for both purposes?  You say that you don't think that hackers would waste their time trying to build a malicious Zip file, but with Moodle being one of the most popular VLEs, and with more companies and institutions investing in VLEs, and with things such as people using the same password for multiple sites, Moodle could very well be a tempting target.

IMHO, when someone says "hash", one's first thought, barring any other constraints or considerations, should be the current best hash algorithm, which is currently generally agreed to be SHA-512.  Some possible constraints or considerations are:

  • hash size: in some cases (e.g., if many hashes are used and/or if the data is small), a large hash size may cause too great of an overhead.  But this isn't the case in this situation, since the hash size is many, many times smaller than the ZIP file.
  • speed: one hash algorithm may be faster than another, and if many hashes are calculated, this may be an issue.  But in this case, a hash is only calculated occasionally, so this is not an issue.
  • availability of implementation: in some cases, the current-best hash algorithm might not be readily available in the target system.  But in this case, PHP 5.3 already includes SHA-512 support, so this is not an issue.

Given that no other issues are, IMHO, relevant, the remaining distinguishing feature for hashes is the security, and in this regard, SHA-512 is the clear winner.  Even if you think that SHA-512 is overkill, at least use SHA-256, or even SHA-1.

Another thing that I would like to point out is that by using MD5, you may be encouraging others, who might not know better, to do the same.  "If it's good enough for Moodle, it's good enough for me."  We should be setting an example of migrating away from MD5 wherever possible, rather than encouraging its use.

As I see it, there is simply no reason to use MD5 here, especially since it is just a one-line change.  (Well, OK, two lines -- one line on the server, and one line on the client.)

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Good point about future-proofing.

So, I think what we are saying is that the hash-checking system should be able to cope with any type of hash that PHP supports. In other words, when you download https://download.moodle.org/qtype_stack.2012092600.hash the the file contents will include both the name of the hash algorithm, and the value to check against.

Looking at http://php.net/manual/en/function.crypt.php is seems this could be achieved if the contents of the .hash file looked like

$6$rounds=5000$somehashstringXX$D4IrlXatmP7rx3P3InaxBeoomnAihCKRVQP22JZ6EY47Wc6BkroIuUUBOov1i.S5KPgErtP/EN5mcO.ChWQW21

which is the kind of thing PHP's crypt function needs to tell it to use SHA512. That gets us future-proof.

This is not really overengineering. Compared to the time to download and install a plugin, the time to compute the has of the zip using any algorithm is trivial.

 
Average of ratings: -
Picture of Hubert Chathi
Re: Automatic updates deployment
 

Yes, specifying the hash type is a good way to make things more flexible.  I think it would also be good to allow for multiple hash types (like what Debian did) to be downloaded, which would make it easier to transition from one hash type to another, if the need should ever arise -- if a weakness is found in hash algorithm foo, then you can provide both hash values: the new one for security, and the old one for code that is not yet able to support the new one.

I don't know if we'd want to use the crypt format, because I don't know if we want to be stuck with the predefined numbers that it uses.  We probably don't want to be using the crypt function for hashing, anyways, since that would require us to read the whole file into memory.  It would be better to use the hash family of functions, like hash_file http://php.net/manual/en/function.hash-file.php

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Oh yes, that is a much better function.

So, a sensible format for the .hash file would be something like

sha512|abc123...
md5|abc123...
...

where the algorithm name is anything that is a valid first arugment to hash_file. I am just noting the hash_algos method, to get a list of supported hash algorithms, since I had to look that up.

The implemenations could be to tests the first, or all, hashes from the .hash file where this PHP version supports that algorithm, with an error if none of the hashes use a supported algorithm.

 
Average of ratings: -
Picture of Hubert Chathi
Re: Automatic updates deployment
 

Yes, something like that would probably work well enough.  The implementation could make up its own mind about what hash algorithms to trust/use, possibly even allowing the administrator to configure this.

 
Average of ratings: -
Picture of Danny Wahl
Re: Automatic updates deployment
Group DevelopersGroup Particularly helpful Moodlers

I think some settings and checks for how it's handled would be very important and in no particular order:

- never notify me about updates / notify me about updates / download updates and notify me to install / download and install updates automatically (notify on failure/success)

- ignore this update (once, always) / view ignored updates

- check compatibility with moodle before updating plugins

- check compatibility with plugins before updating moodle

- try to activate plugins that aren't marked as compatible after core update (maybe it'll still work?)

- backup database before performing update

- set site in maintenance more while updating w/ custom message

- schedule auto-update time via cron

-----

on a deeper level: integrate a way to mark a plugin as working/not working with a specific version of moodle if it's not explicitly marked as compatible (similar to wordpress)

 
Average of ratings:Useful (1)
Picture of Ralf Hilgenstock
Re: Automatic updates deployment
Group DevelopersGroup Translators

Hello David

some suggestions and feature requests:

  • option to hide/disable the feature completely by CFG seeting in config.php
  • option to hide/disable the feature for installing new plugins

As Moodle Parnter we host hundreds of systems and keep the system up to date by our own services. So the support all the time knows which packages are installed for a client.  There are lots of systems with same code basis but seperate databases. If one of the admins updates the system its updated for all other systems.
In some environments several packages are not allowed, mainly due to data protection regulations. If admins who are not aware of the information and install additional packages it could be critical.

  • Reset the system to former version.

During last version updates lots of APIs have changed. Connected systems often got problems. If we simplify the update process lots of system connections  to external systems will crash. So it should be possible to set the system after a quick test back to the former version.

I know there are a lot of admins who like such a feature in Moodle. I also saw a lot of wordpress users who crashed their systems by adding plugins.  We should implement it carefull and with some barriers that it is not so simple and could be disabled. Often teachers are admins and most of the teachers are collectors of features and gamers.  They really like to try new features but often they are not aware about the problems if plugins are not updated and supported continously. I think its different to host a personal blog or to admin a school/corporate leraning platform.

 
Average of ratings:Useful (2)
Picture of Joseph Rézeau
Re: Automatic updates deployment
Group DevelopersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Is this "Automatic updates deployment" feature actually meant to be working in Moodle 2.4? I am currently conducting tests and have come upon a problem with updating the hotpotatoes mod, described in this discussion.

Using Moodle 2.4beta+ (Build: 20121123)

Joseph

 
Average of ratings: -
Picture of David Mudrák
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators
Yes, it should work for most sites. See my post in the other thread you mentioned for details.
 
Average of ratings: -
Mary Cooch
Re: Automatic updates deployment
Group Documentation writersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup TestersGroup Translators

I'm lost - where is the post you say gives more detail in the other thread that Joseph mentioned? When I click on Joseph's link above I get "coding error " in red etc. 

EDIT: Response is here https://moodle.org/mod/forum/discuss.php?d=216584&parent=943037#p943408 but still don't understand the coding error -errro.

 
Average of ratings: -
Picture of Joseph Rézeau
Re: Automatic updates deployment
Group DevelopersGroup Particularly helpful MoodlersGroup TestersGroup Translators

@Mary,

Mea culpa. The link to that discussion I started in the HotPot module forum should have been: see this discussion.

Joseph

 
Average of ratings: -
Just wondering . . .
Re: Automatic updates deployment
Group Particularly helpful Moodlers

Anyone:

Just tryng to get this sorted in one of the sites I work with, but getting errors, and will switch deugging on in the night when I think not too many people will be around.

Just curious: is this working as planned?  Are there people out there using this?

-Derek

 
Average of ratings: -
Picture of David Mudrák
Re: Automatic updates deployment
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful MoodlersGroup Translators
Hi Derek. Yes, automated updates deployment is supposed to work and there are people (like me) using it at production sites. Turning debugging on is definitely a good start to sort out your issues.
 
Average of ratings: -