We started out with Moodle 1.5.2, which is a good, solid product and perfect for our needs. Of course, nothing is perfect out of the box and I ended up doing a lot of heavy customization to sort of pound Moodle as a round peg into the square hole of our business requirements.
Unfortunately, though, this has come back to bite me in the rear. Hearing Martin speak at the Moodle Moot has convinced me that we should move to Moodle 2 when that becomes available, but before that we need to move to Moodle 1.9 (we're currently customizing Moodle 1.8 which I may give up on as a lost cause). The customizations I've made have gotten in the way of doing this easily.
So what are the best ways that you've found to get around this situation? Some approaches I'm taking this time around to make the process smoother next time are:
1. Using includes wherever possible instead of large chunks of customized code;
2. Heavy documentation, and commenting out lines of code I'm replacing instead of deleting them outright; and
3. Instead of making changes to pre-existing tables, create additional tables linked via foreign keys to the existing ones. For example, instead of adding the two additional fields to the user table that we need, create a separate table that contains our specific data and simply link that back to mdl_user by the id key.
I can't be the only one in this situation. What are the best ways other people have found for doing this that can minimize the pain in migrating to future versions of Moodle while maintaining customizations?
What kind of changes do you have? I started with customizations to 1.3/1.4, and then in the later versions I've tried to "phase out" our core customizations to use the ways that Moodle offers for modifying and extending the behavior. For example, since 1.8 there are core tables for adding custom fields for the user data (http://docs.moodle.org/en/Development:Customisable_user_profiles). Other ways of extending Moodle include:
What are your biggest problems when upgrading? Custom code is lost? Code conflicts with version control? Broken function calls? All of the above?
- Themes
- Language packs
- Blocks
- Course formats
- Resource types
- Activity modules
- Question types
- Filters
- Enrolment plugins
What are your biggest problems when upgrading? Custom code is lost? Code conflicts with version control? Broken function calls? All of the above?
Just to reiterate...
My "policy" has always been to use plugins wherever possible and/or to try and code improvements that will be universally useful - your patch gets into the next release!
My "policy" has always been to use plugins wherever possible and/or to try and code improvements that will be universally useful - your patch gets into the next release!
At the OU we unfortunately have extensive changes to core code.
Unlike Howard, I've found that even 'positive' code changes can be hard to get agreed for inclusion in Moodle core, although we certainly have managed to get in many such changes (most of which were a good idea
). We do have direct commit access but I generally don't use this without asking unless it's for a straightforward bugfix.
Sometimes the difficulty in getting it included is because what we did was good for us but doesn't really fit the way Moodle works, in which case it's obviously right for moodle.com to reject it; with minor improvements there is sometimes an attitude that 'well this needs to be completely rewritten for [some future version] so don't change anything now'; but most often it's just hard to get agreement because Moodle.com developers are busy, and of course we are too.
For example I have a big improvement to the language system that allows plurals (and other things) to work properly, and I've asked a couple of times but not really got approval - I'm going to need to create more fuss to get that into core so maybe it will never go in (even though I spent two days coding it, along with significant time discussing it in forums). This isn't even in our local version either - I don't want to put it in there precisely because of the difficulties it will cause in merging future updates, as in this thread - so it might have been a complete waste of my time. MDL-12433 if anyone's interested
Anyhow, sob story over - in addition to these problems, we have many code changes which are not actually a good idea, just a quick hack to solve a local problem. So I certainly wouldn't want to get those changes into official Moodle.
I agree that wherever possible all changes should be done within the plugin framework (modules, blocks, course formats). This will minimise problems. Other code that doesn't fit into this should go in the 'local' folder where possible (which is where local database changes should live). Language changes should all go in a _local folder or in dataroot, with absolutely no changes to en_utf8. For our own custom plugins, we always keep language files with the plugin (inside the plugin folder) rather than in separate lang folders as this makes them easier to manage. Of course we have our own theme (child of 'standard') to do all CSS changes so we generally don't touch standard theme.
When we do need to change core code, we do so as follows:
// ou-specific begins
...
// ou-specific ends
If we were replacing some existing code rather than just adding it, we often (but not always, hm) leave in the original code commented-out.
We also use
// ou-specific begins (until 2.0)
...
// ou-specific ends
if we have code which IS going into moodle for the next version, so we know that the ou-specific section should be overwritten then (but not for a 1.9.x point release).
Or
// ou-specific begins (until MDL-12345)
...
// ou-specific ends
That's if we have a patch in the mentioned bug number which does what we want, but we're waiting to see whether anyone will agree we can include it in moodle.
We also use the same words but with the relevant comment indicators if we need to make changes in other types of file that don't support // comments.
Merging a new version is done by hand. For folders and files that don't contain 'ou-specific' we just overwrite the whole thing with the new version from core. When a file does contain that string, the person who merges code sorts it out manually, making changes to our customisation if necessary to keep it working, and possibly discussing it with whoever wrote the change.
--sam
Unlike Howard, I've found that even 'positive' code changes can be hard to get agreed for inclusion in Moodle core, although we certainly have managed to get in many such changes (most of which were a good idea
Sometimes the difficulty in getting it included is because what we did was good for us but doesn't really fit the way Moodle works, in which case it's obviously right for moodle.com to reject it; with minor improvements there is sometimes an attitude that 'well this needs to be completely rewritten for [some future version] so don't change anything now'; but most often it's just hard to get agreement because Moodle.com developers are busy, and of course we are too.
For example I have a big improvement to the language system that allows plurals (and other things) to work properly, and I've asked a couple of times but not really got approval - I'm going to need to create more fuss to get that into core so maybe it will never go in (even though I spent two days coding it, along with significant time discussing it in forums). This isn't even in our local version either - I don't want to put it in there precisely because of the difficulties it will cause in merging future updates, as in this thread - so it might have been a complete waste of my time. MDL-12433 if anyone's interested
Anyhow, sob story over - in addition to these problems, we have many code changes which are not actually a good idea, just a quick hack to solve a local problem. So I certainly wouldn't want to get those changes into official Moodle.
I agree that wherever possible all changes should be done within the plugin framework (modules, blocks, course formats). This will minimise problems. Other code that doesn't fit into this should go in the 'local' folder where possible (which is where local database changes should live). Language changes should all go in a _local folder or in dataroot, with absolutely no changes to en_utf8. For our own custom plugins, we always keep language files with the plugin (inside the plugin folder) rather than in separate lang folders as this makes them easier to manage. Of course we have our own theme (child of 'standard') to do all CSS changes so we generally don't touch standard theme.
When we do need to change core code, we do so as follows:
// ou-specific begins
...
// ou-specific ends
If we were replacing some existing code rather than just adding it, we often (but not always, hm) leave in the original code commented-out.
We also use
// ou-specific begins (until 2.0)
...
// ou-specific ends
if we have code which IS going into moodle for the next version, so we know that the ou-specific section should be overwritten then (but not for a 1.9.x point release).
Or
// ou-specific begins (until MDL-12345)
...
// ou-specific ends
That's if we have a patch in the mentioned bug number which does what we want, but we're waiting to see whether anyone will agree we can include it in moodle.
We also use the same words but with the relevant comment indicators if we need to make changes in other types of file that don't support // comments.
Merging a new version is done by hand. For folders and files that don't contain 'ou-specific' we just overwrite the whole thing with the new version from core. When a file does contain that string, the person who merges code sorts it out manually, making changes to our customisation if necessary to keep it working, and possibly discussing it with whoever wrote the change.
--sam
I didn't say it was *easy* to get things into core. In fact it gets harder all the time - rightly probably 
I think it's a reasonable aspiration though. The flow of thought often goes...
1. I've had a FANTASTIC idea for a new feature
2. Post proposal
3. Response = that's a really stupid idea, have you thought about...
4. Good point, go to plan B
OR
4. I really have to have this. So reformulate plan OR as a last resort do a local hack.
Even then my view is that if there is a way to do it as a plugin use that route. For example, don't modify an existing question type, take a copy and make a new type. Less pain in the end (sometimes!).
I think it's a reasonable aspiration though. The flow of thought often goes...
1. I've had a FANTASTIC idea for a new feature
2. Post proposal
3. Response = that's a really stupid idea, have you thought about...
4. Good point, go to plan B
OR
4. I really have to have this. So reformulate plan OR as a last resort do a local hack.
Even then my view is that if there is a way to do it as a plugin use that route. For example, don't modify an existing question type, take a copy and make a new type. Less pain in the end (sometimes!).
Sam - I think your description of documenting the code is good as I used a similar process. In addition, I opted to create diff files which I store in a directory tree that mirrors the moodle code so that I can easily recreate the file should I need to. Peace - Anthony
IME, if you are using a decent SCM, it is a bad idea to add "this code is mine" comments.
There are reasons for this:
A good SCM will tell you immediately, effortlessly what files are touched ("dirty") on your branch, even if you've merged repeatedly. If you keep "this code is mine" comments around, you must remove them when you put them upstream -- even if the code does not change. By removing them, you are merging a different patch upstream, and you will get a conflict next time you merge. A trivial conflict which will consume time to resolve where it could have been auto-resolved. If in the same area of code you have other changes that you aren't putting upstream, the conflict becomes more complex to resolve.
On the other hand, if you have the patches "clean", then you can merge them upstream unchanged, and that good SCM of yours will realise, and automerge, not giving you any grief at all. Go home early, relax.
Life is too short - WAY too short - to be spent merging with bad tools. I used CVS for keeping local branches way too long because there weren't any viable alternatives. I am sure I lost a few years of life expectancy because of those horrid merges :-[
Yes, but Martin, what your message doesn't say is that this is just a big plug for Git (the 'good SCM system' you mention), or one or two other obscure systems that also provide these facilities.
Like most people we use CVS (and I don't think SVN is much better in this respect) and have no very clever way of merging core/local changes, which means these marker comments are invaluable.
We do not generally have problems with the situation you described (same patch goes to core) because after checking in a patch to core, we remove any such ou-specific tags from our own code [we know that if we later merge with 1.9.2, it's ok to replace with the core version]. Where we do have a problem is with having to manually merge with core in the first place! I agree it's a total waste of time and we would do better to use a 'good' version control system - but Git's interfaces with real-world use are still not finished (finished support for neither Eclipse nor standard Windows GUI; unfinished projects for some of these appear possibly stalled; not sure if there is any support at all for other IDEs such as NetBeans, or for OS X GUI for instance).
Basically you are right that using Git (or similar, but probably Git) is obviously the future, and that cvs/svn cannot really cope with two moving targets (local moodle changes + core moodle changes). But I also think that anyone who isn't extremely knowledgeable about version control is going to have a lot of trouble getting a Git system set up the way they want, and some inconvenience (lack ot tool/GUI support) once they've set it up.
I do hope that the OU will move to Git for our Moodle development at some time in the relatively near future but I also hope we get a fully-working (or at least nearly-fully-working) Eclipse plugin for it first! Until then let's continue giving horrible merge jobs two or three times a year to somebody who isn't me.
(I doubt we have spare money to fund that development unfortunately... seems like a pretty desperate need.)
--sam
Like most people we use CVS (and I don't think SVN is much better in this respect) and have no very clever way of merging core/local changes, which means these marker comments are invaluable.
We do not generally have problems with the situation you described (same patch goes to core) because after checking in a patch to core, we remove any such ou-specific tags from our own code [we know that if we later merge with 1.9.2, it's ok to replace with the core version]. Where we do have a problem is with having to manually merge with core in the first place! I agree it's a total waste of time and we would do better to use a 'good' version control system - but Git's interfaces with real-world use are still not finished (finished support for neither Eclipse nor standard Windows GUI; unfinished projects for some of these appear possibly stalled; not sure if there is any support at all for other IDEs such as NetBeans, or for OS X GUI for instance).
Basically you are right that using Git (or similar, but probably Git) is obviously the future, and that cvs/svn cannot really cope with two moving targets (local moodle changes + core moodle changes). But I also think that anyone who isn't extremely knowledgeable about version control is going to have a lot of trouble getting a Git system set up the way they want, and some inconvenience (lack ot tool/GUI support) once they've set it up.
I do hope that the OU will move to Git for our Moodle development at some time in the relatively near future but I also hope we get a fully-working (or at least nearly-fully-working) Eclipse plugin for it first! Until then let's continue giving horrible merge jobs two or three times a year to somebody who isn't me.
(I doubt we have spare money to fund that development unfortunately... seems like a pretty desperate need.)
--sam
To be honest, aside from downloading Moodle via CVS and using an SVN repository to track my writing at home, I'm not all that familiar with SCM in general, and with Git in particular, even though I have it installed on my computer. In particular, I'm not at all certain how I'd go about using the two tools to keep my Moodle installation up to date as well as keeping my own changes in place.
My preferred IDE is Quanta. Eclipse, since it's written in Java, is way too slow on my clunky old work machine to make it particularly useful.
My preferred IDE is Quanta. Eclipse, since it's written in Java, is way too slow on my clunky old work machine to make it particularly useful.
Quanta doesn't appear to have git support either as far as I could tell - just CVS and (with plugin) SVN.
By the way, Eclipse performance and memory usage is pretty good - er, relatively - if you are just using it as a Java IDE. Unfortunately the PHP extension seems to be pretty poor; it does swallow a completely infeasible quantity of RAM (and lots of cpu too). For Java development eclipse used to be, if not ideal, then at least perfectly usable on my old AMD Duron 1.6 with 256MB RAM...
--sam
By the way, Eclipse performance and memory usage is pretty good - er, relatively - if you are just using it as a Java IDE. Unfortunately the PHP extension seems to be pretty poor; it does swallow a completely infeasible quantity of RAM (and lots of cpu too). For Java development eclipse used to be, if not ideal, then at least perfectly usable on my old AMD Duron 1.6 with 256MB RAM...
--sam
External to the actual modification of the code a good source code versioning system can go a long way to assisting in migration tasks.
Having gone through a 1.6 to 1.8 upgrade at HSU and now 1.8 to 1.9 upgrades for clients at Moodlerooms the svn repository allows for diffing of files, merging of code and more importantly reversion of code if a mistake is made.
Having gone through a 1.6 to 1.8 upgrade at HSU and now 1.8 to 1.9 upgrades for clients at Moodlerooms the svn repository allows for diffing of files, merging of code and more importantly reversion of code if a mistake is made.
I am also asking me sometimes why Moodle yet did not migrate from CVS to the much better SVN versioning system.
I think I'm in agreement with you there. I switched over from CVS to SVN a few years back for my own projects, and I've never looked back.
Because SVN isn't a particularly interesting SCM, specially when you need local and feature branches. Search this forum for GIT and DSCM for more in-depth discussion of the reasons why.
I'm new to both Moodle and CVS/versioning systems. We have downloaded the 1.9 Moodle using the CVS commands and set it up in a clean install. We've made a few changes to some themes that we copied over from our 1.8 install. I will soon want to bring those changes, plus all of the 1.8->1.9 changes in to upgrade our current 1.8 install which was setup without using CVS.
What is the best way to do this? Since 1.8 was setup with out using CVS, can we download from the central CVS and merge it's changes with our current 1.8?
We plan on making future changes, and may even submit some things to contrib. We want to implement some kind of versioning. If we keep track of our own changes using CVS, will we need two CVS folders in each install? How do you track your own changes and still be able to merge down new versions from Moodle?
Finally, is CVS the best answer for our local versioning? we run 4 different moodle instances, and so I need to track changes in each of them.
Thanks for any help you can give a newbie.
Nate Baxley
What is the best way to do this? Since 1.8 was setup with out using CVS, can we download from the central CVS and merge it's changes with our current 1.8?
We plan on making future changes, and may even submit some things to contrib. We want to implement some kind of versioning. If we keep track of our own changes using CVS, will we need two CVS folders in each install? How do you track your own changes and still be able to merge down new versions from Moodle?
Finally, is CVS the best answer for our local versioning? we run 4 different moodle instances, and so I need to track changes in each of them.
Thanks for any help you can give a newbie.
Nate Baxley
I had a similar problem to yours a couple of weeks ago, namely upgrading from a rather heavily modified 1.8.2 to 1.9.
In anticipation of this, all modifications to the original 1.8.2 had been marked by identifyable comments, so I was able to port my modifications manually to the 1.9-code, by searching for the comments.
I do not delete code, but rather use comments to deactivate it. Thus, if necessary, everything can be returned to its original state without too much of a hassle even without source control.
To stay up to date, I CVS update the development system as well as the production system once a week to MOODLE_WEEKLY.
The development system source code, however, is version controlled by Subversion. This approach has several advantages:
Regards
Ralf Seliger
In anticipation of this, all modifications to the original 1.8.2 had been marked by identifyable comments, so I was able to port my modifications manually to the 1.9-code, by searching for the comments.
I do not delete code, but rather use comments to deactivate it. Thus, if necessary, everything can be returned to its original state without too much of a hassle even without source control.
To stay up to date, I CVS update the development system as well as the production system once a week to MOODLE_WEEKLY.
The development system source code, however, is version controlled by Subversion. This approach has several advantages:
- The CVS update is not hampered by the .SVN-Folders
- Subversion can be told to ignore the CVS-folders when I commit my changes to my Subversion repository.
- There is no need to distinguish between different CVS-folders
Regards
Ralf Seliger
Have you changed other things than the themes too? If not, you are probably just fine without CVS. If you want to change a standard theme, make your own theme to be based on that standard theme (see http://docs.moodle.org/en/Themes for more). Then you can just update Moodle, and your theme will stay as it is (in a folder that is not touched by the Moodle upgrade). As it is based on standard themes, those will be updated. You might have to fix some of the theme settings after the upgrade, if the HTML has changed etc.
Well, we plan to make changes to other areas eventually. My concern is that I need to track changes to our local version, some of which don't make sense to include in the official release. Do I need a second, local CVS?
Yes you should track local changes, either using a local version control system to store the whole thing or possibly by maintaining a set of patch files in such a way that they can be automatically applied. (So whenever you update Moodle, you can just run some script to automatically apply all your patches again, and find out if any of them fail so that you need to alter them. Obviously you would do this on a development server not directly on the live system, then transfer it to the live system once you've fixed all your patches.)
This is the sort of thing git could be useful for automating so if you really feel like getting a headache, now's the time to start learning! I'll let the git advocates tell you how though, since I don't know...
--sam
This is the sort of thing git could be useful for automating so if you really feel like getting a headache, now's the time to start learning! I'll let the git advocates tell you how though, since I don't know...
--sam
Thanks to everyone who's replied! I've learned through this forum and through conversations at the MoodleMoot in SF that there are much better ways of doing things than I had been doing. I wasn't looking closely enough at the plugins, blocks, and extensions as I should have been (Mike Churchward's talk made me realize the importance of this), and I hadn't even considered the value of using CVS to keep my code in check.
We do have some ways of doing things that I doubt would make it in to Moodle core. For example we want to make sure our students can't get into a course at the end of a certain period of time, but unenrolling them is not sufficient, since we still need their scores and work retained in the database. Plus, we need to prevent students from seeing anything past the first lesson until they click through on an agreement page (though I suspect that can be taken care of in Moodle 2.0 with conditional activities).
My priority at this point will be to minimize or local hack impact so that future upgrades will go much more smoothly. I'm very excited about Moodle 2.0 because I think it will meet some of our own needs better than previous versions.
We do have some ways of doing things that I doubt would make it in to Moodle core. For example we want to make sure our students can't get into a course at the end of a certain period of time, but unenrolling them is not sufficient, since we still need their scores and work retained in the database. Plus, we need to prevent students from seeing anything past the first lesson until they click through on an agreement page (though I suspect that can be taken care of in Moodle 2.0 with conditional activities).
My priority at this point will be to minimize or local hack impact so that future upgrades will go much more smoothly. I'm very excited about Moodle 2.0 because I think it will meet some of our own needs better than previous versions.
The team that has been doing this -- making significant changes to moodle, most of which get integrated later -- for the longest time is the Catalyst team. I have posted a lot here about using DSCMs such as GIT for this.
You want to avoid CVS and SVN like the plague for this kind of work. Google for GIT and DSCM on this forum, I've fleshed out the reasons, and the workflow a few times here
You want to avoid CVS and SVN like the plague for this kind of work. Google for GIT and DSCM on this forum, I've fleshed out the reasons, and the workflow a few times here
Ah! Great! Thanks, Martin. I'll take a look for some of your posts on this topic.
Martín,
if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs.
if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs.
if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs
That is a bit contentious point, synonymous with saying that vim is the preffered editor of Moodle
The community is large and has many different views and I don't think you could generalise like this. Having said that, some of the arguments in favour of this discussion is in MoodleDocs and I think many people can see the the decentralised light
You want to avoid CVS and SVN like the plague for this kind of work.
Disclaimer: I use git to track my changes, and I really think it's a better SCM than CVS or SVN.
But I think we need to put Martin Langhoff's assertion into perspective and qualifiy it a bit more.
Most people don't make local changes that later get into core. They just need to track their local changes, so the features Martin L. needs more are 'irrelevant' to those people.
So the only competitive egde of git over CVS or SVN in that particular scenario is the ability to do repetitive merges without having to deal with tags/revision-numbers to remember the last merge point, and the better merge algorithms it uses (I don't take merge speed into account because most people don't do dozens of merges in a day).
Which IMHO is a big plus, but on the other hand you have some 'drawbacks':
- No IDE/Windows GUI integration (as Sam Marshall likes to point out
)
- A steeper learning curve (which later more than pays for itself
Saludos. Iñaki.
Just to point out a different disadvantage of using bleeding-edge software so that I don't get accused of saying the same thing over and over (OH WAIT TOO LATE)...
Basically I think git is probably great and I hope it does gain critical mass but at present I still doubt that it's an appropriate recommendation for anyone who isn't highly comfortable with Unix command-line tools. (Even if you can get the half-finished GUI tools to work, you'll probably still need to use commandline frequently, would be my guess.)
By the way if/when git actually does gain that wider support, it may even have a shallower learning curve than CVS because I presume it doesn't have the ridiculous gotchas - binary/ascii file distinctions come to mind here...
--sam
- Harder to get peer support.
Basically I think git is probably great and I hope it does gain critical mass but at present I still doubt that it's an appropriate recommendation for anyone who isn't highly comfortable with Unix command-line tools. (Even if you can get the half-finished GUI tools to work, you'll probably still need to use commandline frequently, would be my guess.)
By the way if/when git actually does gain that wider support, it may even have a shallower learning curve than CVS because I presume it doesn't have the ridiculous gotchas - binary/ascii file distinctions come to mind here...
--sam