Unfortunately, though, this has come back to bite me in the rear. Hearing Martin speak at the Moodle Moot has convinced me that we should move to Moodle 2 when that becomes available, but before that we need to move to Moodle 1.9 (we're currently customizing Moodle 1.8 which I may give up on as a lost cause). The customizations I've made have gotten in the way of doing this easily.
So what are the best ways that you've found to get around this situation? Some approaches I'm taking this time around to make the process smoother next time are:
1. Using includes wherever possible instead of large chunks of customized code;
2. Heavy documentation, and commenting out lines of code I'm replacing instead of deleting them outright; and
3. Instead of making changes to pre-existing tables, create additional tables linked via foreign keys to the existing ones. For example, instead of adding the two additional fields to the user table that we need, create a separate table that contains our specific data and simply link that back to mdl_user by the id key.
I can't be the only one in this situation. What are the best ways other people have found for doing this that can minimize the pain in migrating to future versions of Moodle while maintaining customizations?
- Language packs
- Course formats
- Resource types
- Activity modules
- Question types
- Enrolment plugins
What are your biggest problems when upgrading? Custom code is lost? Code conflicts with version control? Broken function calls? All of the above?
Unlike Howard, I've found that even 'positive' code changes can be hard to get agreed for inclusion in Moodle core, although we certainly have managed to get in many such changes (most of which were a good idea ). We do have direct commit access but I generally don't use this without asking unless it's for a straightforward bugfix.
Sometimes the difficulty in getting it included is because what we did was good for us but doesn't really fit the way Moodle works, in which case it's obviously right for moodle.com to reject it; with minor improvements there is sometimes an attitude that 'well this needs to be completely rewritten for [some future version] so don't change anything now'; but most often it's just hard to get agreement because Moodle.com developers are busy, and of course we are too.
For example I have a big improvement to the language system that allows plurals (and other things) to work properly, and I've asked a couple of times but not really got approval - I'm going to need to create more fuss to get that into core so maybe it will never go in (even though I spent two days coding it, along with significant time discussing it in forums). This isn't even in our local version either - I don't want to put it in there precisely because of the difficulties it will cause in merging future updates, as in this thread - so it might have been a complete waste of my time. MDL-12433 if anyone's interested
Anyhow, sob story over - in addition to these problems, we have many code changes which are not actually a good idea, just a quick hack to solve a local problem. So I certainly wouldn't want to get those changes into official Moodle.
I agree that wherever possible all changes should be done within the plugin framework (modules, blocks, course formats). This will minimise problems. Other code that doesn't fit into this should go in the 'local' folder where possible (which is where local database changes should live). Language changes should all go in a _local folder or in dataroot, with absolutely no changes to en_utf8. For our own custom plugins, we always keep language files with the plugin (inside the plugin folder) rather than in separate lang folders as this makes them easier to manage. Of course we have our own theme (child of 'standard') to do all CSS changes so we generally don't touch standard theme.
When we do need to change core code, we do so as follows:
// ou-specific begins
// ou-specific ends
If we were replacing some existing code rather than just adding it, we often (but not always, hm) leave in the original code commented-out.
We also use
// ou-specific begins (until 2.0)
// ou-specific ends
if we have code which IS going into moodle for the next version, so we know that the ou-specific section should be overwritten then (but not for a 1.9.x point release).
// ou-specific begins (until MDL-12345)
// ou-specific ends
That's if we have a patch in the mentioned bug number which does what we want, but we're waiting to see whether anyone will agree we can include it in moodle.
We also use the same words but with the relevant comment indicators if we need to make changes in other types of file that don't support // comments.
Merging a new version is done by hand. For folders and files that don't contain 'ou-specific' we just overwrite the whole thing with the new version from core. When a file does contain that string, the person who merges code sorts it out manually, making changes to our customisation if necessary to keep it working, and possibly discussing it with whoever wrote the change.
I think it's a reasonable aspiration though. The flow of thought often goes...
1. I've had a FANTASTIC idea for a new feature
2. Post proposal
3. Response = that's a really stupid idea, have you thought about...
4. Good point, go to plan B
4. I really have to have this. So reformulate plan OR as a last resort do a local hack.
Even then my view is that if there is a way to do it as a plugin use that route. For example, don't modify an existing question type, take a copy and make a new type. Less pain in the end (sometimes!).
IME, if you are using a decent SCM, it is a bad idea to add "this code is mine" comments.
There are reasons for this:
A good SCM will tell you immediately, effortlessly what files are touched ("dirty") on your branch, even if you've merged repeatedly. If you keep "this code is mine" comments around, you must remove them when you put them upstream -- even if the code does not change. By removing them, you are merging a different patch upstream, and you will get a conflict next time you merge. A trivial conflict which will consume time to resolve where it could have been auto-resolved. If in the same area of code you have other changes that you aren't putting upstream, the conflict becomes more complex to resolve.
On the other hand, if you have the patches "clean", then you can merge them upstream unchanged, and that good SCM of yours will realise, and automerge, not giving you any grief at all. Go home early, relax.
Life is too short - WAY too short - to be spent merging with bad tools. I used CVS for keeping local branches way too long because there weren't any viable alternatives. I am sure I lost a few years of life expectancy because of those horrid merges :-[
Like most people we use CVS (and I don't think SVN is much better in this respect) and have no very clever way of merging core/local changes, which means these marker comments are invaluable.
We do not generally have problems with the situation you described (same patch goes to core) because after checking in a patch to core, we remove any such ou-specific tags from our own code [we know that if we later merge with 1.9.2, it's ok to replace with the core version]. Where we do have a problem is with having to manually merge with core in the first place! I agree it's a total waste of time and we would do better to use a 'good' version control system - but Git's interfaces with real-world use are still not finished (finished support for neither Eclipse nor standard Windows GUI; unfinished projects for some of these appear possibly stalled; not sure if there is any support at all for other IDEs such as NetBeans, or for OS X GUI for instance).
Basically you are right that using Git (or similar, but probably Git) is obviously the future, and that cvs/svn cannot really cope with two moving targets (local moodle changes + core moodle changes). But I also think that anyone who isn't extremely knowledgeable about version control is going to have a lot of trouble getting a Git system set up the way they want, and some inconvenience (lack ot tool/GUI support) once they've set it up.
I do hope that the OU will move to Git for our Moodle development at some time in the relatively near future but I also hope we get a fully-working (or at least nearly-fully-working) Eclipse plugin for it first! Until then let's continue giving horrible merge jobs two or three times a year to somebody who isn't me.
(I doubt we have spare money to fund that development unfortunately... seems like a pretty desperate need.)
My preferred IDE is Quanta. Eclipse, since it's written in Java, is way too slow on my clunky old work machine to make it particularly useful.
By the way, Eclipse performance and memory usage is pretty good - er, relatively - if you are just using it as a Java IDE. Unfortunately the PHP extension seems to be pretty poor; it does swallow a completely infeasible quantity of RAM (and lots of cpu too). For Java development eclipse used to be, if not ideal, then at least perfectly usable on my old AMD Duron 1.6 with 256MB RAM...
Having gone through a 1.6 to 1.8 upgrade at HSU and now 1.8 to 1.9 upgrades for clients at Moodlerooms the svn repository allows for diffing of files, merging of code and more importantly reversion of code if a mistake is made.
What is the best way to do this? Since 1.8 was setup with out using CVS, can we download from the central CVS and merge it's changes with our current 1.8?
We plan on making future changes, and may even submit some things to contrib. We want to implement some kind of versioning. If we keep track of our own changes using CVS, will we need two CVS folders in each install? How do you track your own changes and still be able to merge down new versions from Moodle?
Finally, is CVS the best answer for our local versioning? we run 4 different moodle instances, and so I need to track changes in each of them.
Thanks for any help you can give a newbie.
In anticipation of this, all modifications to the original 1.8.2 had been marked by identifyable comments, so I was able to port my modifications manually to the 1.9-code, by searching for the comments.
I do not delete code, but rather use comments to deactivate it. Thus, if necessary, everything can be returned to its original state without too much of a hassle even without source control.
To stay up to date, I CVS update the development system as well as the production system once a week to MOODLE_WEEKLY.
The development system source code, however, is version controlled by Subversion. This approach has several advantages:
- The CVS update is not hampered by the .SVN-Folders
- Subversion can be told to ignore the CVS-folders when I commit my changes to my Subversion repository.
- There is no need to distinguish between different CVS-folders
This is the sort of thing git could be useful for automating so if you really feel like getting a headache, now's the time to start learning! I'll let the git advocates tell you how though, since I don't know...
We do have some ways of doing things that I doubt would make it in to Moodle core. For example we want to make sure our students can't get into a course at the end of a certain period of time, but unenrolling them is not sufficient, since we still need their scores and work retained in the database. Plus, we need to prevent students from seeing anything past the first lesson until they click through on an agreement page (though I suspect that can be taken care of in Moodle 2.0 with conditional activities).
My priority at this point will be to minimize or local hack impact so that future upgrades will go much more smoothly. I'm very excited about Moodle 2.0 because I think it will meet some of our own needs better than previous versions.
You want to avoid CVS and SVN like the plague for this kind of work. Google for GIT and DSCM on this forum, I've fleshed out the reasons, and the workflow a few times here
if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs
That is a bit contentious point, synonymous with saying that vim is the preffered editor of Moodle .
The community is large and has many different views and I don't think you could generalise like this. Having said that, some of the arguments in favour of this discussion is in MoodleDocs and I think many people can see the the decentralised light
You want to avoid CVS and SVN like the plague for this kind of work.
Disclaimer: I use git to track my changes, and I really think it's a better SCM than CVS or SVN.
But I think we need to put Martin Langhoff's assertion into perspective and qualifiy it a bit more.
Most people don't make local changes that later get into core. They just need to track their local changes, so the features Martin L. needs more are 'irrelevant' to those people.
So the only competitive egde of git over CVS or SVN in that particular scenario is the ability to do repetitive merges without having to deal with tags/revision-numbers to remember the last merge point, and the better merge algorithms it uses (I don't take merge speed into account because most people don't do dozens of merges in a day).
Which IMHO is a big plus, but on the other hand you have some 'drawbacks':
- No IDE/Windows GUI integration (as Sam Marshall likes to point out )
- A steeper learning curve (which later more than pays for itself )
- Harder to get peer support.
Basically I think git is probably great and I hope it does gain critical mass but at present I still doubt that it's an appropriate recommendation for anyone who isn't highly comfortable with Unix command-line tools. (Even if you can get the half-finished GUI tools to work, you'll probably still need to use commandline frequently, would be my guess.)
By the way if/when git actually does gain that wider support, it may even have a shallower learning curve than CVS because I presume it doesn't have the ridiculous gotchas - binary/ascii file distinctions come to mind here...