We have been using Moodle for the last few months and everything has been running pretty well. We've made quite a few modifications to the code and use Subversion to track everything. I know many people are probably in similar situations. How do you go about merging your changes when new Moodle code is released? Do you branch?
Our trunk's first checkin was a 1.5.2+ Moodle release. No custom modifications. I like knowing that everything started with core Moodle code and we can always do a diff/blame to see if we broke something or if it was already in Moodle. With the pending release of 1.5.3 I'd love to hear any suggestions about how others do this.
Thanks,
Bryce
We started off life using gnu arch as our internal scm, and have since switched to git. Here's how we do it.
we have an 'upstream vendor branch' which tracks the MOODLE_15_STABLE tag, when we had it in arch we updated it manually, but Martin Langhoff has set up an automatic cvs -> git script.
Then we have a branch off that called mdl-local which we use to develop any customisations that we want to go to all of our installs (and most of which we plan to merge into HEAD for 1.6)
Then off that branch we have any further branches for installs that need really specific customisations that don't have anything in common with the other ones. When we update MOODLE_15_STABLE branch to mdl-local we then merge everything on mdl-local to the other ones.
using gnu arch / tla was ok for awhile but was very slow and caused conflicts when patches came back from cvs - if we committed something to mdl-local and then merged it with MOODLE_15_STABLE, it would conflict coming back, but git is smart enough to see that it's the same patch and not cause conflict.
You can see this all in action here: http://locke.catalyst.net.nz/gitweb?p=moodle.git;a=heads
MOODLE_15_STABLE is upstream, mdl-local is our generic custom branch and then you'll also see mdl-topnz and mdl-nmit which are essentially branched off mdl-local.
git doesn't really think about them as 'branches' but I use that terminology as it'll be familiar to you with svn & cvs.
There's a lot of other heads there as well - some of them (the ones starting with moodle--) are from when we were migrating from arch to git.
Hope that's food for thought! I'm sure Martin L will chime in with more comments as his git-fu is stronger than mine
we have an 'upstream vendor branch' which tracks the MOODLE_15_STABLE tag, when we had it in arch we updated it manually, but Martin Langhoff has set up an automatic cvs -> git script.
Then we have a branch off that called mdl-local which we use to develop any customisations that we want to go to all of our installs (and most of which we plan to merge into HEAD for 1.6)
Then off that branch we have any further branches for installs that need really specific customisations that don't have anything in common with the other ones. When we update MOODLE_15_STABLE branch to mdl-local we then merge everything on mdl-local to the other ones.
using gnu arch / tla was ok for awhile but was very slow and caused conflicts when patches came back from cvs - if we committed something to mdl-local and then merged it with MOODLE_15_STABLE, it would conflict coming back, but git is smart enough to see that it's the same patch and not cause conflict.
You can see this all in action here: http://locke.catalyst.net.nz/gitweb?p=moodle.git;a=heads
MOODLE_15_STABLE is upstream, mdl-local is our generic custom branch and then you'll also see mdl-topnz and mdl-nmit which are essentially branched off mdl-local.
git doesn't really think about them as 'branches' but I use that terminology as it'll be familiar to you with svn & cvs.
There's a lot of other heads there as well - some of them (the ones starting with moodle--) are from when we were migrating from arch to git.
Hope that's food for thought! I'm sure Martin L will chime in with more comments as his git-fu is stronger than mine
Hi!
What you are after is usually called "tracking the vendor branch" or "tracking upstream" (both phrases give you good Google results). You are going to need a very good SCM to help you do that. Our team (Hi Penny!) tracks several upstream projects (GForge/phpWiki/Serendipity for http://eduforge.org , and Moodle for our custom Moodle setups).
At work we use a lot of different SCMs. What I found is that CVS, SVN/SVK and similar won't help you much -- in fact, they won't help you at all. Next we worked with patch-oriented SCMs (ie: Arch/Bazaar and Darcs). We used Arch for about a year, and we found that it had huge performance/scalability problems, and it didn't support tracking an upstream very well.
The third attempt -- with GIT and Cogito -- is going great. I definitely recomend that you try GIT or a similar SCM. Alternatives would be BitKeeper, Mercurial, Monotone and BazaarNG -- of those, BitKeeper is the only really mature tool, and it is proprietary and expensive.
We are currently maintaining a CVS2GIT gateway -- feel free to use it to track the Moodle 'upstream' as we will be maintaining it for a long time
To try with Cogito (which is the cvs-like interface to GIT), install a recent version of GIT and Cogito, and do
cg-clone http://locke.catalyst.net.nz/git/moodle.git#MOODLE_15_STABLE moodle
In the resulting checkout, you'll see that there are two 'heads' (git's name for branches): master (yours) and origin (the one on the public repo).
And then you can just hack on it freely, commit to your local repo -- those commits show up on 'master'. Hacking sessions go like this:
# similar to cvs status
cg-status
# hack hack hack
emacs lib/moodlelib.php
# diff
cg-diff
# commit!
cg-commit
To see your local changes, just do
# commit entries that are local
cg-log -r origin:master
# the diff
cg-diff -r origin:master
To bring in any new changes from the 1.5.x branch, you just do
# brings in the changes, will autocommit if it merges
# cleanly, it will ask you to fix up conflicts if
# there are any...
cg-update origin
# if there are conflicts
emacs lib/conflictedlib.php
cg-commit
To export the individual patches that make up your local customizations in a format that is good to post to forums/bugtrackers:
git-format-patch --mbox --signoff origin
The output of git-format-patch is also good when 1.6 comes out and you have to reapply your patches to the new version. To visualize how the changes go, you can use gitk which is included, or qgit, a very qute Qt UI. See http://sourceforge.net/project/screenshots.php?group_id=139897
If you do try it, jump over to the git@vger.kernel.org mailing list (archive at http://marc.theaimsgroup.com/?l=git ).
This works on Linux and almost any Unix, including MacOSX. Works on cygwin, too, so Windows users don't need to feel left out...
In reply to Martín Langhoff
Re: Merging Custom Moodle Code With Stable Releases
аз Martín Langhoff -
BTW, there are two good SVN to GIT importers. The git-svnimport command included in GIT, and tailor.py
Thanks for your input. You guys have quite a setup.
I did some Googling using the phrases you mentioned and found some info about doing exactly what I need in SVN. It's amazing what you can find when you know what to look for.
The SVN book contains the following info on managing vendor branches:
http://svnbook.red-bean.com/en/1.1/ch07s05.html
There are a decent amount of steps involved, but it looks like a feasible solution. With some scripts to glue some of this together, I think this will work for our needs. I've never looked into using git before. I'm sure it's a great tool, but we just started using Subversion and have been pretty happy with it so far. I don't really want to move to a new system if we don't have to.
I'll try to update this thread with my progress.
Thanks Again,
Bryce
I did some Googling using the phrases you mentioned and found some info about doing exactly what I need in SVN. It's amazing what you can find when you know what to look for.
http://svnbook.red-bean.com/en/1.1/ch07s05.html
There are a decent amount of steps involved, but it looks like a feasible solution. With some scripts to glue some of this together, I think this will work for our needs. I've never looked into using git before. I'm sure it's a great tool, but we just started using Subversion and have been pretty happy with it so far. I don't really want to move to a new system if we don't have to.
I'll try to update this thread with my progress.
Thanks Again,
Bryce
Thanks for your input. You guys have quite a setup.
Cool! Glad to be of help. We do hack on a lot of feature and custom branches, so we got this part well oiled
It is true, SVN does have a basic support for tracking vendor branches, but it is designed based on what CVS does, and that carries some significant limitations. Where this has broken down for us is in the scenario that follows...
It is in your best interest to keep your delta with the upstream minimal, so any features or bugfixes you have you will want to get merged into upstream's release. When you merge again from the vendor branch, CVS/SVN will not recognise that particular change is already in your 'local' branch, and see that change as a conflict.
This is ok at the beginning, but as the customizations you make get more interesting, and the patches you feed upstream get more interesting too, the whole thing gets really messy.
The other thing is that when you are resolving conflicts, you don't get a good history of both sides. You only see the history of your side, which is somewhat limiting.
Anyway, we are SCM geeks We customise and develop Moodle a lot as part of the NZVLE project, and we put almost all the code upstream (check out the stuff coming in 1.6!). But the code transforms in transit. Small bugfixes get quickly into 1.5.x -- and are quickly "echoed back" into our current branches. Feature work transforms quite a bit, usually getting more polish and improvements as it is merged into HEAD for the next Moodle release.
Our situation is quite complex in terms of version control, with lots of bits of code in different stages of merging upstream and being reworked along the way. It's fun stuff, though it makes my head hurt...
Anyway, we are SCM geeks
No kidding! This thread is exceedingly interesting.
Earlier you described that you have gone through a whole evolution from CVS to your current setup. Would that experience enable you to outline what makes moving from one tool/way to next manageable? My totally uninformed guess is that setup like yours has quite a bit of overhead and we "amateurs" shouldn't try to imitate it.
We're nowhere near needing setup like yours internally, but that day may come, and I guess I'm kinda neurotic about anticipating future needs. And now that I think of it, our success with student projects as been less than what was hoped for partly (largely?) because "Moodle is a moving target" (i.e. the lack of SCM).
No kidding!
Earlier you described that you have gone through a whole evolution from CVS to your current setup. Would that experience enable you to outline what makes moving from one tool/way to next manageable? My totally uninformed guess is that setup like yours has quite a bit of overhead and we "amateurs" shouldn't try to imitate it.
We're nowhere near needing setup like yours internally, but that day may come, and I guess I'm kinda neurotic about anticipating future needs. And now that I think of it, our success with student projects as been less than what was hoped for partly (largely?) because "Moodle is a moving target" (i.e. the lack of SCM).
I'm very interested in setting up git as well. I currently have a
CVS server I've been using for tracking the changes I've been making
for my School Engine project. Those include some massive gradebook and parental access changes that I'd like to get into Moodle CVS eventually, or at least have them considered.
But I definitely need to get on board with a great SCM and have looked at Subversion, but think maybe git is what I want. What client program do you use? I use TortoiseCVS and it's great. Is there something like that for git? Where do you recommend I start in setting up my own git server? And do you have any tips before I start?
But I definitely need to get on board with a great SCM and have looked at Subversion, but think maybe git is what I want. What client program do you use? I use TortoiseCVS and it's great. Is there something like that for git? Where do you recommend I start in setting up my own git server? And do you have any tips before I start?
Where do you recommend I start in setting up my own git server?
If it's just yourself hacking, you don't need a server. Get a checkout of MOODLE_15_STABLE as I've described before, and it will automatically know about "master" (your local changes) and "origin" (official MOODLE_15_STABLE).
If you need a repo to coordinate with a team... Let's nag Richard Wyles -- if he's happy with it, I'll enable GIT project hosting on Eduforge so you can host it for free
And do you have any tips before I start?
Join the mailing list, and play a bit with cogito. There isn't a 'TortoiseGIT' -- don't do this if you aren't comfortable with command line cvs! There are some graphical tools, mostly to visualize history -- but the workflow is command-line-driven.
In reply to Martín Langhoff
Re: Merging Custom Moodle Code With Stable Releases
аз Pieterjan Heyse -
If it's just yourself hacking, you don't need a server. Get a checkout of
MOODLE_15_STABLE as I've described before, and it will automatically know about "master" (your local changes) and "origin" (official MOODLE_15_STABLE).This is interesting, does this actually mean that when I have one local moodle cvs cvheckout tree, and I change things in the source code and then do a cvs -dP that cvs wil automagically merge my changes with the updated moodle source tree?
This is fantastic news!
In reply to Pieterjan Heyse
Re: Merging Custom Moodle Code With Stable Releases
аз Martín Langhoff -
I was talking about a Cogito/GIT checkout of the MOODLE_15_STABLE branch as I described earlier in the thread ;) That branch is actually grabbed from CVS by a CVS2GIT gateway, every 24hs.
Git has excellent support for tracking your own changes locally. CVS has some very limited support too. If you've made small changes to a few files, you can still do cvs update -dP and CVS will usually do the right thing. But you cannot 'commit' them to a local branch; unless you do the "vendor branch" thing, which sucks big rocks through thin straws
No kidding!
Earlier you described that you have gone through a whole evolution from CVS to your current setup. Would that experience enable you to outline what makes moving from one tool/way to next manageable?
You need someone to go ahead and explore the new SCM and work through it in detail -- a lead explorer that sorts out all the import kinks, and maps out all the "when we now do bla, we will have to do bleh", and with a good enough understanding of the new system that when the team hits a snag, can sort it out.
Because you always hit snags, mostly small confusions and paradigm changes.
It's usually me in the teams I work with... I am an SCM geek myself -- not by choice but after being burned badly with CVS before. As the explorer for these transitions, I often find myself writing or fixing import/export scripts. So I dabbled quite a bit with the cvs2arch scripts, fixed up git-cvsimport somewhat and wrote a git-archimport script from scratch.
Last week I wrote git-cvsexportcommit, which makes it really easy to say 'apply to cvs this git commit'. Now one of the Wine developers seems to be wrapping it into an automated git2cvs gateway, as Wine development is moving to GIT soon.
On the technical side it's easy: right now you need people used to commandline cvs or similar, open to make a change If you are playing with it, you can easily get a checkout of
MOODLE_15_STABLE from the gateway.
If you have a small team, you'll want to setup a shared repo to sync your 'local' branches. If you've set CVS over SSH before, the setup is very similar. Nothing too hard...
The good thing about using GIT in this way is that it makes it really easy and transparent to look and your patches and push them back into the upstream selectively. It does involve a bit of a cultural shift, though...
In reply to Martín Langhoff
Re: Merging Custom Moodle Code With Stable Releases
аз Brian Jorgensen -
"It is in your best interest to keep your delta with the upstream minimal, so any features or bugfixes you have you will want to get merged into upstream's release."
I know that this thread has been dormant for quite a while; however, I started at Athabasca University about six months ago and it seems that being able to reference well-respected external sources is an important part of building arguments to modify our local workflow.
So I would really appreciate if the moodle community could help me put together a set of basic arguments for why we should be wary of making too many local changes, and why we should be trying:
I know that this thread has been dormant for quite a while; however, I started at Athabasca University about six months ago and it seems that being able to reference well-respected external sources is an important part of building arguments to modify our local workflow.
So I would really appreciate if the moodle community could help me put together a set of basic arguments for why we should be wary of making too many local changes, and why we should be trying:
- to push bug fixes upstream whenever possible;
- to work with the community whenever possible to determine if features would be appropriate to push upstream; and
- to learn to customize by using hooks (function calls) into libraries in a "local" folder whenever possible, extending or subclassing if reasonable, and including/requiring code snippet files if necessary.
Hi Brian,
Generally the best argument (other than the warm fuzzy feeling you get from helping the community) for pushing code back upstream (as well as keeping stuff in local/ and avoiding customising core) is to minimise the pain (and cost!) of upgrading later.
Say you start with 1.9 and you customise it a lot and don't bother pushing any of it upstream. After a year, Moodle 2.0 comes out and people of course want to upgrade, to get to use all the shiny new features. But suddenly you realise that you have all these customisations - and you essentially have to analyse every single one, figure out if it's still relevant, if it is, find where to repatch the code, if it's not, discard it, but realise that maybe Moodle has changed in another subtle way that means you have to customise something else.
Repeat every release cycle ad nauseum.
It gets a bit easier if you have good SCM, but even so.. it's a challenging thing to go through , and it costs money. Every time.
Hope that helps!
Penny
Generally the best argument (other than the warm fuzzy feeling you get from helping the community) for pushing code back upstream (as well as keeping stuff in local/ and avoiding customising core) is to minimise the pain (and cost!) of upgrading later.
Say you start with 1.9 and you customise it a lot and don't bother pushing any of it upstream. After a year, Moodle 2.0 comes out and people of course want to upgrade, to get to use all the shiny new features. But suddenly you realise that you have all these customisations - and you essentially have to analyse every single one, figure out if it's still relevant, if it is, find where to repatch the code, if it's not, discard it, but realise that maybe Moodle has changed in another subtle way that means you have to customise something else.
Repeat every release cycle ad nauseum.
It gets a bit easier if you have good SCM, but even so.. it's a challenging thing to go through , and it costs money. Every time.
Hope that helps!
Penny
I am impressed by the professional solutions to the problem described in this discussion. However perhaps for the more casual developers a simpler solution will be sufficient. It is what I am using and while I have been criticised for the fact that it is not elegant, it is immensely convenient:
I checked out Moodle from SourceForge CVS and then committed the whole thing to our local Subversion repository. So our Subversion repository is holding not only the Moodle code but also the CVS information.
I am therefore now working on code that has both the CVS and the SVN information. So I can diff my working copy against both the SourceForge CVS to see what changes I have made against the main Moodle code but I can also diff against my Subversion repository to see what changes I have made against my own submitted code. This way I can also easily commit any bug changes that I have made to the Moodle SourceForge CVS, at least for those files which I have not customized.
Periodically I update from CVS, all changes to central Moodle are then automatically merged into my code and I then submit that updated code to Subversion, including all the updated CVS information of course.
I am working under Windows and have both TortoiseCVS and TortoiseSVN installed which makes this setup even more convenient, I can do everything by right-clicking on the files in the Windows explorer. No need to learn CVS or SVN syntax.
I checked out Moodle from SourceForge CVS and then committed the whole thing to our local Subversion repository. So our Subversion repository is holding not only the Moodle code but also the CVS information.
I am therefore now working on code that has both the CVS and the SVN information. So I can diff my working copy against both the SourceForge CVS to see what changes I have made against the main Moodle code but I can also diff against my Subversion repository to see what changes I have made against my own submitted code. This way I can also easily commit any bug changes that I have made to the Moodle SourceForge CVS, at least for those files which I have not customized.
Periodically I update from CVS, all changes to central Moodle are then automatically merged into my code and I then submit that updated code to Subversion, including all the updated CVS information of course.
I am working under Windows and have both TortoiseCVS and TortoiseSVN installed which makes this setup even more convenient, I can do everything by right-clicking on the files in the Windows explorer. No need to learn CVS or SVN syntax.
That's pretty much what I'm looking for, and I like the Windows Explorer integration that both programs have. I've installed it and will try it out.
So I've set up my main code in /trunk/. Do you use the branches and tags directories, and if so can you give me your file tree structure and when I might need to branch and tag? I just want to set it up correctly. Thanks.
So I've set up my main code in /trunk/. Do you use the branches and tags directories, and if so can you give me your file tree structure and when I might need to branch and tag? I just want to set it up correctly. Thanks.
You don't have to use branches or tags if all you are doing is hacking Moodle for your own installation. You can simply do all your work in your trunk which you would periodically update from Moodle's stable branch. If however you are working on several projects, for example several clients, each with different modifications, then you would use one branch for each of them. But in general I always try to keep things as simple as possible.
I've been using the local SVN & CVS combo recommended by Gustav (thanks!) and it's being going well. I don't like creating forks of great open source apps so I'm trying to keep my version as close as possible to MOODLE_15_STABLE for submitting patches.
I'd like to send in some patches in the near future (I don't like the moodle bug tracker after using the great bug tracker at http://www.drupal.org), but first I think I'm going to change my local SVN repository according to http://svnbook.red-bean.com/en/1.1/ch07s05.html so I can create some good patches and keep my local modified copy up-to-date. I like the idea of keeping the CVS separate and using SVN to merge the changes.
I'd like to send in some patches in the near future (I don't like the moodle bug tracker after using the great bug tracker at http://www.drupal.org), but first I think I'm going to change my local SVN repository according to http://svnbook.red-bean.com/en/1.1/ch07s05.html so I can create some good patches and keep my local modified copy up-to-date. I like the idea of keeping the CVS separate and using SVN to merge the changes.