Moodle in English: Upgrade to new major version using git

Skip to main content

General developer forum

Upgrade to new major version using git

We track the upstream Moodle code by adding the Moodle Git repository as a remote to our repo and then merging our branches with MOODLE_21_STABLE. This seems to work. We can make small changes to core code, and still track the upstream code easily.

But yesterday I tried to merge ou Moodle21 based branch with the new upstream MOODLE_22_STABLE branch and that led to problems. I did

git merge -s recursive -X theirs origin/upstream22

but the problem is that the same code is committed twice, in the 2.1 and 2.2 branches of the Moodle code (example: commit in 2.1, commit in 2.2 ). I think this makes it impossible to do what I tried to do: upgrade to 2.2 doing a merge with my 2.1 based branch: how is git going to understand that the two commits are the same code?

If my understanding is correct, how should I upgrade our 2.1 based branch?

Average of ratings: -

Permalink | Reply

This is a great question, I know a few of us are trying to come up with the best workflow with this and i'd be quite interested to hear what people decide on as a good solution.

The problem is that when we branch, we diverge from master and so a merge will not do because it will try and merge the 'branched' changes as well as your customisations.

Because the stable branches diverge from master there is just no way to do a simple merge that I can think of because you'd be merging in the divereged branch. So I think that the right approach will be to rebase your changes from (say) 21_STABLE onto 22_STABLE:

 

git rebase --onto origin/MOODLE_22_STABLE MOODLE_21_STABLE

Does that produce a branch with all the customations in place for you?

(For us, this didn't work correctly and i'm trying to work out if it was because of the complex customisations we did or because there is some other git-rebase subtility i'm missing)

Average of ratings:Useful (1)

Permalink | Show parent | Reply

We are also currently struggling with the best way to manage this sort of thing and rebase looks like the best option so far for core tweaks. It has the drawback of making a mess of other people's checked out code when they merge, so we have been thinking of a plan where we tell everyone to commit all changes, then rebase everything off the latest Moodle as a different branch. We can then use the date in the branch name to keep track of what happens, and tell everyone to switch to the new branch.

We have also been looking at this rather nice git branching model by Vincent Driessen and are going to adopt it for all our internal dev work. The strategy there is to always commit any bug fixes on feature branches into the develop branch as well as releases, so you theoretically never really diverge. Moodle doesn't use this way of doing things right now, but it may help to do it more.

If you like that model, BTW, there are some git extensions made by the developer that make it even easier to follow.

Long term, it would be great if Moodle added a hook system akin to Drupal's, which would make core hacks less necessary, but that's another discussion entirely

Average of ratings: -

Permalink | Show parent | Reply

yes, that works nicely for me with these two scenarios:

* a branch whose only customizations were added files

* a branch with a handful of customizations, including changes to standard Moodle files

Thanks for suggesting it! And I hope you can get it working for your case.

This was with git version 1.7.7.2.

Average of ratings: -

Permalink | Show parent | Reply

As a follow-on to Dan's post, for one of our 2.2 upgrades, I've basically followed this procedure but for each feature branch instead of the deployment as a whole, and I've rebased them onto the v2.2.0 tag, rather than the MOODLE_22_STABLE branch.

Each of our feature branches now contains the moodle version - e.g. [customer]-feature-2.2-[feature_name]. This way we can (if we so desire) keep historical branches if we discovered something broke.

I've then created a new staging branch on the v2.2.0 tag, and merged each of our feature branches in.

To make life easier for the next release, as I've gone along, I've also created issues in our local redmine installation to ensure that we don't forget any feature branches in the future (I based this idea after reading something similar in a blog post by Tim Hunt - http://tjhunt.blogspot.com/2011/07/keeping-discipline-of-not-changing.html).

For any upstream fix that we cherry-pick in, we're also creating an issue in our redmine for that.

When 2.3 comes out, we should now be in a much better state so that we can simply:

take each feature branch and rebase it onto the v2.3.0 tag (should be much easier now I've put the leg work in);
merge each of the new feature branches into our new staging/deployment branches; and
check whether each of the upstream fixes we've applied have made it into 2.3, or whether they still need to be included.

To check we've not missed anything, I've also run a git diff against the release tag of our current version, and then applied this to a clean v2.2.0 tag; then compared the diff with our new release.

Andrew

Average of ratings: -

Permalink | Show parent | Reply

As San said, the problem is that the MOODLE_21_STABLE branch has diverged a long way from master or MOODLE_22_STABLE because of the way we handle bug fixes. Therefore, even if you do

git checkout -b test MOODLE_21_STABLE
git merge MOODLE_22_STABLE

you will get a lot of merge conflicts. So, if you try it with your custom branch, which we will suppose is close to MOODLE_21_STABLE,

git checkout my_customisations
git merge MOODLE_22_STABLE

then you will get a mess of merge conflicts, some of which are because of the 2.1 and 2.2 differences, and others which might be because of your changes.

What I am planning to try to upgrade the OU code (probably later this week) is

git checkout -b merge_helper_branch MOODLE_22_STABLE
git merge --strategy=ours MOODLE_21_STABLE
git checkout my_customisations
git merge --strategy-option=patience merge_helper_branch

To explain that, the merge command can take some options to control how it does the merge. The --strategy=ours bit says "make a commit that is recorded as a merge of the two branches, but just use the code from our branch, ignore the code in the other branch." That creates a commit that has all the 2.2 code, but where git can see how that relates to 2.1.

Then we go to our code, and merge in that new branch, rather than trying to merge in 2.2. The --strategy-option=patience bit bascially tells git to try harder at sorting out merge conflicts.

I have no idea how well this works, but it is probably worth a try. I have no idea whether --strategy-option=patience will make any difference.

Average of ratings:Useful (3)

Permalink | Show parent | Reply

Hi,

And this? (with an existing local_21_STABLE full of customizations tracking MOODLE_21_STABLE)

git checkout -b local_22_STABLE MOODLE_22_STABLE

git format-patch MOODLE_21_STABLE..local_21_STABLE --stdout | git am

There is a significant difference between the results of rebase and the results of patch + am?

Average of ratings:Useful (2)

Permalink | Show parent | Reply

Yes, this is what I do on my production servers on the major upgrade. It is also described at http://docs.moodle.org/dev/Git_for_developers#Applying_a_set_of_patches

Average of ratings: -

Permalink | Show parent | Reply

Just to add to this as I tripped up a few times, the full workflow I used was:

git checkout MOODLE_21_STABLE
git pull # Ensure the 2.1 branch is up to date against upstream
git checkout local_21_STABLE
git merge MOODLE_21_STABLE # Ensure your local branch is up to date
php admin/cli/upgrade.php # It seems sensible to upgrade to the latest 2.1 at this point
git checkout MOODLE_22_STABLE
git pull # As above, but with the 2.2 branch
git checkout -b local_22_STABLE MOODLE_22_STABLE
mkdir .patches
git format-patch -o .patches MOODLE_21_STABLE..local_21_STABLE # Create the patches - check you've got what you were expecting
git am -3 .patches/* # Apply the patches
rm -rf .patches
php admin/cli/upgrade.php # Run the upgrade

If you encounter and fix conflicts while running git am, ensure you run git add {filename} before running git am --resolved or it will complain about "unmerged paths in your index".

Average of ratings: -

Permalink | Show parent | Reply

Hi Tim,

I have tested this method and it seems to have worked for me upgrading from 2.0 to 2.1. I still have to do some indepth testing to do, but at first glance it looks good. It only threw up 2 conflicts which were pretty trival based on my custom code. The option --strategy-option=patience threw up error as didn't exist, probably due to my git version. I had tried the rebase technique mentioned below , that was a painful process that didn't work.

Thanks,

Troy

Average of ratings: -

Permalink | Show parent | Reply

Thanks for the feedback.

Average of ratings: -

Permalink | Show parent | Reply

Hi all,

the solution provided by Tim works great for me between a custom branch based on the MOODLE_22_STABLE upgraded to the MOODLE_23_STABLE.

Thanks a lot, it solved me a lot of time and headaches.

all my best

Eric

Average of ratings: -

Permalink | Show parent | Reply

git checkout your_branch
git checkout -b your_branch_22_test
git rebase --onto upstream/MOODLE_22_STABLE upstream/MOODLE_21_STABLE your_branch_22_test

This should replay your commits not present upstream on top of 22.

Average of ratings: -

Permalink | Show parent | Reply

Yes, it should (and what you type there is the same as what Dan said in his post, the current branch is the default for the last argument). The interesting thing is that this does not work, at least it was not working when Dan and I discussed this on Jabber.

Average of ratings: -

Permalink | Show parent | Reply

Hmm, Interesting; I've done it a few times now with no trouble so I wonder what gives. It seems like git for me ignores the 'new' things on the chosen upstream - most recently, took a 2.0.2 site to 2.1.2.

Average of ratings: -

Permalink | Show parent | Reply

Unfortunately, I don't have time or opportunity to investigate on my own, but I'm curious how a tool such as stacked git or guilt would do for handling Moodle customizations. These tools allow you to maintain a patch set on top of a git repository, which should allow patches to be applied on top of other branches relatively easily. (Unfortunately, guilt's usual download location is currently unavailable, but it can be retrieved from other sources, such as Debian.)

Average of ratings: -

Permalink | Show parent | Reply

I'm wondering if this problem could be solved by modifying core's workflow? A very interesting workflow is well documented in the git help pages if you type "git help workflows". It's a merging based workflow, but basically anyone in the Moodle community who manages their customizations ontop of Moodle using git could track against the "maint" branch which is basically a pointer for the latest stable version. Moodle can still support their old versions with "maint-X.Y.(Z-1)" which will diverge due to specific fixes for those older versions, but that's OK as anyone using that should consider that as only maitenance and not for primary development. Also, because this is a merge based workflow, one shouldn't see different commit objects for the same fix in the latest stable (maint) and in the next major version of Moodle (master).

The real point here is that a new major version of Moodle would be merged into maint and, since Moodle core developers know the code the best, they would be well suited for resolving any conflicts. Then, everyone in the community would benefit from that merge and would be able to upgrade seamlessly.

Cheers, Mark

Average of ratings: -

Permalink | Show parent | Reply