Upgrading to 2.0 via Git

Upgrading to 2.0 via Git

Brandon Tilley གིས-
Number of replies: 22

Greetings, all,

We currently keep our Moodle build in a Git branch (let's call it custom). We have modules we've added as well as some custom changes to the core source. To upgrade, we merge changes from MOODLE_19_STABLE, review changes (and any possible conflicts with our custom changes), test, and push to live.

I'm curious about the best path to take to upgrade to 2.0. master does not merge cleanly with custom, but it also doesn't merge cleanly with MOODLE_19_STABLE. (I expected some conflicts where we modified the code, since a lot of core has been rewritten, but the many merge conflicts in files we didn't touch surprised me.) Is this normal? Or am I just not performing the merge correctly?

Thanks in advance.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Brandon Tilley

Re: Upgrading to 2.0 via Git

Colin Fraser གིས-
Documentation writers གི་པར Testers གི་པར

You probably are, but the changes are not just to code, they are to the database as well. I am suggesting it is this interactivity that is the source of the issues you are having, but I am not a developer so I may be wrong. Another cause of tension may come from different naming conventions, different theme structures, missing php pages or newly created pages or same named pages but with completely different code and so on. I have just been looking at Petr Skoda's Book module for 2.x, and the structure is just so completely different I am looking at something completely new.

I know this does not help, but the magnitude of the changes made is just unbelievable - so little wonder things are not always simply working. Cheers.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Brandon Tilley

Re: Upgrading to 2.0 via Git

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

There are big changes between 1.9 and 2.0, it does not surprise me that they do not merge.

How big is your git diff MOODLE_19_STABLE..custom?

I suggest you make a new branch, custom2 off master, and re-impelemnt all your changes (perhaps with the help of git rebase or git cherry-pick). That the opportunity to carefully review all your customisations, and decide if you really still need them.

Also think about whether any of the customisations could be moved to proper plugins, or handled using the new renderer system in your theme, rather than by customising Moodle core code.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།:Useful (1)
In reply to Tim Hunt

Re: Upgrading to 2.0 via Git

Brandon Tilley གིས-

I suppose I didn't consider that MOODLE_19_STABLE and master diverged significantly during development; that makes plenty of sense now that I think propertly about it.

The diff is not as bad as I thought; about 2,000 lines of actual changed source in core.

This may actually be a blessing in disguise--we started pretty heavy customizations early on in our implementation without any sort of version control system, so "starting fresh," as it were, gives us an opportunity to do it right.

Thank you for your helpful response!

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Brandon Tilley

Re: Upgrading to 2.0 via Git

Sven Laudel གིས-

I also got a question regarding git and local moodle developement, because i'm not sure how to organize the workflow to do the following scenario (we are also reimplementing our core changes for 2.0).

First i start with cloning the moodle.org git repo. This seems to be easy, but the next step confuses me a bit because i don't know exactly how to organize my branches. On which origin-branch should i base my branch?

I think i would branch from origin/MOODLE_2.0_STABLE, right? But should this be a tracking branch or not? So i would do "git branch (--no-tracking) mylocalmoodle origin/MOODLE_2.0_STABLE".

Now we apply our local customizations to our local branch. This branch is being kept up to date with moodle.org's repo for a while. Synchronizing my local repo works with "git fetch" (does it need further parameter or fetches this all from moodle.org?) and then "git checkout mylocalmoodle" and "git merge origin/MOODLE_2.0_STABLE" to merge in changes from upstream? Would this work?

At some point in the future the next release of moodle (e.g. MOODLE_2.1_STABLE) will be published. Then or some time later i want to switch my local moodle to this new release. How would i do this? It's not clear to me how to change to a new major version. Would it work to merge in "origin/MOODLE_2.1_STABLE" from then on? Or do i need to do something else?

Maybe i should organize my workflow in a completely different way?
All help will be appreciated.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Sven Laudel

Re: Upgrading to 2.0 via Git

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Well, to start with, there is no MOODLE_20_STABLE branch (yet). Currently, there is only the master branch. At some point, the MOODLE_20_STABLE will branch off that, but it has not happened yet.

Anyway, for now you want to make your own mylocalmoodle branch off that.

It is not big deal whether you set that up as a tracking branch or not. If it is not a tracking branch, then to do an update, you would have to type:

git fetch
git merge origin/master (or origin/MOODLE_20_STABLE, once the branch is made.)

Setting it up as a tracking branch is really just a convenience that lets you type git pull instead of those two separate commands. I feeling is that since updating to the latest moodle version is something you would do at most once per week, it is better to type the two commands each time, because it forces you to think about what you are doing.

 

In terms of what happens when Moodle 2.1 comes out. Well, you don't really have to decide until the times comes. There are basically three approaches.

1. You could try doing it as

git merge origin/MOODLE_21_STABLE

from your mylocalmoodle branch.

2. You could try creating a new branch mylocalmoodle21 that is a new branch on top of MOODLE_21_STABLE. You might be able to do that with rebase:

git checkout -b mylocalmoodle21 mylocalmoodle
git rebase origin/MOODLE_21_STABLE

you will probably end up manually fixing merge conflicts with that.

3. You could do a more manual job of creating a mylocalmoodle21 branch, taking the opportunity to review all the changes you have made, and whether you really want them. git cherry-pick might help automate part of that.

 

As to which of those is likely to work best, I don't have enough experience to guess yet. At least you know that if you work in git, you are capturing all the information about what you have done, which should maximise your options when 2.1 is released.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Tim Hunt

Upgrading from 2.0 to 2.1 via Git

Urs Hunkler གིས-
Core developers གི་པར

Now we have Moodle 2.1 and I don't find the best way to upgrade 2.0 with patches to 2.1. (about 10 files are changed)

Tim's proposal

1.) gives 400 something merge conflicts - I expected maximal 10 in the files I have changed. Most are the 2.0 to 2.1 changes and I don't understand why they conflict?

2.) I started to try to rebase into a new branch and it seams that the same merge conflicts show up as in 1.)

3.) With several project repos this is a very time consuming task.

Isn't there an easier way to do something like?

_ create a new branch
_ merge the differences in all changed files of the original branch into the new one with one command

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Urs Hunkler གིས-
Core developers གི་པར
Does anybody know how to upgrade 2.0 to 2.1 via Git?
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

sam marshall གིས-
Core developers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Basically, something like

git fetch
git checkout MOODLE_21_STABLE

Edit: Oh, you mean if you changed a zillion other things and they conflict? Eh, well, that depends how you did it...

--sam

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to sam marshall

Re: Upgrading from 2.0 to 2.1 via Git

Daniel Nelson གིས-

Today we tried doing a merge as well. Also received over a hundred conflict messages. 99 percent of them in files we have not touched. 

Just to make sure there wasn't something wonkey in my local repo I created an entirely new clone of MOODLE_20_STABLE and then merged that with MOODLE_21_STABLE. I still got a huge number of conflicts. Why would a 'clean' 20 generate such a huge number of conflicts from a clean 21?

For us as we have few changes...maybe 20...we are thinking it might be easier not to do a merge at all and just do a clean 2.1 install and manually apply our changes to that. 

Daniel 

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Daniel Nelson

Re: Upgrading from 2.0 to 2.1 via Git

Petr Skoda གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར
Hello,

merging or rebasing between MOODLE_20_STABLE based branch and MOODLE_21_STABLE is not going to work much because we are not merging patches during integration - each branch is going its own way. This may not be optimal unfortunately there is no other easy way because we are doing way too many changes in the stable branches.

Git makes it easy to find out what changes you did in your custom branch, the upgrade path is to reapply the patches on top of the new stable branch and then continue normal merging of each weekly.

The following approaches work fine for me, I am sure there must be at least 10 more ways to do this:

1/ cherry picking - I would personally use some unique identifier in each of my hacking commit, then I could search the git history easily and cherry pick one by one (you should carefully review each change again, there is no guarantee it is going to work even if it applies without collisions)

2/ soft reset, stash, branch, unstash - if you do not care about the individual commits and you want just all the changes you can use soft reset on the old branch, then save all your modified files from the original branch into stash, then create a new branch and apply the stash

Solution also depends on your git UI preference - for me GUI tools make it a lot easier, other developers prefer git command line...

Please remember to backup everything in case it does not works as expected.

Petr
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།:Useful (1)
In reply to Petr Skoda

Re: Upgrading from 2.0 to 2.1 via Git

Erik Ordway གིས-

I was able to rebase but I think that was because I froze our deployed MOODLE_20_STABLE in late june ( '2.0.3+ (Build: 20110623) ) but that may not be the case for others.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Tim Hunt གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

How are your changes recorded in git? I guess there are two main options:

  1. You have not done anything, so git status shows them as uncommitted changes.
  2. You have committed them as a private branch (that is, in the past, you have done something like git checkout -b mycustomisations MOODLE_20_STABLE, and then git add ..., git commit ... several times.
If you are in state 1., get your self into state 2. by following the outline instructions I just gave.
Then, to decide how to proceed, we really need to consider two sub-cases of 2:
2a. you have only done a few customisations on your mycustomisations branch, so it is a short branch.
2b. you have done many customisations on your mycustomisations
branch, so it is a very long branch. 
In case 2a, it is probably easiest to build a new branch with your changes, starting from a clean check-out of Moodle 2.1. There are two ways to do this:
A. Using rebase:
  1. git checkout mycustomisations
  2. git rebase --onto MOODLE_21_STABLE MOODLE_20_STABLE
B. Using cherry-pick
  1. git log MOODLE_20_STABLE...mycustomisations (this gives you a list of the commit hashes of your changes, then for each commit has do... )
  2. git cherry {commit hash}
For both A. and B. there may be merge conficts that you have to resolve.
In case 2b, doing a rebase is probably not feasible, so your only option is to do
  1. git checkout mycustomisations
  2. git merge MOODLE_21_STABLE
but as you found there are quite a lot of merge conflicts you may have to resolve. However, before you start do something like
git diff --stat MOODLE_20_STABLE...mycustomisations
which should give you a summary of which files you have changed.
Then, once you have started the git merge, and have all the conflicts to deal with, you can quickly solve most of them. For any folder, or file, where the git diff --stat showed that you have not made any changes, you can resolve all the merge conflicts in that file or folder by doing
git checkout --theirs folder/or/file/path
git add folder/or/file/path
The approach that works for 2b will also work for 2a, if you prefer.
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།:Useful (4)
In reply to Tim Hunt

Re: Upgrading from 2.0 to 2.1 via Git

Frank Ralf གིས-
I took the liberty of taking this thread as a starting point for the newly created Git FAQ. This was just a quick copy&paste job so feel free to amend and add to it མིག་ཁྱབ་

Frank
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Frank Ralf

Re: Upgrading from 2.0 to 2.1 via Git

Erik Ordway གིས-

So I do not know if using named repositories is the norm but I did so based on some of the early moodle.org using git documentation (moodleorg in this case).  Also I found it necessary to specify that my branch (evergreen20 in this case)  using --set-upstream should now follow MOODLE_21_STABLE. 

?  git checkout evergreen20
?  git rebase --onto moodleorg/MOODLE_21_STABLE moodleorg/MOODLE_20_STABLE
### tell evergreen20 to track moodleorg/MOODLE_21_STABLE
?  git branch --set-upstream evergreen20 moodleorg/MOODLE_21_STABLE
 

Because I did not know if this was in the range of normal I did not update the wiki.  I think that those of us that do not write to git.moodle.org maybe more likely to maintain our own staging server and therefor use named repositories.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Urs Hunkler གིས-
Core developers གི་པར
Thank you everybody for your answers.

One thing I learned - with Git upgrading to a new version is much more work than with CVS in the past.

To find the best approach I will need to try some of the suggested ones - I guess there will be special cases in the details of each approach.
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Petr Skoda གིས-
Core developers གི་པར Documentation writers གི་པར Particularly helpful Moodlers གི་པར Peer reviewers གི་པར Plugin developers གི་པར
If you do not commit your changes to git and keep your modifications in the working tree then it is 99% the same as in CVS. It may seem harder to use git but believe me, once you get used to it you will not want to touch CVS any more.
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Petr Skoda

Re: Upgrading from 2.0 to 2.1 via Git

Urs Hunkler གིས-
Core developers གི་པར
I am working with Git now for some time and I second you saying it's similar easy as CVS when you update within one version.

But upgrading to another version like 2.0 to 2.1 is quite different - as we can see from the many proposals in this thread.
དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

sam marshall གིས-
Core developers གི་པར Peer reviewers གི་པར Plugin developers གི་པར

Yes, it is a bit different when you have complex setups with a lot of custom code. In particular, we're impressed by the reduction in effort required, thanks to using git instead of cvs. Our systems team were able to apply the 2.0->2.1 upgrade to our development systems faster than would have been the case in our 1.9/CVS-based system. (There are other reasons for this as well as git being better than cvs, but that was a factor.)

--sam

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Andrea Bicciolo གིས-

Hello Urs,

I second Petr about Git, even if it appears more complex, you can bend git according to your needs. For example, using git just for upgrading Moodle sites can be much like using CVS: very simple, much faster and scriptable together with the Moodle CLI. Once the git repo is initialized, you only need "git pull" to get the updated files.

 

However, using git for maintaining custom branching may need a more elaborate setup.

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -
In reply to Urs Hunkler

Re: Upgrading from 2.0 to 2.1 via Git

Hubert Chathi གིས-

I haven't used CVS for a long time, but I don't see how you'd do the same thing in CVS any easier.  Can you explain how you would do the same thing in CVS?

དཔྱ་སྙོམས་ཀྱི་སྐུགས་ཚུ།: -