Local collaboration and Git - Yeah another git question!!

Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Number of replies: 20
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Several badly thought through parts to this question...

I want to do this...

* git.moodle.org
** my organisation's git repository (pulled from git.moodle.org)
*** developer 1 (local git)
*** developer 3 (local git)
*** etc.

So, probably nothing unusual. I want to collaboratively develop Moodle stuff pushing and pulling from my local upstream repository.

On top of that I want the upstream repository itself to pull updates from the upstream git.moodle.org. What's the best way to organise this and/or what are the pitfalls.

Part II

Some of the developers are already working with git and pulling directly from git.moodle.org. How can I put their current work into new branches (I assume I mean branches) in the local repository and not break it horribly?

Ta smile
Average of ratings: -
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Mary Cooch -
Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators
Every time I hear the word git now, I think of you Howard!big grin
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I have been playing around a lot with git branches over the last day to try to get my head fully around them.

I think I'm getting there now...

Firstly, do not worry about developers pulling updates direct from git.moodle.org, git is very clever at spotting changes that have already been applied and doesn't try to add them again.

If you want each developer to have a branch in a shared repository, you can (as far as I have been able to work out from personal experimentation) do the following.

1. set up your shared repository with: git init --bare --shared OR git clone --bare --shared git.moodle.org (the latter is probably what you want)
(I'm going to assume, from now on, that the full path to this repository is 'mymoodle.git')
2. get each developer to push to their own branch on this repository:
git push mymoodle.git master:developername
(that takes their master branch and copies it to the shared repository in a (new) branch called 'developername', with each developer putting their own branch name in after the colon)
(note 2 - they could also do: git push mymoodle.git someotherbranch:developername instead)
3. To make things more convenient for the developers, they probably want to do the following config settings:
git config remote.origin.url mymoodle.git
git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
git config branch.master.remote origin
git config branch.master.merge developername

Now if they do a 'git pull' or 'git push' from their 'master' branch, it will be sent to/from their personal branch in the shared repository

4. If someone wants to grab the work done by another person, they could type:
git pull origin/name_of_other_developers_branch
and that would pull all the changes from that branch into their current branch
OR, if they just wanted to have a look at another branch, then a good method would be the following:
git config --global branch.autosetupmerge true (This line may no longer be necessary, I think this is now the default)
git checkout -b name_of_other_developers_branch origin/name_of_other_developers_branch

This will make a local copy of the other developer's branch that will be automatically updated whenever any 'git pull' operation is done (although a 'git merge' will also be needed if that branch is not currently checked out).

This may be the wrong way to use git, and I am willing to stand corrected, but I think it *should* work.

Thinking about it, you could also do just step 4 above, to pull another developers work into a branch on YOUR development space (rather than going via a shared repository), with the command being something like this:
git remote add sensible_alias_name path_to_other_developers_work
git checkout -b local_name_for_branch sensible_alias_name/branch_you_want

I hope that's not too confused...
In reply to Davo Smith

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Cheers...

I think it's more likely that the branches on the shared respository would be different projects (that multiple devs would work on). I'm reasonably confident about that though.

I guess then that someone comes along every so often and does 'git pull' on the shared repository to get the latest stuff from git.moodle.org. I'm much less sure how that might be merged into project branches if that is required. Can that just be done in the same was as you do it locally?

I don't think I've seen that 'colon' notation before. What does that do?
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
In reverse order, the colon notation (when doing a 'push') gives the name of the local branch to push from and the name of the (possibly new) remote branch to push to.

e.g. 'git push origin master:davomaster' would create a new branch on the remote repository pointed to by 'origin', the branch would be called 'davomaster' on the repository and would be linked to the 'master' branch in my local development repository.

As for merging in the latest changes, the safest way would be for someone who is working on that branch to pull from 'git.moodle.org' (as that way, they can deal with any conflicts / breakages), but you may be able to get away with going to the shared repository and then calling:
git pull git.modle.org (which will pull the latest updates into master)
git push . master:name_of_other_branch (which will push the updates from the master branch into the other branch, but will only work if there are no merge conflicts, the other developers would then need to do a 'git pull', just the same as when any other changes had been pushed to their branch on the shared repository)

Remember, that if you are on a branch that already has updates pulled in from 'git.moodle.org' and then you pull in changes from another branch that has not had the updates from 'git.moodle.org', then this will either work without problems (90% of the time) or will require exactly the same merge conflict resolutions as would have been needed if the branch *had* pulled from 'git.moodle.org' (at the time when that pull was done, if you get my meaning).

Similarly, as already mentioned, it is perfectly safe for two people to independently update from 'git.moodle.org' and then merge their changes together.


In reply to Davo Smith

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Ooohh.... my head's hurting again.... I'm not really clear on the best way of getting Moodle updates. So if you pull (locally) and then do a push to the shared repository, this will deal with the *all* (moodle) branches or just the bit you're working on.

I'm not sure I fully understand push and pull yet sad
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Ooohh.... my head's hurting again.... I'm not really clear on the best way of getting Moodle updates. So if you pull (locally) and then do a push to the shared repository, this will deal with the all (moodle) branches or just the bit you're working on.

If you don't specify any branch names in your push commands, it depends on how you have configured your remote repository (ranging from only a small subset to all the branches you fetched/cloned from there). So the easier way to do it is always specify the branches you want to push:

git push shared-repo my-local-branch-to-push my-other-local-branch-to-push ...

Saludos. Iñaki.

In reply to Davo Smith

Re: Local collaboration and Git - Yeah another git question!!

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

but you may be able to get away with going to the shared repository and then calling: git pull git.modle.org (which will pull the latest updates into master)

This won't work, because you can't pull on a 'bare' repository (there's no working copy to merge the changes with). You need to 'fetch' instead of 'pull', but that won't merge the changes into your local branches unless you do that manually, so that's why I thinkg pulling from git.moodle.org on developers local repos and then pushing to the shared repository is the way to go.

Saludos. Iñaki.

In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I want to do this...

If you can read Spanish smile, I've got a detailed document that I submitted to the Spanish Barcelnoa '08 Moot which talks specifically about this. It uses Catalyst's repo in the examples (git.moodle.org wasn't available at that time), but it's otherwise still 100% valid.

You can get it at http://tinyurl.com/4eyj3s. It's under a CC-BY-SA license.

On top of that I want the upstream repository itself to pull updates from the upstream git.moodle.org. What's the best way to organise this and/or what are the pitfalls.

Some of the developers are already working with git and pulling directly from git.moodle.org. How can I put their current work into new branches (I assume I mean branches) in the local repository and not break it horribly?

You add a new remote repository pointing to each developer's git repo specifying the remote branches you want to fetch. Say you want to fetch my 'MOODLE_19_STABLE_LDAP_REFACTOR' branch (as a tracking branch) from my ldap-refactor.git repository at git.mondragon.edu (calling it 'mondragon'), and fetch the branch inmediatly:

git remote add -t MOODLE_19_STABLE_LDAP_REFACTOR -f mondragon git://git.mondragon.edu/ldap-refactor.git/

Now you can create a local branch (I prefer to create them as tracking branches, but this is optional) based on that remote branch with:

git branch --track my-19-ldap-refactor mondragon/MOODLE_19_STABLE_LDAP_REFACTOR

and hack it to pieces without breaking anything at all, except your local branch smile

Saludos. Iñaki.

In reply to Iñaki Arenaza

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I have enough trouble reading English big grin

You've lost me, sorry.....

I can't see what 'git remote add' does. I read the man page and that's made the confusion worse sad

Are you sure we're talking about the same thing? Say I've been working on my local machine with git and I've cloned git.moodle.org and created some branches and done some work. Now another developer has been doing the same thing and want to work together. How do I make that work. I think we need to integrate our current branches into a common repository on our dev server but I don't see how.
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I can't see what 'git remote add' does.

'git clone' does several operations on your behalf:

  1. It creates an empty local repository.
  2. It adds a remote repository, called 'origin' by default.
  3. It fetches the objects (files, directories, commits, etc.) and references[1] (branches, tags, etc.) from the origin repository. By default it fetches all the objects and references of the remote repository.
  4. It creates a local branch called 'master' from the remote head 'HEAD' branch
  5. It checks out the master branch to the working directory.

Plus a few other details that are not relevant in this discussion.

[1] In git a reference is a 'pointer' to a commit object (to any commit object in the whole repository history), and a head is a reference for the last commit of a given commit tree. A branch is a head and all the commits leading to that head (i.e., the head history).

If you want to work with branches from different remote repositories in your local repository, you can't use 'git clone' for this (as it creates a new local repository in step 1., which you don't want). So you need to perform some or all of the steps yourself.

That's where 'git remote' comes into play. 'git remote' allows you to perform steps 2 and 3 and select wich references you want to fetch in step 3, and then 'git branch'/'git checkout' allow you to perform steps 4 and 5.

Then you can work on your local branches. When you want to push your local branches to a remote repository, you first need to tell git what to push there (which local branches) and where to (into what remote branches). And you use 'git config' for that.

I know this is going to be a bit longwinded (it already is smile), but I'll show all the commands to setup the whole thing and the regular workflow (I'll be rather terse, the full details are on the paper mentionned before smile.

Some names:

  • devel-1: the first developer
  • devel-2: the second developer
  • shared-repo: the name of the shared repository directory

Some assuptions:

  • Linux/Unix environment
  • All the repositories are on the same machine (if not, only need to change git transports from git:// to git+ssh://, etc.)

Devel-1 local repo:

Create & config:

cd /path/to/devel-1/dir mkdir devel-1.git cd devel-1.git git init git config --global user.name 'Devel-1 Real Name' git config --global user.email 'devel-1@email.address' git config --global i18n.commitEncoding 'utf8' git config --global i18n.logOutputEncoding 'utf8'

Get branches from git.moodle.org repo:

git remote add -t cvshead -t MOODLE_19_STABLE -t MOODLE_18_STABLE -m cvshead moodleorg git://git.moodle.org/moodle.git git fetch moodleorg

Create local (work) branches:

git branch --track cvshead-devel-1 moodleorg/cvshead git branch --track mdl19-devel-1 moodleorg/MOODLE_19_STABLE git branch --track mdl18-devel-1 moodleorg/MOODLE_18_STABLE

Shared repo:

Create & config:

cd /path/to/shared-repo/dir mkdir shared-repo.git cd shared-repo.git git --bare init --shared=all chmod g=rwxs,o=rx . # Give appropiate permissions to users in sudo chgrp -R git-moodle . # group 'git-moodle'.

Populate shared repo from local devel-1 repo:

cd /path/to/devel-1/dir/devel-1.git git remote add shared-repo /path/to/shared-repo/dir/shared-repo.git git push shared-repo +cvshead-devel-1: +mdl19-devel-1: +mdl18-devel-1:

Devel-1 local repo:

Configure local branches to merge the right remote branch when pulling:

We need to do this only for the first developer, as the shared repo didn't exist when we created the branches. cd /path/to/devel-1/dir/devel-1.git git config branch.cvshead-devel-1.remote shared-repo git config branch.cvshead-devel-1.merge refs/heads/cvshead-devel-1 git config branch.mdl19-devel-1.remote shared-repo git config branch.mdl19-devel-1.merge refs/heads/mdl19-devel-1 git config branch.mdl18-devel-1.remote shared-repo git config branch.mdl18-devel-1.merge refs/heads/mdl18-devel-1

Devel-2 local repo:

Create & config:

cd /path/to/devel-2/dir git clone -o shared-repo /path/to/shared-repo/dir/shared-repo.git devel-2.git cd devel-2.git git config --global user.name 'Devel-2 Real Name' git config --global user.email 'devel-2@email.address' git config --global i18n.commitEncoding 'utf8' git config --global i18n.logOutputEncoding 'utf8'

Get branches from git.moodle.org repo:

cd devel-2.git git remote add -t cvshead -t MOODLE_19_STABLE -t MOODLE_18_STABLE -m cvshead moodleorg git://git.moodle.org/moodle.git git fetch moodleorg

Create local (work) branches:

From moodle.org:

git branch --track cvshead-devel-2 moodleorg/cvshead git branch --track mdl19-devel-2 moodleorg/MOODLE_19_STABLE git branch --track mdl18-devel-2 moodleorg/MOODLE_18_STABLE

From the shared-repo:

git branch --track cvshead-devel-1 shared-repo/cvshead-devel-1 git branch --track mdl19-devel-1 shared-repo/mdl19-devel-1 git branch --track mdl18-devel-1 shared-repo/mdl18-devel-1

Configure local branches to merge the right remote branch when pulling:

Nothing to do.

Now you can pull and push from any of the configured remote repositories (git.moodle.org and your shared local repo) by specifiying the remote repository name when using 'git pull' or 'git push'

Clear as mud, isn't it smile

If I can find a couple of hours, I'll dump all this into the Moodle Docs and add some useful comments around the commands.

Saludos. Iñaki.

Average of ratings: Useful (1)
In reply to Iñaki Arenaza

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Wow, thanks!

I'm going to read this carefully when I'm more awake wink

Expect more questions big grin
In reply to Iñaki Arenaza

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Large MacDonalds coffee later.....

My immediate reaction is that it seems spectacularly complicated.

Why can I not just clone git.moodle.org as a bare repository to be shared repository and then my devs clone from that to their personal repo? That's intuitively what I would have tried.

I would have expected to have to go to the shared repo and do a 'pull' every morning but I can live with that.

Nothing's ever as simple as it first seems with git wink
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

My immediate reaction is that it seems spectacularly complicated. Up to a certain point, I agree with you. But then you need to remember that you are trying to use a distributed tool in a centralized mode. While possible, it's a bit more convoluted because it was designed with a different work flow in mind.

Why can I not just clone git.moodle.org as a bare repository to be shared repository and then my devs clone from that to their personal repo? That's intuitively what I would have tried.

The problem with this scenario is that while objects (files, directories, commits, etc.) are global, references (heads, branches, tags, etc.) are local. In other words, the MOODLE_19_STABLE branch in git.moodle.org and the MOODLE_19_STABLE branch in mi local repository have the same commit history (up to a certain point), but they are two different references.

This means that making some more commits in git.moodle.org/MOODLE_19_STABLE doesn't make local-repo/MOODLE_19_STABLE pick up those changes. If I do a 'fetch', I get the objects of those commits locally, but I don't touch the local reference. I need to do a 'merge' for that to happen (btw this is what 'pull' does: a fetch followed by a merge).

But a merge needs a working copy (i.e., a checked out copy of the reference) to apply the changes and record the new state in a new commit. So you while you can do this in a regular repo, you can't do it in a bare repository. That's why I propose to do all the merges/pulls in the developer repositories and then push the results (if needed) to the shared repository.

By the way, if you are thinking about using a non-bare repository on the shared repo to avoid this problem, don't do it smile. It's a very bad idea. Pushing to non-base repositories can have all sort of problems if the checked out branch in the shared repo and the branch you are pushing are not the same. I seem to remember git develpers were thinking about checking for this and aborting the push operation altogether.

I would have expected to have to go to the shared repo and do a 'pull' every morning but I can live with that.

You'd need to pull twice anyway: once in your shared repo from git.moodle.org, and a second time in each developer repo from the shared repo. You can save some bandwith (and a little bit of time if your WAN is much slower than your LAN), but that's pretty much the only difference I can think of.

Saludos. Iñaki.

In reply to Iñaki Arenaza

Re: Local collaboration and Git - Yeah another git question!!

by Martín Langhoff -
Excellent writeup! Recommended -- this should be "upgraded" to a wikipage.

Howard -- tracking a 'local' project with various developers in a simple way does require having a local repository (there is actually an alternative strategy, but it's conceptually a bit harder).

I've done this is various SCMs and git is actually the most straightforward one.

Having said that, Iñaki is giving you the full commands that you need to use to set things up, which you do _only once_. And he's using all the key options fully spelt out. Good to get your head around it -- he's not relying on any defaults of dark magic.

Once you're setup and handy with it, it takes a single command to create a new branch on the server.

And I think I could get the setup to be done in 1/3 of the steps, but it'd be rather obscure wink
In reply to Martín Langhoff

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I finally got around to watching Linus Torvald's famous Google talk and now realise that this might not have been a sensible question in the first place even.

I guess a better question might be... how do we all use git and not loose stuff?

One advantage with a central repository of some sort is that you know where the thing to back up is and you know where the first place to look is.
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by Dan Poltawski -

I guess a better question might be... how do we all use git and not loose stuff?

First thing I would say: git reflog. I didn't learn about the reflog for quite a while and its a really nice way to recover from accidents. It allows me to sleep easier at night when using git commands. Makes it feel less 'fragile'. This writeup on gitready is probably better than I can do now.

But also if you think of git push as the same operation (in terms of 'backup') like svn ci and push to a remote branch on your central repository to backup then its essentially the same.
In reply to Howard Miller

Re: Local collaboration and Git - Yeah another git question!!

by François Marier -
Hi Howard,

Here's what we now do at Catalyst:

- We cloned git.moodle.org into a bare repository on our office server (git.catalyst.net.nz).

- We have a cron job which fetches the latest changes every 30 minutes from git.moodle.org and updates the local git.catalyst.net.nz repo.

- Developers pull from the Catalyst git repo and push their own feature branches there as well.

- Each developer is responsible for merging the local copy of MOODLE_xx_STABLE into their own feature branch.

Note that in a setup like this, it's of course important that nobody pushes directly to the local copy of the MOODLE_xx_STABLE branches.

Cheers,
Francois
In reply to François Marier

Re: Local collaboration and Git - Yeah another git question!!

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think is how I thought it might be done.