Github and Moodle

Github and Moodle

by Shaun T -
Number of replies: 6

Hi,

I work for a large post secondary that runs a Moodle environment. We have been managing our changes with github and git but are looking to improve they way we do things.

Currently with each new version we download a copy of the Moodle source and create a new branch and drop in the code. Then we manually apply our changes and customizations and commit it. We run multiple apps servers and use chef to sync our code between them. The chef scripts pull our code from github.

What I'm looking to do is have our repository track from the Moodle repository so we can pull/merge the changes in when we upgrade. We also want to continue to host our code in github and mange our changes with the github for windows as we are familiar with committing changes this way.

Where I'm a little stuck is setting up the a new repository with 2.6.x code that's tracking the official moodle repository. I've tried the instructors in Git for Administror's but this only allows me to recreate a local repository. I added my private repository and push it up which seems to have broken something as I now have a Detached Head.

I would really appreciate any suggestions as to how to best accomplish this or any alternatives that may work better.

Thanks,

Shaun

Average of ratings: -
In reply to Shaun T

Re: Github and Moodle

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Shaun,

In my two previous jobs, I've handled running Moodle based on a git repository, so I have some experience of this.

We went for a fairly comprehensive approach which I've blogged on. I understand that the Open University in the UK follow a similar kind of pattern, though I'm not aware of exactly how they do things from a technical standpoint. Since I wrote my blog post (and joined Moodle HQ), David Mudrak has moved most of the sites at Moodle HQ to using this same method.

The relevant blog post is at http://thamblings.blogspot.com.au/2013/07/upgrading-moodle-from-git.html - apologies, it's a fairly technical post. I should note that much of the original work in this was conceived by Dan Poltawski and others, and that I've made some refinements to it for sensibility. As I say, since that blog post, Moodle HQ has moved down the same route, and I've spoken to another UK-based University who have based their strategy on the same concept.

Onto your specific problem, I'm afraid that what you've been doing to date has been wrong and this is why you get a detached head. It sounds as though what you've been doing is:

mkdir moodle
cd moodle
git init
git checkout -b v2.5.0
unzip ~/path/to/moodle_2.5.0.zip
git add .
git commit -m 'v2.5.0'
// Wait some time until the next release.
git checkout -b v2.5.1
unzip ~/path/to/mooodle_2.5.1.zip
git add .
git commit -m 'v2.5.1'
// Wait some time until the next release.
git checkout -b v2.6.0
unzip ~/path/to/mooodle_2.6.0.zip
git add .
git commit -m 'v2.6.0'

This has different pre to what we push in Moodle, so when you've added our remote, this is why you get a detached head - the pre is not correctly related to the Moodle prebase and has a different history. What you should try is:

git clone https://github.com/moodle/moodle.git
cd moodle
# Create a new empty repository on github.
git remote add public git@github.com:andrewnicols/mydeploymentrepository.git

# Checkout a new branch for your school based on the v2.5.0 release tag.
git checkout -b myschool_25 v2.5.0

# Add your own changes here.
cp ~/path/to/plugin mod/plugin
git add mod/plugin
git commit -m 'mod_plugin: Add version 201402700'

# Rinse, Wash, Repeat.
cp ~/path/to/plugin block/plugin
git add block/plugin
git commit -m 'block_plugin: Add version 201402700'

# Tag your release - this is best practice and highly recommended.
# Use a concatanation of the actual Moodle version (e.g. v2.5.0, and your sub-version so that you can include new features on v2.5.0
git tag 'myschool_v2.5.0-0'

# Push everything to your public github:
git push public myschool_25
git push public --tags

# Wait some time until the next release.

# Fetch the latest commits from upstream:
git fetch origin

# Merge in the latest branch:
git merge v2.5.1

# Re-tag and push:
git tag 'myschool_v2.5.1-0'
git push public myschool_25
git push public --tags

###
# Oh noes - we need some random bugfix from v2.5.2 which hasn't been released yet:

# Fetch the Moodle repo
git fetch origin

# Cherry-pick the commit(s):
git cherry-pick 
git cherry-pick 
git cherry-pick 

# Re-tag and push:
git tag 'myschool_v2.5.1-1'
git push public myschool_25
git push public --tags

###
# Updating to the next release of Moodle.
git fetch origin
# Make sure that we're on our current 2.5 branch
git checkout myschool_25

# Now create a new branch for 2.6 based on the 2.5 branch
git checkout -b myschool_26

# And we do a rebase where we pick all of the commits since 2.5.0, and place them on top of v2.6.0 instead.
git rebase --onto v2.6.0 v2.5.0

##
# Rinse, wash, repeat.

Note, this isn't the exact strategy that I describe in my blog post, or that we use at HQ, so there may be some issues with that final command because you've merged v2.5.1 and v2.5.2 and ... into your 25 branch but hopefully that should point you in the right direction.

I should also get around to adding some other points which we've found important:

  • Giving each site a unique name to make them easily tab-completable
  • Keep names short
  • Keep capital letters out of your namespace
  • Consider using a different namespace using a / as a separator to allow you to fetch a glob - e.g. luvle/* - I'll blog more on this when I get a chance
  • Read the git rebase manual

Hope that this helps

Andrew

Average of ratings: Useful (5)
In reply to Andrew Lyons

Re: Github and Moodle

by Shaun T -

Thanks, this looks to be what I had in mind. I was originally thinking that it could be done with one branch but I don't think that's possible. Let me see if I understand this.  The idea is to have two branches, one unmodified moodle.org and one custom, and when new updates come out just pull the changes to the moodle.org branch and then merge into the custom branch. Is that a fair one line summary?

I'll have to spend some time trying this and review the blog. Thanks again.

In reply to Shaun T

Re: Github and Moodle

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

As a one-line summary, that's about right.

You don't really want to be tracking master, so you really need one branch per stable version. You can't have one version which tracks multiple stable branches because sometimes when we backport a patch, it conflicts with the same patch in another branch (e.g. API changes mean things have to be done differently).

At Lancaster, and Moodle HQ, we have one branch per (non-core) plugin too, but you may not need this added complication. It worked well for us, but an upgrade event probably took a day of preparation, planning, and git hyjinx. That said, a day spent doing it well once, is a drop in the ocean compared with dealing with merge conflicts, broken files, broken plugins, etc. if we were to do it wrong. This method makes it much harder to make mistakes.

Another note that's worth mentioning is that you need to be careful with git pull - use git fetch instead. By default, git pull is the same as doing git fetch, followed by an implicit git merge which you do not want. You ideally want to merge a tag, not a branch.

Good luck, and please feel free to ask any more questions or for further clarifications.

I'll try and get David Mudrak to write a blog post now that he's been trying this method for a few months with any tips, tricks, gotchas, etc.

Andrew

Average of ratings: Useful (2)
In reply to Andrew Lyons

Re: Github and Moodle

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I wrote a blog post explaining (at least some parts of) how we do things at the OU: http://tjhunt.blogspot.co.uk/2014/01/moving-ou-moodle-code-to-moodle-261.html

The approach Andrew describes probably requires a bit more work, but is more disciplined, and probably better in the long run.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Github and Moodle

by Michael O'Brien -

Thanks to those who started and contribute to this thread. In the spirit that there is no such thing as a silly question can I ask how db changes are applied into a production system when using a Git repo for the code?

Forgive any incorrect use of terminology but I can see the benefit of using Git and source control even if I don't have the terms correct just yet and not a coder by trade : )

Assuming I've setup my repo and I have production code I now want to run on my web server but there are also db updates that have to be applied

Is the process

  1. Login as admin to the production site
  2. Switch production site into maintenance mode
  3. Backup db, moodledata and config.php
  4. Push the production Git repo code to the web root (or should that be pull, or merge?)
  5. Visit the site and complete the upgrade as per "normal" moodle which would include the db scripts
  6. Exit maintenance mode and enjoy your newly upgraded production moodle

 

Or does using git mean you tackle it a different/better/ less hands on way?

My reason for asking is I'm interested to know how to automate the deployment of moodle including upgrades with as little human intervention as possible outside of actually making the decision to upgrade, that way its documented, repeatable and not prone to human error by skipping a step or 2