Best way to use Git

Best way to use Git

by Kieran Briggs -
Number of replies: 12

Hi.

I'm the sole developer on our Moodle VLE and currently use git to keep my local development and production server in sync.  I forked the moodle repo and then created a branch "MyPD" and in that branch, i've added a number of plugins, my theme etc.  I've not changed any core code.  I then push this back up to the remote and when I update it I just pull the lastest version of my branch down to the production server.

This has worked fine while I'm not trying to upgrade the version of Moodle, but can someone tell me how I could go about upgrading the version of Moodle in this situation?  The version is currently 3.5.2 but I want to move to 3.7 . If I just check out the 3.7 branch I'll lose all the plugins that i've installed.

Thanks for any help.

Kieran

Average of ratings: -
In reply to Kieran Briggs

Re: Best way to use Git

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Hi Kieran,

We have a similar situation with a lot of different customers whose code we manage.

There are lots of different ways to handle this situation (so I'm sure others will weigh-in with alternatives), but the way we handle it is as follows:

  • For minor releases (3.5.2 => 3.5.3, etc.):
    •  we checkout the latest Moodle code (MOODLE_35_STABLE)
    • then checkout our branch
    • then create a new branch based on our current branch (all our branches are named m_3.5.2_customer, in order to keep track of our exact version)
    • then use git rebase to move our new branch on top of the latest MOODLE_35_STABLE
    • deploy this new branch
    • Note, an alternative is to simply rebase the current branch on top of the latest MOODLE_35_STABLE, but after doing a rebase you have to do a push --force, which can cause issues if other people were working on that branch
  • For major releases (3.5 => 3.7):
    • Create a new branch based off the Moodle code (MOODLE_37_STABLE)
    • Download the latest versions of all plugins and add them to the code (many of the plugins will have newer versions released, so it is easier to install them all fresh, rather than trying to go through and add the updated versions)
    • Commit each plugin as a separate commit (makes it a lot easier to review the list of custom plugins)
    • Cherry-pick any core changes + custom plugins (sometimes taking an opportunity to squash the commits if there are a lot of minor bug fixes we don't need to retain the history of).

Depending on how many plugins there are to install, the major upgrade prep rarely takes more than 30-60 minutes for a customer.

I hope that helps.

Average of ratings:Useful (2)
In reply to Kieran Briggs

Re: Best way to use Git

by Andreas Grabs -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Translators

Hi Kieran,

as Davo already said there are a lot of ways you can hold your plugins and moodle uptodate.

One way could be that:

  1. Create a repository where you put all your plugins and store it as a branch suitable to your moodle version.
  2. Create a repository with your moodle and as you already have done create a branch of your own suitable to your moodle version.
  3. Merge the repository from 1) into the repository from 2)

As commandline it should look like this.

# creating the plugin repository
git init
# Create a branch for your moodle version e.g. 3.7
git checkout -b mdl_37
git remote add origin <your-git-url>/myplugins.git
# Put your first plugin into this repository. Do this in the correct directory, as it should be later.
git add .
git commit -m "added plugin xyz"
# Do the same for all plugins
# push the repository to the git server
git push origin mdl_37

# clone moodle in another repository
git clone -b MOODLE_37_STABLE https://github.com/moodle/moodle.git moodle
# create your own branch
git checkout -b mdl_37
# Add the first repository as remote and fetch it
git remote add myplugins <your-git-url>/myplugins.git
git fetch myplugins
# merge the plugins
git merge --allow-unrelated-histories myplugins/mdl_37
# Push it to the git server
git push origin mdl_37

This way, you can update your plugins and your Moodle version separately and always see which plugins you use.

Maybe that is a way for you too.

Best regards
Andreas

Average of ratings:Useful (1)
In reply to Andreas Grabs

Re: Best way to use Git

by Kieran Briggs -
Thanks both of you. These make so much more sense that some of the documentation i've been trying to read.
In reply to Kieran Briggs

Re: Best way to use Git

by Mark Sharp -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

We keep the core moodle repo and our changes separate and merge them together using rsync. It helps make sure we don't accidentally make changes to core moodle.

So our structure & process is something like:

~/moodle (moodle repo, or download)
~/pluginsthemesetc (same directory structure as Moodle)
$ rsync -avz ~/moodle/ /var/www/html/
$ rsync -avz ~/pluginsthemesetc/ /var/www/html/
$ php /var/www/html/upgrade.php

I use bash scripts to manage the process.

If I absolutely have to change anything in core, then I include and run patch files, but I keep these to an absolute minimum. I have one patch file for a community plugin atm, and that's it.

In reply to Kieran Briggs

Re: Best way to use Git

by David Mudrák -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators

You might be able to "replant" your MyPD branch onto 3.7 with the git rebase --onto command.

Basically something like

# git rebase --onto MOODLE_37_STABLE MOODLE_35_STABLE MyPD

which does something like "take all commits on the MyPD branch that are not present on the MOODLE_35_STABLE branch and re-apply them on top of the MOODLE_37_STABLE branch". Read the fine manual for more details.

Like others already mentioned above, there are many creative ways you can use Git to compose code for your customized Moodle sites. In HQ we ended up with maintaining a separate branch for every single plugin or core modification. We follow a strict naming scheme for these branches which allowed us to have helper scripts for common tasks such as "take that one particular core hack we used on moodle.org and re-use it on another site" etc.

In reply to Kieran Briggs

Re: Best way to use Git

by Colin Perepelken -
Aside from the what the others have mentioned, if you aren't modifying core code, I find using submodules to manage plugins works very well. https://git-scm.com/book/en/v2/Git-Tools-Submodules

To add a submodule to a repository, use:

git submodule add [source_url] mod/myplugin

You can update all submodules at once using:

git submodule update

It is very easy to keep the plugins and core Moodle code separate this way, and could make upgrading simpler in the future.

In reply to Colin Perepelken

Re: Best way to use Git

by Kieran Briggs -
I was thinking about Submodules. If I clone a branch with submodules will it bring the modules with it too. So for example, as I said, I have a localhost development machine and then the production server is where I just use a git pull to update. If I update the submodules on the remote will it pull them down on a git pull onto the production server?
In reply to Kieran Briggs

Re: Best way to use Git

by Colin Perepelken -
Make sure you add your submodules to the main repo (at the Moodle root). Then when you want to deploy to production for the first time, you would:
1. Git clone / git pull your master (moodle root) repository.
2. Run git submodule update --init

To answer your question, yes for any subsequent changes you can just push / pull the master repo and the submodule changes will be pulled down as well. However you will have to make sure these submodule changes have been committed to the master repo as well as to the submodule, so Git knows which submodule commit to pull. Submodules will be checked out to a specific commit, rather than a branch.
Average of ratings:Useful (1)
In reply to Kieran Briggs

Re: Best way to use Git

by Andreas Grabs -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Translators

Hi Kieran,

there are a few things you keep in mind when using submodules.

  1. You never know whether or not the submodule is available while updating your git on the production mashine
  2. You never know whether or not the git origin of your submodule is consistent.

So if you really want to use submodules in a production environment, you'll need to create your own repositories for all your plugins and then use those repositories as submodule in your main repository. Otherwise you might end up in a broken productive system.

Best regards
Andreas

Average of ratings:Useful (2)
In reply to Andreas Grabs

Re: Best way to use Git

by Brian Merritt -
Picture of Particularly helpful Moodlers
Good points all.

We use submodules and generally clone repositories only where we have to make changes (which we try not to do) or where the plugin is not receiving love and care.

Each new Moodle major release just requires adding the submodules (which can be done in php by reading the .gitmodules from the previous release and running a git submodule add script to pull them in)

We use php deployer https://deployer.org/ for deployment, with branches MASTER (production quality test), STABLE (production quality deploy to live) and FEATURE (development quality).

Having a staging server is essential as you can run a test deploy and make sure it works before going live.

Because plugins can be nested, we tend to use often git submodule update --init --recursive

A few other things to keep in mind re submodules

1. when git adds the submodule the branch may be wrong. make sure you checkout the right branch and commit that
2. if changing the branch (maybe the plugin has a MOODLE_37_MASTER branch) so check releases whether they have a more appropriate branch available
3. the git commit of the main Moodle sets the submodule to a specific commit. If the remote plugin is updated you need to do a git pull of the submodule and commit that change back to Moodle main.
In reply to Brian Merritt

Re: Best way to use Git

by Brian Merritt -
Picture of Particularly helpful Moodlers
Oh yeah - each Moodle version we keep moving forward, so Moodle master get's branched off when 3.8 goes into development, so we have Moodle 3.7, 3.8 and master moving forward over time (which then becomes 3.8, 3.9, and master). It takes a bit of time, but makes security fixes and future development testing much simpler.
In reply to Kieran Briggs

Re: Best way to use Git

by Pawan Pandey -
Hi there!

I think in your architecture you have to utilize Git Submodules and Subtree. Submodule can be used for external plugins in which you are performing you changes,
whereas Subtree can be utilized for managing dependencies which doesnt include changes by you.

For that you have to find source control url for all external plugins, clone all those plugins in separate plugins, using tools like Sourcetree which helps you to make the state of plugins as same in your Moodle 3.5.2 , use code comparison tools like Winmerge to find plugins which are changed by you. Create submodules for those with changes, and subtree for those without changes.

For moving from One Major Upgrade to Another, what i can see their is you can write some scripts to intialize the submodules and subtree in the new system. For core changes, i will recommend try your best to keep them at minimum. Yeah you can cheery pick to move those core changes, but i am not sure whether that will be robust approach. Better to keep one document, to maintain core changes and perform on new version.


Only problem with above architecuture is that it requires lot of Git skills, and in case of multiple people working together

Moving core changes wont be easy. The best one is stick to Moodle LTS version currenlty 3.5 , then wait till May 2020 for latest Moodle 3.9 version so that you dont have to do such much operations frequently.