Moodle in English: Handy Git tip for tracking 3rd-party modules and plugins

Handy Git tip for tracking 3rd-party modules and plugins

by Mark Johnson - Monday, 24 January 2011, 5:41 PM

I just discovered git submodule and it's so cool I thought I'd share it with everyone in case you hadn't found it yet.

Basically, the submodule command allows you to create a sort of "sub-repository" in a subdirectory of your source tree, with it's own history - the main ("super") repository just tracks the existence of the subdirectory. This means that if there was, for example, a block hosted on github that you wanted to install, you could do so by simply doing:

git submodule add git://github.com/user/moodle-block_foo.git blocks/foo

Which would clone the github repo as a submodule, allowing you to pull changes for just the block with ease. If you've got write access to the cloned repo, you can push changes back too.

Similarly, if you were creating a new module or plugin, you could create an empty github repo add it as a new submodule (or create a local folder outside the super tree and add that, if you're not publishing the code).

More info:

http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
http://book.git-scm.com/5_submodules.html

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Hubert Chathi - Monday, 24 January 2011, 11:12 PM

With git submodules, git doesn't just track the existence of the directory. It also tracks the version of the submodule that you're using. In other words, every time that you update the submodule, git will treat that as a change in the main repository. This makes it unsuitable if you want to track the submodule independantly from the main repository — just check out the module in the blocks/foo directory without using git submodule if you want to do that. This is, however, very useful if you want to keep track of changes to the submodule, or clone your Moodle repository along with the exact version of the submodule.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Mark Johnson - Tuesday, 25 January 2011, 6:23 PM

Hubert,

Sorry, my use of "existence" was incorrect, bad choice of wording.

Also, I didn't realise that you could clone a repo into the subdirectory of an existing one cleanly (i.e. without the parent repository tracking the files in the subdirectory too). Sounds like that's another good solution.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Tuesday, 15 March 2011, 10:43 AM

I'm still wrapping my head around git and would love to see some best practices documentation setup. For example, let's assume that a site wants to use git to keep their code base easy to modify and easy to update. Further, let's assume that they want to use some contrib code in that installation. What is the best way for them to go about doing that? Another scenario are for folks that want to test contrib code. What is the best procedure for pulling in that code and testing it? I think having some best practices around this theme in the Moodle Docs would be very helpful. Any takers? Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Joseph Rézeau - Tuesday, 15 March 2011, 3:49 PM

Yes, Anthony,

As I keep saying in this forum, we (moodle contributors) do need that GIT "best practises" documentation in the Moodle docs.

Joseph

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Tuesday, 15 March 2011, 4:05 PM

You keep getting good advice (e.g. http://moodle.org/mod/forum/discuss.php?d=168094#p740124).

Have you copied that to Moodle docs yet?

I find it slightly ironic that Moodle espouses social constructivist Pedagogy, but what people here want to learn something, just ask for the most backwards sort of pedagogy there is (give me a manual).

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Pierre Pichet - Thursday, 17 March 2011, 4:01 PM

"give me a manual"

As long as you drive an automatic american car for 30 years or so, when you buy a new one you don't need the car manual for not much than using the new gps.

When I got a manual drive Renault when landing in Paris, I was very happy (if not anxious) that they show me how to set for forward and reverse before getting out of the airport to drive for (and in ) Paris and I need the manual for other basic settings as the clim...

Pierre

P.S. As this was my first car with a manual drive in my life , I got also some driving school lessons before taking the plane...

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Mark Johnson - Tuesday, 15 March 2011, 5:40 PM

I've knocked up a few pages for the Moodle Specific Guides on http://docs.moodle.org/en/Git

Certainly not "best practice" yet, but "practice" at least. Please edit/add/delete as necessary.

Average of ratings: Useful (2)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Wednesday, 16 March 2011, 8:57 AM

Thanks Mark, this is a most helpful start. Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Thursday, 17 March 2011, 3:04 AM

I am still playing around with ideas for how best to manage working with contrib code in Moodle and setting up servers to make the most effective use of Git. One issue I ran into was working with my local version of Moodle 2.0.

I started with my cloned 2.0 Moodle install. I created a branch to work on the birthday block - for that I cloned my Github moodle-block_birthday repository into the /blocks/birthday folder.

I then wanted to stop working on the birthday blcok and attend to a request to review someone else's code (in this case the course_menu block). Within Moodle I deleted the birthday block. I did a checkout master on my local copy of moodle and created a new branch for testing course_menu.

I added the new code and wanted to go through and install it. Now the blocks/birthday directory still exists but it was empty (which is what I wanted as I like to isolate the pieces when testing). When I logged in the upgrade screen came up wanting to install the birthday block first and then course_menu. To work around this I simply hid the birthday directory by renaming it to .birthday.

The experience made me wonder whether we should use the existence of the directory as an indicator that there is code to install or whether it makes sense to check for version.php or some other file instead. If we did that, then I would not be prompted to install something that really does not exist.

I'm also wondering if I were to want to manage customizations for some production sites if I should create a separate repository for each one or if I should be able to manage those with branches. The current thing preventing me from using branches for each production site is that one site may have the birthday block installed and the other may not. I'd like to be able to simply switch branches, update the config.php and begin testing the custom code for that site. As it is now, it seems I would either need to have a separate repository for each production site or I would have to hide all of the unused plugins for that particular site. I could easily enough create some scripts to do this but not having to would be all the better.

I'm open to suggestions or other ideas about how I might better manage such scenarios. I appreciate your thoughts, wisdom, and insights. Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Thursday, 17 March 2011, 3:15 AM

I think the most sensible way to handle this is to have a separate git repostory for each plugin.

Then you should able to manage including them in your customised Moodle using the git submodule command. However, this is the bit I have not really experimented with myself. I am just told it is the proper approach. I suggest you read the git submodule man page and experiment a bit.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Thursday, 17 March 2011, 1:04 PM

Thanks Tim - I do have a separate git repository for each plugin but others who have done something similar had seemed to be have a preference to cloning those repositories inside the local clone of the moodle code. I've played a bit with that and for the most part it seems to work reasonably well. I've read a bit about submodule and will experiement with that and wrap my head around the advantages and disadvantages of both approaches tomorrow. Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Thursday, 17 March 2011, 8:00 PM

And since we are talking about how to manage Git and CONTRIB code, do we have or want to establish a repository naming convention for patches. What would folks suggest for the forum approval patch? Perhaps, moodle-patch_mod_forum? I would think the name should easily distinguish between plugins and patches. Looking at http://github.com/moodle we have some repositories called custom-{name} which might indicate that custom-forumpostapproval is the way to go but I think I would like to see moodle in there somewhere to make it clear that it is a moodle customization.

Alternatively, rather than just having the affected code (for example a diff file) or even the modified version of the forum activity module in the repository, I wonder if it would be better to treat patches as branches of a forked moodle repository. That way those that want the diff have it and can merge in the changes to their code. Other less technical folks may simply want the latest zip file that the maintainer has merged with HEAD (i.e. master). As Sam continues working toward the new M&P database, having some greater clarity about how we want to handle patches in Git would be help.

I appreciate any input and suggestions folks may have. Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Thursday, 17 March 2011, 11:18 PM

I think the sensible way to handle changes to existing code is as a branch.

That is, you need to fork https://github.com/moodle/moodle to get https://github.com/mygitstuff/moodle, and create a branch called something like mod_forum-postapproval in there.

Then people who want to add that to their Moodle need to fetch your changes, and merge them into the custom branch.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Anthony Borrow - Friday, 18 March 2011, 2:31 PM

So for the forum post approval patch there is a 1.9 version and a 2.0 version. I'm still wondering waht the best naming convetion would be. For consistency, I like moodle-patch_forumpostapproval but we need to be able to distinguish between the branch for 1.9 and the branch for 2.0. I'm hoping the differences will not be too many but there could be some with the API changes. In any case, I'm leaning toward moodle19-patch_forumpostapproval and moodle20-patch_forumpostapproval. Peace - Anthony

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Hubert Chathi - Friday, 18 March 2011, 9:24 PM

For situations where we have both a 1.9 and a 2.0 version, we use the same repository, but different branches in that repository (we generally use the branch names MOODLE_19_STABLE and MOODLE_20_STABLE).

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Hubert Chathi - Thursday, 17 March 2011, 10:00 PM

In my post above, I explain when using git submodules may or may not be appropriate. Fortunately, the same repository structure (i.e. one git repository per plugin) supports both methods of installing plugins (either via git submodules, or just cloning the plugin repository in a directory inside your main Moodle install).

We have cloned parts of contrib into our own git repository, plus our own custom plugins, using one repository per plugin (except for patches, or plugins that have more than one component), and it has been working fairly well for us.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by David Monllaó - Friday, 4 November 2011, 1:11 AM

Hi,

About git submodule for 3rd party plugins, I've found a problem working with submodules on different local branches of the main moodle repository (unable to delete the submodule dirs when changing from one branch to another) it seems to be a git submodule implementation issue (link) Has someone experienced the same problem and/or have a solution?

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Friday, 4 November 2011, 2:29 AM

Actually, don't use submodules.

Just clone the plugin into the appropriate folder, and then add that folder to the .git/info/exclude file of the top-level moodle check-out.

Then, you can independantly update moodle or the plugin.

Average of ratings: Useful (1)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Joseph Rézeau - Saturday, 5 November 2011, 5:35 AM

I agree with Tim that using submodules is not the way to go. I just clone the 3rd-party plugins in their appropriate folders and that works fine.

However, I do not understand the need for adding those folders containing 3rd-party modules to the .git/info/exclude file of the top-level moodle check-out. I don't do it and it works fine without.

Joseph

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Saturday, 5 November 2011, 5:58 AM

If you type 'git status' in the top level moodle checkout, then you will see all the plugin folders listed as 'Untracked files'. Which is messy.

More seriously, if you are actively doing development in that Moodle checkout, and if you have those folders sititng there as untracked files, then 'git add .' followed by 'git commit' is a nasty mistake. Adding those folder names to git ingnore avoids that.

Average of ratings: Useful (3)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Gareth J Barnard - Sunday, 6 November 2011, 1:31 AM

Hi Joseph,

If you are doing 3rd party development and wish to keep up with moodle core development, then submodules are required as you wish to have your own code tracked in its own right.

I've found that I need to exclude them in the 'exclude' file as Tim states. Also a file at the top called '.gitmodules' is created which needs to be excluded and unstaged. Then a 'git reset HEAD' on the submodule folders to unstage them.

I'm using TortoiseGit but had to do the last bit on the command line.

But once you have done that effort it makes it really easy to keep up with developments in GitHub.

Cheers,

Gareth

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Joseph Rézeau - Sunday, 6 November 2011, 11:00 PM

@Gareth,

I also use TortoiseGit which I find very user-friendly, but I do have to resort to the command line from time to time (as a last resort ).

I do 3rd party development and also have forked versions of moodle on my github. On my local moodle installations (1.9, 2.0, 2.1 and master) I have in the same folders both the moodle core files and my own 3rd-party plugins (and other people's 3rd-party plugins). I do not use "submodules" (although I'm not sure what this means), I do not need to "exclude" anything and everything works fine so far.

Actually, in my local moodle folders, any 3rd-party plugin is located within its own folder and has its own .git subfolder. Those 3rd-party git folders work totally independent from the main git moodle core repository.

This may not be an "orthodox" way of doing things but it does work.

Joseph

PS.- Attached example of a folder showing the .git for moodle core files and the .git for one of my 3rd-party plugins, the glossary_export_to_quiz block.

Average of ratings: Useful (1)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Gareth J Barnard - Friday, 11 November 2011, 4:34 AM

@Joseph,

Sorry for not responding sooner. Thank you for your lengthy post. I have a similar setup but find I have issues with the main moodle because it thinks the new directories should be a part of it. Which is why I went down the sub-module and exclusion route. However I have not done the pure setup from scratch as you have demonstrated from the outset. I just read up about sub-modules when I wanted to do that setup when I did not know so much about Git as I do now.

When I get a chance, I'll try , cheers,

Gareth

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by David Monllaó - Monday, 7 November 2011, 8:22 PM

Hi Tim,

We are a development team working against a dev server which contains the git clones of moodle and other 3rd party plugins, if we ignore the 3rd party plugins inside the moodle repo. each developer must clone each of the 3rd pty plugins because we are all working against the same core + plugins base, so I'll ignore just the .git folder of the 3rd party plugins.

Thanks for your comments

Average of ratings: Useful (1)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Tuesday, 8 November 2011, 1:14 AM

Your scenario is exactly what git submodule was invented for.

If you are not planning to make other changes to Moodle core, then I think not using submodule is easier.

Average of ratings: Useful (1)

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Gareth J Barnard - Friday, 11 November 2011, 4:39 AM

So Tim, in essence, sub-modules are there for when you wish to add addtional functionality and then 'stage it' for others to refer to your particular 'clone' without having to manually add in the sub-modules too. And exclusion (but not using sub-modules) is there for when you wish to do 3rd party development on your own and keep up-to-date with Moodle core but not actually change it?

Cheers,

Gareth

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Tim Hunt - Friday, 11 November 2011, 5:14 AM

Yes. No. Perhaps.

Like everything with git. There are lots of ways you could do it, and they all probably 'work' unless you screw it up, so it is really a matter of how you want to do things.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Gareth J Barnard - Friday, 11 November 2011, 5:29 AM

@Tim,

Ok, thanks - umm, I need the situation where I track a stable version of Moodle core from GitHub but do not change it (but 'pull' on a weekly basis) and then have my plugins in their correct location refering to their own git and GitHub. What would you suggest and should this be in the CONTRIB developer docs? For which I would be willing to contribute in writing.

Cheers,

Gareth

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Dan Poltawski - Friday, 11 November 2011, 8:15 AM

I particularly dislike submodules (having previously used them in production for a few years) because it provides no protection from having a submodule which is not present or up to date and I have be bitten by this many times because it makes it so easy to do (which actually I think is uncharacteristic for git).

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Hubert Chathi - Monday, 14 November 2011, 10:09 PM

Submodules have their uses, but they are definitely one of the trickiest and hardest to understand things in git. (Maybe worse than merge vs. rebase.) It doesn't seem to be as polished as other parts of git -- I suspect because they it is used so infrequently.

Average of ratings: -

Re: Handy Git tip for tracking 3rd-party modules and plugins

by Gilles-Philippe Leblanc - Wednesday, 4 June 2014, 3:38 AM

Sorry, i'm late in the conversation but what do you use instead of the submodule to handle third party plugin in your production server ? merge them in your moodle branch ?

Average of ratings: -