General developer forum

Best practices for code modification?

 
Emperor Joshua Norton I, Emperor of the United States and Protector of Mexico.
Best practices for code modification?
 
We started out with Moodle 1.5.2, which is a good, solid product and perfect for our needs. Of course, nothing is perfect out of the box and I ended up doing a lot of heavy customization to sort of pound Moodle as a round peg into the square hole of our business requirements.

Unfortunately, though, this has come back to bite me in the rear. Hearing Martin speak at the Moodle Moot has convinced me that we should move to Moodle 2 when that becomes available, but before that we need to move to Moodle 1.9 (we're currently customizing Moodle 1.8 which I may give up on as a lost cause). The customizations I've made have gotten in the way of doing this easily.

So what are the best ways that you've found to get around this situation? Some approaches I'm taking this time around to make the process smoother next time are:

1. Using includes wherever possible instead of large chunks of customized code;
2. Heavy documentation, and commenting out lines of code I'm replacing instead of deleting them outright; and
3. Instead of making changes to pre-existing tables, create additional tables linked via foreign keys to the existing ones. For example, instead of adding the two additional fields to the user table that we need, create a separate table that contains our specific data and simply link that back to mdl_user by the id key.

I can't be the only one in this situation. What are the best ways other people have found for doing this that can minimize the pain in migrating to future versions of Moodle while maintaining customizations?


 
Average of ratings: -
Picture of Samuli Karevaara
Re: Best practices for code modification?
Core developersParticularly helpful MoodlersTranslators
What kind of changes do you have? I started with customizations to 1.3/1.4, and then in the later versions I've tried to "phase out" our core customizations to use the ways that Moodle offers for modifying and extending the behavior. For example, since 1.8 there are core tables for adding custom fields for the user data (http://docs.moodle.org/en/Development:Customisable_user_profiles). Other ways of extending Moodle include:
  • Themes
  • Language packs
  • Blocks
  • Course formats
  • Resource types
  • Activity modules
  • Question types
  • Filters
  • Enrolment plugins
...and so on. Old news, yes, but sometimes you can do wonders with "just" those. There is still the issue of testing and possibly fixing the additions before doing an upgrade. However, it avoids a lot of those version control merges and code conflicts, making the upgrade process slightly less messy.

What are your biggest problems when upgrading? Custom code is lost? Code conflicts with version control? Broken function calls? All of the above?
 
Average of ratings: Useful (1)
Picture of Howard Miller
Re: Best practices for code modification?
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers
Just to reiterate...

My "policy" has always been to use plugins wherever possible and/or to try and code improvements that will be universally useful - your patch gets into the next release!
 
Average of ratings: -
Picture of sam marshall
Re: Best practices for code modification?
Core developersPlugin developersTesters
At the OU we unfortunately have extensive changes to core code.

Unlike Howard, I've found that even 'positive' code changes can be hard to get agreed for inclusion in Moodle core, although we certainly have managed to get in many such changes (most of which were a good idea smile). We do have direct commit access but I generally don't use this without asking unless it's for a straightforward bugfix.

Sometimes the difficulty in getting it included is because what we did was good for us but doesn't really fit the way Moodle works, in which case it's obviously right for moodle.com to reject it; with minor improvements there is sometimes an attitude that 'well this needs to be completely rewritten for [some future version] so don't change anything now'; but most often it's just hard to get agreement because Moodle.com developers are busy, and of course we are too.

For example I have a big improvement to the language system that allows plurals (and other things) to work properly, and I've asked a couple of times but not really got approval - I'm going to need to create more fuss to get that into core so maybe it will never go in (even though I spent two days coding it, along with significant time discussing it in forums). This isn't even in our local version either - I don't want to put it in there precisely because of the difficulties it will cause in merging future updates, as in this thread - so it might have been a complete waste of my time. MDL-12433 if anyone's interested smile

Anyhow, sob story over - in addition to these problems, we have many code changes which are not actually a good idea, just a quick hack to solve a local problem. So I certainly wouldn't want to get those changes into official Moodle.

I agree that wherever possible all changes should be done within the plugin framework (modules, blocks, course formats). This will minimise problems. Other code that doesn't fit into this should go in the 'local' folder where possible (which is where local database changes should live). Language changes should all go in a _local folder or in dataroot, with absolutely no changes to en_utf8. For our own custom plugins, we always keep language files with the plugin (inside the plugin folder) rather than in separate lang folders as this makes them easier to manage. Of course we have our own theme (child of 'standard') to do all CSS changes so we generally don't touch standard theme.

When we do need to change core code, we do so as follows:

// ou-specific begins
...
// ou-specific ends

If we were replacing some existing code rather than just adding it, we often (but not always, hm) leave in the original code commented-out.

We also use

// ou-specific begins (until 2.0)
...
// ou-specific ends

if we have code which IS going into moodle for the next version, so we know that the ou-specific section should be overwritten then (but not for a 1.9.x point release).

Or

// ou-specific begins (until MDL-12345)
...
// ou-specific ends

That's if we have a patch in the mentioned bug number which does what we want, but we're waiting to see whether anyone will agree we can include it in moodle.

We also use the same words but with the relevant comment indicators if we need to make changes in other types of file that don't support // comments.

Merging a new version is done by hand. For folders and files that don't contain 'ou-specific' we just overwrite the whole thing with the new version from core. When a file does contain that string, the person who merges code sorts it out manually, making changes to our customisation if necessary to keep it working, and possibly discussing it with whoever wrote the change.

--sam
 
Average of ratings: Useful (1)
Picture of Howard Miller
Re: Best practices for code modification?
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers
I didn't say it was *easy* to get things into core. In fact it gets harder all the time - rightly probably tongueout

I think it's a reasonable aspiration though. The flow of thought often goes...

1. I've had a FANTASTIC idea for a new feature
2. Post proposal
3. Response = that's a really stupid idea, have you thought about...
4. Good point, go to plan B

OR

4. I really have to have this. So reformulate plan OR as a last resort do a local hack.

Even then my view is that if there is a way to do it as a plugin use that route. For example, don't modify an existing question type, take a copy and make a new type. Less pain in the end (sometimes!).
 
Average of ratings: -
Anthony Borrow
Re: Best practices for code modification?
Core developersMoodle HQPlugin developersPlugins guardiansTesters
Sam - I think your description of documenting the code is good as I used a similar process. In addition, I opted to create diff files which I store in a directory tree that mirrors the moodle code so that I can easily recreate the file should I need to. Peace - Anthony
 
Average of ratings: -
Martin Langhoff - Sailing
Re: Best practices for code modification?
 

IME, if you are using a decent SCM, it is a bad idea to add "this code is mine" comments.

There are reasons for this:

A good SCM will tell you immediately, effortlessly what files are touched ("dirty") on your branch, even if you've merged repeatedly. If you keep "this code is mine" comments around, you must remove them when you put them upstream -- even if the code does not change. By removing them, you are merging a different patch upstream, and you will get a conflict next time you merge. A trivial conflict which will consume time to resolve where it could have been auto-resolved. If in the same area of code you have other changes that you aren't putting upstream, the conflict becomes more complex to resolve.

On the other hand, if you have the patches "clean", then you can merge them upstream unchanged, and that good SCM of yours will realise, and automerge, not giving you any grief at all. Go home early, relax.

Life is too short - WAY too short - to be spent merging with bad tools. I used CVS for keeping local branches way too long because there weren't any viable alternatives. I am sure I lost a few years of life expectancy because of those horrid merges :-[

 
Average of ratings: -
Picture of sam marshall
Re: Best practices for code modification?
Core developersPlugin developersTesters
Yes, but Martin, what your message doesn't say is that this is just a big plug for Git (the 'good SCM system' you mention), or one or two other obscure systems that also provide these facilities. smile

Like most people we use CVS (and I don't think SVN is much better in this respect) and have no very clever way of merging core/local changes, which means these marker comments are invaluable.

We do not generally have problems with the situation you described (same patch goes to core) because after checking in a patch to core, we remove any such ou-specific tags from our own code [we know that if we later merge with 1.9.2, it's ok to replace with the core version]. Where we do have a problem is with having to manually merge with core in the first place! I agree it's a total waste of time and we would do better to use a 'good' version control system - but Git's interfaces with real-world use are still not finished (finished support for neither Eclipse nor standard Windows GUI; unfinished projects for some of these appear possibly stalled; not sure if there is any support at all for other IDEs such as NetBeans, or for OS X GUI for instance).

Basically you are right that using Git (or similar, but probably Git) is obviously the future, and that cvs/svn cannot really cope with two moving targets (local moodle changes + core moodle changes). But I also think that anyone who isn't extremely knowledgeable about version control is going to have a lot of trouble getting a Git system set up the way they want, and some inconvenience (lack ot tool/GUI support) once they've set it up.

I do hope that the OU will move to Git for our Moodle development at some time in the relatively near future but I also hope we get a fully-working (or at least nearly-fully-working) Eclipse plugin for it first! Until then let's continue giving horrible merge jobs two or three times a year to somebody who isn't me. smile

(I doubt we have spare money to fund that development unfortunately... seems like a pretty desperate need.)

--sam
 
Average of ratings: -
Emperor Joshua Norton I, Emperor of the United States and Protector of Mexico.
Re: Best practices for code modification?
 
To be honest, aside from downloading Moodle via CVS and using an SVN repository to track my writing at home, I'm not all that familiar with SCM in general, and with Git in particular, even though I have it installed on my computer. In particular, I'm not at all certain how I'd go about using the two tools to keep my Moodle installation up to date as well as keeping my own changes in place.

My preferred IDE is Quanta. Eclipse, since it's written in Java, is way too slow on my clunky old work machine to make it particularly useful.
 
Average of ratings: -
Picture of sam marshall
Re: Best practices for code modification?
Core developersPlugin developersTesters
Quanta doesn't appear to have git support either as far as I could tell - just CVS and (with plugin) SVN.

By the way, Eclipse performance and memory usage is pretty good - er, relatively - if you are just using it as a Java IDE. Unfortunately the PHP extension seems to be pretty poor; it does swallow a completely infeasible quantity of RAM (and lots of cpu too). For Java development eclipse used to be, if not ideal, then at least perfectly usable on my old AMD Duron 1.6 with 256MB RAM...

--sam

 
Average of ratings: -
ME
Re: Best practices for code modification?
Core developers
External to the actual modification of the code a good source code versioning system can go a long way to assisting in migration tasks.

Having gone through a 1.6 to 1.8 upgrade at HSU and now 1.8 to 1.9 upgrades for clients at Moodlerooms the svn repository allows for diffing of files, merging of code and more importantly reversion of code if a mistake is made.


 
Average of ratings: -
Robert Allerstorfer
Re: Best practices for code modification?
 
I am also asking me sometimes why Moodle yet did not migrate from CVS to the much better SVN versioning system.
 
Average of ratings: -
Emperor Joshua Norton I, Emperor of the United States and Protector of Mexico.
Re: Best practices for code modification?
 
I think I'm in agreement with you there. I switched over from CVS to SVN a few years back for my own projects, and I've never looked back.
 
Average of ratings: -
Martin Langhoff - Sailing
Re: Best practices for code modification?
 
Because SVN isn't a particularly interesting SCM, specially when you need local and feature branches. Search this forum for GIT and DSCM for more in-depth discussion of the reasons why.
 
Average of ratings: -
Picture of Nate Baxley
Re: Best practices for code modification?
Plugin developers
I'm new to both Moodle and CVS/versioning systems. We have downloaded the 1.9 Moodle using the CVS commands and set it up in a clean install. We've made a few changes to some themes that we copied over from our 1.8 install. I will soon want to bring those changes, plus all of the 1.8->1.9 changes in to upgrade our current 1.8 install which was setup without using CVS.

What is the best way to do this? Since 1.8 was setup with out using CVS, can we download from the central CVS and merge it's changes with our current 1.8?

We plan on making future changes, and may even submit some things to contrib. We want to implement some kind of versioning. If we keep track of our own changes using CVS, will we need two CVS folders in each install? How do you track your own changes and still be able to merge down new versions from Moodle?

Finally, is CVS the best answer for our local versioning? we run 4 different moodle instances, and so I need to track changes in each of them.

Thanks for any help you can give a newbie.

Nate Baxley
 
Average of ratings: -
Picture of Ralf Seliger
Re: Best practices for code modification?
 
I had a similar problem to yours a couple of weeks ago, namely upgrading from a rather heavily modified 1.8.2 to 1.9.

In anticipation of this, all modifications to the original 1.8.2 had been marked by identifyable comments, so I was able to port my modifications manually to the 1.9-code, by searching for the comments.

I do not delete code, but rather use comments to deactivate it. Thus, if necessary, everything can be returned to its original state without too much of a hassle even without source control.

To stay up to date, I CVS update the development system as well as the production system once a week to MOODLE_WEEKLY.

The development system source code, however, is version controlled by Subversion. This approach has several advantages:
  • The CVS update is not hampered by the .SVN-Folders
  • Subversion can be told to ignore the CVS-folders when I commit my changes to my Subversion repository.
  • There is no need to distinguish between different CVS-folders
So far this approach works to my complete satisfaction. All CVS update conflicts due to my changes could be resolved easily.

Regards
Ralf Seliger


 
Average of ratings: -
Picture of Samuli Karevaara
Re: Best practices for code modification?
Core developersParticularly helpful MoodlersTranslators
Have you changed other things than the themes too? If not, you are probably just fine without CVS. If you want to change a standard theme, make your own theme to be based on that standard theme (see http://docs.moodle.org/en/Themes for more). Then you can just update Moodle, and your theme will stay as it is (in a folder that is not touched by the Moodle upgrade). As it is based on standard themes, those will be updated. You might have to fix some of the theme settings after the upgrade, if the HTML has changed etc.
 
Average of ratings: -
Picture of Nate Baxley
Re: Best practices for code modification?
Plugin developers
Well, we plan to make changes to other areas eventually. My concern is that I need to track changes to our local version, some of which don't make sense to include in the official release. Do I need a second, local CVS?
 
Average of ratings: -
Picture of sam marshall
Re: Best practices for code modification?
Core developersPlugin developersTesters
Yes you should track local changes, either using a local version control system to store the whole thing or possibly by maintaining a set of patch files in such a way that they can be automatically applied. (So whenever you update Moodle, you can just run some script to automatically apply all your patches again, and find out if any of them fail so that you need to alter them. Obviously you would do this on a development server not directly on the live system, then transfer it to the live system once you've fixed all your patches.)

This is the sort of thing git could be useful for automating so if you really feel like getting a headache, now's the time to start learning! I'll let the git advocates tell you how though, since I don't know...

--sam


 
Average of ratings: -
Emperor Joshua Norton I, Emperor of the United States and Protector of Mexico.
Re: Best practices for code modification?
 
Thanks to everyone who's replied! I've learned through this forum and through conversations at the MoodleMoot in SF that there are much better ways of doing things than I had been doing. I wasn't looking closely enough at the plugins, blocks, and extensions as I should have been (Mike Churchward's talk made me realize the importance of this), and I hadn't even considered the value of using CVS to keep my code in check.

We do have some ways of doing things that I doubt would make it in to Moodle core. For example we want to make sure our students can't get into a course at the end of a certain period of time, but unenrolling them is not sufficient, since we still need their scores and work retained in the database. Plus, we need to prevent students from seeing anything past the first lesson until they click through on an agreement page (though I suspect that can be taken care of in Moodle 2.0 with conditional activities).

My priority at this point will be to minimize or local hack impact so that future upgrades will go much more smoothly. I'm very excited about Moodle 2.0 because I think it will meet some of our own needs better than previous versions.
 
Average of ratings: -
Martin Langhoff - Sailing
Re: Best practices for code modification?
 
The team that has been doing this -- making significant changes to moodle, most of which get integrated later -- for the longest time is the Catalyst team. I have posted a lot here about using DSCMs such as GIT for this.

You want to avoid CVS and SVN like the plague for this kind of work. Google for GIT and DSCM on this forum, I've fleshed out the reasons, and the workflow a few times here wink
 
Average of ratings: -
Emperor Joshua Norton I, Emperor of the United States and Protector of Mexico.
Re: Best practices for code modification?
 
Ah! Great! Thanks, Martin. I'll take a look for some of your posts on this topic.
 
Average of ratings: -
Robert Allerstorfer
Re: Best practices for code modification?
 
Martín,

if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs.
 
Average of ratings: -
Dan at desk in Moodle HQ, Perth
Re: Best practices for code modification?
Core developersMoodle Course Creator Certificate holdersMoodle HQPlugin developersTesters

if GIT is preferred over CVS, it would be very useful to mention this secret on the Wiki or elsewhere in the docs


That is a bit contentious point, synonymous with saying that vim is the preffered editor of Moodle evil.

The community is large and has many different views and I don't think you could generalise like this. Having said that, some of the arguments in favour of this discussion is in MoodleDocs and I think many people can see the the decentralised light tongueout
 
Average of ratings: -
Picture of Iñaki Arenaza
Re: Best practices for code modification?
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

You want to avoid CVS and SVN like the plague for this kind of work.

Disclaimer: I use git to track my changes, and I really think it's a better SCM than CVS or SVN.

But I think we need to put Martin Langhoff's assertion into perspective and qualifiy it a bit more.

Most people don't make local changes that later get into core. They just need to track their local changes, so the features Martin L. needs more are 'irrelevant' to those people.

So the only competitive egde of git over CVS or SVN in that particular scenario is the ability to do repetitive merges without having to deal with tags/revision-numbers to remember the last merge point, and the better merge algorithms it uses (I don't take merge speed into account because most people don't do dozens of merges in a day).

Which IMHO is a big plus, but on the other hand you have some 'drawbacks':

  • No IDE/Windows GUI integration (as Sam Marshall likes to point out evil)
  • A steeper learning curve (which later more than pays for itself wink)

Saludos. Iñaki.

 
Average of ratings: -
Picture of sam marshall
Re: Best practices for code modification?
Core developersPlugin developersTesters
Just to point out a different disadvantage of using bleeding-edge software so that I don't get accused of saying the same thing over and over (OH WAIT TOO LATE)... smile
  • Harder to get peer support.
There are millions of people across the world (and about four within my line-of -sight, if I turn my chair slightly) who are knowledgeable-to-expert level in CVS, which means that if I or somebody else here gets confused with something, it's easy to find a person to ask. The numbers for git are 'much much fewer' and '0', respectively. smile

Basically I think git is probably great and I hope it does gain critical mass but at present I still doubt that it's an appropriate recommendation for anyone who isn't highly comfortable with Unix command-line tools. (Even if you can get the half-finished GUI tools to work, you'll probably still need to use commandline frequently, would be my guess.)

By the way if/when git actually does gain that wider support, it may even have a shallower learning curve than CVS because I presume it doesn't have the ridiculous gotchas - binary/ascii file distinctions come to mind here...

--sam
 
Average of ratings: Useful (1)