Scripts to automate installing and upgrading

Scripts to automate installing and upgrading

by Dashamir Hoxha -
Number of replies: 10

I have been using Moodle for about 1 year and it seems that updates/upgrades are rather frequent. This is great because each update/upgrade fixes some bugs and/or adds some new features. However it makes Moodle maintenance a bit challenging and time consuming. I think that the best thing would be to automate the update/upgrade process somehow (or maybe I am just obsessed with automating everything).

If you have any scripts that you use for this process, could you please share them with me as well? I don't mind if they are specific only for your case. You can sanitize them before sharing (removing any sensitive or personal information, like passwords etc.).

I also found useful these previous discussions:

If you know any other useful discussions about good practices etc. please let me know.

Thanks.

Average of ratings: -
In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Bret Miller -

I don't have a script for installing as I rarely ever install a fresh copy, only duplicate existing Moodle installs. I sanitized some stuff. Use at your own risk. Here you go:

#!/bin/bash
goodtogo=1
issnapshot=0
if [ -z $1 ]; then
	echo 'test or live not specified.'
	goodtogo=0
elif [ $1 == 'live' ]; then
	site='moodle.example.com'
	db='databasename'
	files='moodledatadirectory'
elif [ $1 == 'test' ]; then
	site='testmoodle.example.com'
	db='testdatabasename'
	files='testmoodledatadir'
elif [ -d $1.example.com ]; then
	# For snapshot sites
	site="$1.example.com"
	db="$1database"
	files="moodledata_$1"
	echo "Site: $site"
	echo "DB: $db"
	echo "Data: $files"
	issnapshot=1
else
	echo "$1 is not test or live or a snapshot site name."
	goodtogo=0
fi

if [ $goodtogo -eq 1 ]; then
	if [ -z $2 ]; then
		echo "Updating $site to current release..."
		newbranch=''
	else
		echo "Upgrading $site to MOODLE_$2_STABLE..."
		newbranch="MOODLE_$2_STABLE"
	fi

sqlhost='localhost'
sqluser='databaseuser'
sqlpass='databaseuserspassword'
mydate=$(date +%F)

cd /home/siteuser

echo "Putting site in maintenance mode..."
cd $site
/usr/bin/php5.6 admin/cli/maintenance.php --enable

echo "Purging caches..."
/usr/bin/php5.6 admin/cli/purge_caches.php
cd .. 

if [ "$site" == "moodle.example.com" ]; then
echo "Live site: creating SQL backup of $db..."
mysqldump --add-drop-table -h$sqlhost -u$sqluser -p$sqlpass $db > $db-backup-$mydate.sql

echo "Backing up programs and files..."
tar -cpzf _backup/$site-backup-$mydate.tgz $site $files $db-backup-$mydate.sql
rm $db-backup-$mydate.sql

fi

echo "Backing up modified files in $site..."
cd $site
git config --global user.email "admin@example.com"                                           
git config --global user.name "Admin Name"
git stash

if [ "$newbranch" != "" ]; then
	echo "Switching to $newbranch..."
	git branch --track $newbranch origin/$newbranch
	git checkout $newbranch
fi

echo "Pulling current branch files..."
git pull

echo "Restoring modified $site files..."
git stash apply

echo "Running upgrade..."
/usr/bin/php5.6 admin/cli/upgrade.php --non-interactive

echo "Purging caches again..."
/usr/bin/php5.6 admin/cli/purge_caches.php

echo "Returning site to normal mode..."
/usr/bin/php5.6 admin/cli/maintenance.php --disable

echo "Running cron..."
/usr/bin/php5.6 admin/cli/cron.php
cd ..

echo 'Done.'
echo ''
echo "Don't forget 'git diff' to verify modified files!"

else
	echo ''
	echo 'Use git to update/upgrade Moodle'
	echo ''
	echo 'Syntax: ./upgrade-moodle live|test [upgrade-branch]'
	echo '  upgrade branch like 28 for MOODLE_28_STABLE'
	echo '  no upgrade-branch updates to the current release in the installed branch'
fi


Average of ratings: Useful (4)
In reply to Bret Miller

Re: Scripts to automate installing and upgrading

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Obviously, you need to first set up git on your site to even start looking at this script.  You might find git alone is a good place to start.

In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Ken Task -
Picture of Particularly helpful Moodlers

Please see:

https://sos.tcea.org/upbash.txt

That one script can be used for updating as well as upgrading ... depending upon which lines are active and which lines a commented out (# lines are commented out thus no execution).

This one happens to have a pause in it so one can see the backup files before progressing.

The script could be broken into two ... one for updating and one for upgrading ... and could be executed from /usr/local/bin/ ... IF one adding a cd line to the beginning of the script so that one can execute the admin/cli/ scripts.

'good practice' ... backup before updating or upgrading ... test your backups from time to time on a test server.   The issue with good/usable full site backups would center around moodledata/filedir/.   DB tables related to files needs to match what's in filedir.

Since backing up moodledata might actually be more than you need when updating, one could forgo the backup of moodledata ... could do minimal and backup only moodledata/filedir/

2 cents! ;)

'spirit of sharing', Ken



Average of ratings: Useful (2)
In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Take a backup, run 'git pull', run the upgrade script.  Why do you need an upgrade script for that?

If you are upgrading between major versions and you have loads of plugins then you need to copy and test the upgrade anyway.

In reply to Howard Miller

Re: Scripts to automate installing and upgrading

by Matt T -
^ This. Although, I do use a script myself, but it does this:

1. puts site in maintenance mode

2. Rsyncs Moodledata files to a backup server

2. Backs up the old code dir to a tar.gz

3. Dumps moodle db to a tar gz

4. Runs git pull

5. runs CLI upgrade script

6. takes out of maintenance mode


since making that upgrading is a breeze

Average of ratings: Useful (2)
In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Dashamir Hoxha -

Thanks everybody for your help (especially those that shared their scripts).

I have made an attempt on update (I am still working on an upgrade script). Could you have a look and see if there is any thing that I have missed: update.sh .

It follows the git approach (the code of moodle is installed from git). So, basically it needs just a 'git pull' to get the latest version of the stable branch. In particular, I don't touch any installed plugins at all. I assume that an update does not change any APIs or DB structure, so it cannot break any installed plugins. Is this correct?

Another question: I have made the entire moodle directory owned by www-data (apache user). Is this OK?

In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Ken Task -
Picture of Particularly helpful Moodlers

The cli scripts in moodle code read the config.php file of the moodle + require other libraries for cli commands, thus one has to be in the moodle code directory when the those cli scripts are executed.   Check out the upgrade.php file in moodlecode/admin/cli/ and you'll see the lines that require other moodle code.

Run the cron job before doing anything related to update/upgrade/git.   When I first created the bash shell script I use, cron would not run if in the maintenance mode.

Git will acquire core code only ... unless you installed a plugin via git.   The actual changes to the DB happen only when the upgrade.php is called.   Not sure you need the git stash/git diff as git does show what files it's adding/changing, etc..

Note: you can test the git acquistion of code outside of the production moodle code on the same server or even on a workstation .. I have a Mac with git installed and a MDLTEST directory where I test the acquistion of new code for moodle versions.

cd /pathto/test/code/dir/

git pull

Here's what it shows:

Kens-MacBook-Pro:moodle33 ktask$ git pull
remote: Counting objects: 1164, done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 755 (delta 577), reused 753 (delta 575)
Receiving objects: 100% (755/755), 179.33 KiB | 0 bytes/s, done.
Resolving deltas: 100% (577/577), completed with 137 local objects.
From git://git.moodle.org/moodle
   5012651..2f0bd18  MOODLE_33_STABLE -> origin/MOODLE_33_STABLE
   cc60733..469c46a  MOODLE_32_STABLE -> origin/MOODLE_32_STABLE
   91cbdda..22e99d6  master           -> origin/master
Updating 5012651..2f0bd18
Fast-forward
 admin/tool/mobile/classes/api.php                  |  11 +-
 competency/classes/api.php                         |   3 +-
 competency/tests/api_test.php                      |  15 +-
 install/lang/mi/langconfig.php                     |  33 +++++
 install/lang/oc_lnc/admin.php                      |   6 +-
 install/lang/oc_lnc/error.php                      |   2 +-
 install/lang/oc_lnc/install.php                    |   6 +-
 install/lang/sv_fi/install.php                     |   1 +
 lib/accesslib.php                                  |  35 +++--
 lib/tests/accesslib_test.php                       | 160 +++++++++++++++++++++
 login/signup_form.php                              |   4 +-
 mod/chat/classes/external.php                      |   3 +-
 mod/chat/tests/externallib_test.php                |   7 +-
 mod/folder/edit.php                                |   6 +-
 .../manualgraded/tests/walkthrough_test.php        |  39 +++++
 question/behaviour/rendererbase.php                |   2 -
 search/classes/document.php                        |   2 +
 theme/boost/scss/preset/default.scss               |   1 +
 version.php                                        |   4 +-
 19 files changed, 306 insertions(+), 34 deletions(-)
 create mode 100644 install/lang/mi/langconfig.php
Kens-MacBook-Pro:moodle33 ktask$

The version.php file always shows 4 +- cause only the lines in version.php file that relate to versioning are changed.

And, in my scripts I sometimes add a line:

fgrep '$release' version.php to see the version aquired by git.

Kens-MacBook-Pro:moodle33 ktask$ fgrep '$release' version.php
$release  = '3.3.2+ (Build: 20170922)'; // Human-friendly version name

I also have in some scripts:

Kens-MacBook-Pro:moodle33 ktask$ git branch -a |head -n 1

To assure the master branch is pointed correctly.

I have, in the past, inserted a pause at that point to assure myself git has acquired the version of code desired.

Both an update script and an upgrade script are pretty much like a batch file in Windows and really (IMHO) don't need branching.   Update script without backups won't change.   It's the upgrade script that would require either input to the script and checking that input or editing prior to executing.

Uhhhh ... think one will be updating more often than upgrading.

Me leaving the upgrae scripts in  a state that require editing ... ie, changing the backup file names, and the informational echo statements means I have to see them.   Helps prevent me from 'out-smarting' myself1

Again ... my 2 cents.

'spirit of sharing', Ken



In reply to Ken Task

Re: Scripts to automate installing and upgrading

by Dashamir Hoxha -

> Not sure you need the git stash/git diff as git does show what files it's adding/changing, etc.

If there are any local modifications to the code, 'git pull' will fail to update the code. In this case I backup these local changes (with 'git stash'), and after 'git pull' I restore them (with 'git stash pop'). I must have seen this somewhere, but I don't remember where.

> fgrep '$release' version.php to see the version aquired by git.

In case of update, there is no change in the version/branch, so maybe this is not necessary.

By the way, I clone the git repository from https://github.com/moodle/moodle (instead of git://git.moodle.org/moodle.git): Dockerfile (it seems to me that it is faster). I think that these two are exactly the same and it makes no difference which one we use, isn't it?


In reply to Dashamir Hoxha

Re: Scripts to automate installing and upgrading

by Ken Task -
Picture of Particularly helpful Moodlers

IF you make modificaitons to core code then, yes .... then git stash etc. lines needed ... but if you have committed those to your own repo of Moodle, then are they needed IF you are pulling from your own cloned repo of Moodle?

Reason for checking version.php file is NOT the base version ... obviiously, the x.x won't change ... but the build date will.   Moodle HQ does release new code each week ... many times just fixes ... nothing major ... but worth knowing about/keeping track of me thinks.

Ahhhhsoooo ... we now get info on two new things added to the mix ... you clone the Moodle repo to your own repo and pull from there *and* Docker.

So your stuff is once removed from the source + Docker.   Hmmmm .... guess I;'m just too simple minded and prefer not to add layers that might complicate on top of something that is already complicated enough, me thinks.

Suggest next time you have a question like this one ... include all information.

'spirit of sharing', Ken