Upgrading to 1.6 and may need to switch to PostgreSQL

Upgrading to 1.6 and may need to switch to PostgreSQL

by Andrew Steele -
Number of replies: 19
We are planning to upgrade the various Moodle installations we manage to v1.6 but, currently, our server doesn't support the required MySQL.

Reading the documentation there is an indication that we could use, instead, PostgreSQL.  We do have that installed on the server.

Has anyone any experience making such a switch and any idea of what we would need to do to make sure it all works?  smile

Whilst typing, our ISP has advised they can offer an update to MySQL 5.0 - so as an alternative can anyone advise if that will work with Moodle 1.6?

Andrew
Average of ratings: -
In reply to Andrew Steele

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
Has anyone any experience making such a switch and any idea of what we would need to do to make sure it all works?  smile

We are test-driving it at the moment (we plan to do it on production servers in 7-10 days). We are first switching MySQL and Postgres with our current 1.5.3+ installation, and then upgrading to 1.6+ and converting the whole thing to UTF-8.

There are a few rough edges at the moment (a few differences between the database schemas between MySQL and Postgress for the same Moodle version, a few 1.6 upgrade scripts that use Postgres 8.x specific syntax that choke on our 7.4.x Postgres, etc.) and we are trying to document them all.

It is doable, but it's not click & go sad

On the other hand, MySQL 5.0 will work without problems with 1.6, provided you disable the STRICT_TRANS_TABLES option in MySQL (see the details here: http://docs.moodle.org/en/Installing_Moodle#Creating_a_database )

Saludos. Iñaki.
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Andrew Steele -
Many thanks for the helpful response.  Looks like MySQL 5.0 is the way to go smile

Andrew
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Przemyslaw Stencel -
There are a few rough edges at the moment (...) and we are trying to document them all.

Hi Iñaki,

Have you already made the switch to postgres? Could you share your insight into the process? We're considering the same move (mysql -> postgres) and I'll appreciate any help on doing this.

Cheers,
Przemek
In reply to Przemyslaw Stencel

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
Yes, we did it on mid-July. I don't have my notes at hand
now (they are at work) but this is roughly what we did:

1.- We setup a second moodle installation using the exact
    same version of the code (in fact, we copied the files
    we were using in production). We did a standard install
    from scratch. This makes sure the postgres tables are
    correctly created.

2.- We stopped both the mysql moodle and the postgres
    moodle, to make data static.

3.- We dumped mysql database with the '-c' option (this is
    very important, otherwise postgres wasn't very happy
    with the inserts), using something like:

    mysqldump -c -t moodle > moodle.dump

4.- We stripped all the mdl_log table data, as it was huge
    and made the import into postgres very long, and
    stripped the '`' from the mysql table names (postgres
    doesn't like them), with

    grep '^INSERT INTO ' moodle.dump | grep -v '`mdl_log`' \
      | sed -e 's/`//g' > postgres.dump

5.- We emptied all the postgres tables (so the import
    doesn't duplicate content or stops due to duplicated
    values), but left the tables themselves. We ended up
    using a sql script, as we needed to do this several
    times (see why below).

6.- We connected to postgres as the moodle user an issued
    the following commands:

      \set ON_ERROR_STOP on
      \o /tmp/postgres-logs.txt
      BEGIN;
      \i /path/to/postgres.dump
      COMMIT;

    We set ON_ERROR_STOP to stop processing the dump file as
    soon as it encounters an error. Having a look at
    /tmp/postgres-logs.txt we could see why it was
    stopping.

    We also used transactions (the BEGIN/COMMIT commands) to
    speedup the insertion process quite a bit.

7.- As the table definitions are no 100% equal between mysql
    and postgres (at least they weren't in our 1.5.3+
    version), we had to loop through 5 and 6 several times,
    correcting the offending data in postgres.dump where
    possible, or 'fixing' the table definitions in MySQL to
    make them compatible with the postgres ones, and going
    to step 3 again.

8.- Once we had a successful import (after only a few
    changes), we needed to reset the postgres sequences to
    keep them in sync with real data. Patrick Li wrote the
    following one-liner
    (http://moodle.org/mod/forum/discuss.php?d=40836#193749):

    PGPASSWORD={password} psql -qAt -U {username} -c "\ds" {database} \
      | cut -d '|' -sf 2 \
      | sed "s/^\(\(.*\)_id_seq\)$/SELECT setval\('\1', \(SELECT MAX\(id\) FROM \2\), true\);/" \
      | PGPASSWORD={password} psql -U {username} {database}

    Replace {username}, {password} and {database} with your
    own ones.

As far as I remember, that was more or less what we did.

Saludos. Iñaki.
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Marcin Wojciechowski -

We've got the following error while procedding with dump insert into pgsql database:

moodle=#    \set ON_ERROR_STOP on
moodle=#       \o /tmp/postgres-logs.txt
moodle=# \i /root/postgres.dump
psql:/root/postgres.dump:2: ERROR:  syntax error at or near "," at character 126
psql:/root/postgres.dump:2: LINE 1: ...) VALUES (1,'activity_modules',2004041001,0,0,1,0),(2,'admin...
psql:/root/postgres.dump:2:                                                              ^

 

do you have by any chance shell script to fix this error in mysql dump file?

In reply to Marcin Wojciechowski

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
Not that I remember of. But from the piece of text you paste, I guess you dumped your MySQL database with the '-e' option (extended insert), which produces INSERT statements that are not compatible with PostgreSQL.

Saludos. Iñaki.
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Marcin Wojciechowski -
nope, it was exactly like stated in point 3 of your instrucion, with -c flag (complete-insert).

you wrote "we had to loop through 5 and 6 several times,
correcting the offending data in postgres.dump where
possible, or 'fixing' the table definitions in MySQL to
make them compatible with the postgres ones"

Maybe I should investigate other mysqldump options and try them out?

What kind of fixing was it in your case? Small hand-fixing or some sort of global changes?

Also we're trying to put this dump into pgsql8, not pgsql7 - but IMO it shouldn't cause this kind of errors.

Thanks.
In reply to Marcin Wojciechowski

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers

nope, it was exactly like stated in point 3 of your instrucion, with -c flag (complete-insert)

I don't know how, but your dump is using the extended INSERT format, with multiple tuples per INSERT, instead of the 'simple' format with just one tuple. Maybe your global settings for mysqldump are setting that extra option, or you have a personal .my .cnf file with that option, but I'm pretty sure this is the real root of the problem you are facing (maybe there are others too down the line, but the error message you posted is what makes me think this way).

I would try making mysqldump creating a dump with 'one-line' INSERTs first of all. Then try feeding that to PostgreSQL.

you wrote "we had to loop through 5 and 6 several times, correcting the offending data in postgres.dump where possible, or 'fixing' the table definitions in MySQL to make them compatible with the postgres ones"

Yeah, but these were either things like content having quotes in a way incompatible with PostgreSQL or mismatching table definitions (where MySQL had a field Postgres didn't have --that wasn't used at all in the code, but made the import fail--, or similar things). At least this is what I can remember of.

Also we're trying to put this dump into pgsql8, not pgsql7 - but IMO it shouldn't cause this kind of errors.

I agree with you.

In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Rob Byrd -
Inaki,
You mentioned stripping the log table to save some conversion problems. I plan on doing this upgrade (MySQL 4.1 to PostgreSQL 8.1 and 1.5.3+ to 1.6) between semesters and would be able to strip ALL student-course table data and keeping only student profiles, courses, instructor materials, online quiz banks, etc. Would this be effective, or would finding all the correct tables to strip before mysqldump'ing be more risky than repeating steps 5 and 6 additional times. (Finding and fixing those table incompatibilities seems daunting right now.)

Thank you, Inaki, for you pioneering work in this area.

Rob







In reply to Rob Byrd

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
We stripped the log table not to avoid conversion problems, but because it was very large and it made the conversion take waaaaay longer. We didn't need the logs, as we did the conversion at the end of the academic year, so we simply stripped them smile

As for the table differences between MySQL and Postgres versions, this has improved in 1.6 and 1.7. So maybe you could try the other path: first upgrade to 1.6.x and then switch to Postgres. We didn't have this option (as we had MySQL 4.0.x, which isn't supported in 1.6) so we needed to do switch to Postgres first and upgrade to 1.6 later.

Saludos. Iñaki.
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Rob Byrd -
Thank you I'll try it that way first (upgrade moodle, then migrate data from MySQL to Postgres). I'll let you know (in December) how it went.

Rob
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Michelangelo Sommaruga -

Hi Iñaki,

I was wondering how you set moodle to use postgres 8 in the config.php since the options are Mysql and postgres 7. I have everything set to use postgres 8 but I can't connect. I already try in a command way and it works, but not with moodle.

I'm installing  Moodle 1.7.1+

Gracias de antemano,

Michel

In reply to Michelangelo Sommaruga

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
postgres7 is the generic database driver name in Moodle for both Postgres 7.x and 8.x (just don't ask me why, as I don't know wink). Just make sure you have Postgres support in PHP enabled.

Saludos. Iñaki.
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Michelangelo Sommaruga -

Hi Iñaki (vasco supongo)

Check this out. I couldn't install Moodle with Postgres8 client in a server, so I downgrade the versions of Postgres (even apache and php) to postgres7 and I'm connecting to a Postgres8 linux box. It works ... but at high occupancy (we have 17k users) it seems to perform very slow. I checked the DB server and I'm tuning it.  I know that for now the bottleneck is there, postgres8 seems to be too slow. I just wanted to tell you that a client 7 can connect to a server 8. BDW, I'm using Moodle 1.6.4, and the dump file from postgres is giving me some extrange chars for accents and ñ, I don't know why.

Saludos,

Michel.

In reply to Marcin Wojciechowski

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Bartosz Cisek -
I had similar to Marcin's problem. I was using '-c' flag, and in spite of this mysqldump put several tuples in one row.

Solution is '--skip-extended-insert' option which I finally found on mysql web site. Hope it will help someone.
Average of ratings: Useful (1)
In reply to Iñaki Arenaza

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Crafton Williams -
Thank you very much for this guide, it worked flawlessly in my migration from mysql to postgres using moodle 1.8 and postgres 8.1.
In reply to Crafton Williams

Re: Upgrading to 1.6 and may need to switch to PostgreSQL

by Andy Cravens -

I know this is an old post but does anybody have any additional information on migrating from mysql to postgresql as it relates to version 1.9.x?  I'm going to try this in a development environment.