Referential integrity in Moodle

Referential integrity in Moodle

од Tim Hunt -
Број на одговори: 42
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
This post is in response to a rant by sam in the Moodle developers Jabber chat yesterday. (If you have CVS access and Jabber, and don't yet know about this, you can find information at http://moodle.org/mod/cvsadmin/view.php?cid=1.) I decided to move the discussion here to get more brains involved. (That is the same reason I am posting here and not in the Database forum.)

This issues is this: since Moodle 1.7, the install.xml files that define the structure of the Moodle database have included definitions for foreign keys. That is, role_assignemnt.userid refers to user.id, and so on. This acts as a type of constraint. If you try to instert some data into the role_assignments table with a userid that does not exist in the user table, then the database can check this for you and report an error, and you might end up finding a bug in your code that otherwise you would have missed. That is the theory anyway.

However, Moodle has never enforced these rules in the past (old versions of MySQL did not support foreign keys), so any Moodle system is likely to have some data in the database that does not satisfy the constraints. Therefore, adding the foreign key constraints now would cause things to break, at least until we had done a lot of work to find and fix bugs.

So there is a big practical hurdle in the way of turning on foreign keys and getting the robustness benefits they will bring, and several times in the past groups of Moodle developers have had discussions about whether the long term gain is enough to justify the short term pain. There is not a clear right answer, there are good points on both sides.

That is all a summary of where we are now. My actual reason for posting is to make some specific suggestions:


1. We should make a report so that we can easily find out how close any Moodle site is to obeying all the foreign-key constraints.

In the XMLDB editor, there are already several check options available, like [Check Indexes] [Check Defaults] [Check Bigints]. What these do is to compare the structure of the database you actually have, with what is defined in the install.xml files. So, for example, if your Moodle started life as Moodle 1.5, and has been upgraded several times since then, you can check whether the result of upgrading your old database exactly matches what you would get by doing a fresh install.

I think we should add a new check: [Check foreign keys]. This should be possible because if you have a foreign key like "(userid) references user (id)" on role_assignments, then the query
SELECT count(*)
FROM {role_assignments}
WHERE NOT EXISTS (
 SELECT * FROM {user}
 WHERE {role_assignments}.userid = {user}.id
)
Will count the number of rows in role_assignments that violate the constrain - and that query can be automatically constructed from the foreign key definition.


2. We should create an option for developers, so they can selectively enable certain foreign keys.

The main advantage of foreign keys is they help developers find bugs. Therefore we should make it possible for developers to get that benefit, without causeing extra errors on productions sites. I think this is just like DEVELOPER debug level or XHTML strict headers.

One way to do this would be in the report 1., for every foreign key constraint where there are no rows that currently violate the constraint, have a button to acutally create the foreign key in the database.


That is probably enough for now. If we do thises two things, then we will start to get hard evidence to feed into the debate about whether we should enforce all foreign keys in some future Moodle release.
Просек на рејтинзи: -
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
There is code to do 1. attached to MDL-16975 and MDL-16976. I am planning to check it in once someone has reviewed it. (I am afraid this is going into HEAD only, but then this is mainly aimed at developers.)
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
This is now in HEAD.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Dan Poltawski -
I think this is going to be really useful - great work Tim!

It might be a good idea to tabulate the results in a machine readable format so that we can do something with this really useful data - finding top constraint offenders from multiple moodle sites and see if there is a pattern indicating a bug.

Submit reports to a dev.moodle.org percentage constraint hitlist perhaps?


For what its worth, I just ran the report on a randomly chosen real site and have attached the results.
Во одговорот до Dan Poltawski

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
What do you mean, the current format is perfectly machine readable: http://xkcd.com/208/

But more seriously, an export as CSV option on that page would be useful, and you are most welcome to implement it. Hmm that might be a little tricky given the structure of XMLDB check code, but I am sure a resourceful person like you can get round that. Anyway, the code is in CVS, you are most welcome to enhance it. (I don't have time.)
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Dan Poltawski -
I haven't quite got round to doing an elegant solution - but I have created a CLI script which works with 1.9 to produce a report of violated constraints - this is available in cvs:/contrib/tools/devtools/xmldb_refintegreport.php

I've used this to generate a summary of violated constraints from my sample of 700 moodles, which is now in MDL-17623.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од sam marshall -
Слика од Core developers Слика од Peer reviewers Слика од Plugin developers
ARGH SWEAR SWEAR SWEAR I just accidentally hit back button and the piece-of-swear html editor lost my massive post. I'll try to write it shorter this time.

1) Great!

2) Can we have a flag at top level of an xmldb file to enable integrity for that file for new installs only [there would be no upgrade for this, any users wanting to upgrade could manually use the code tim's done or similar, so it doesn't break existing installs]. As each set of tables (one xmldb file) is checked, integrity can be enabled for those tables, so we can progress gradually toward the end goal.

3) For this to be sane and not have a potential of damaging anything, there must be no CASCADE options ie you have to delete in right order.

4) Activity/etc uninstall should use xmldb file to drop the tables in the right order or more likely just drop constraints before dropping the table (think it currently still just does DROP modulename*)

5) Is there some way to force developers to enable all appropriate constraints, e.g. have a big red YOU DIDN'T TURN ON CONSTRAINTS warning (like the YOUR DATAROOT IS LEAKING one except that it only appears on developer debug level). Because if developers don't have constraints they might stop something working.

--sam
Во одговорот до sam marshall

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
2) 3) Eloy says there is already a way to turn on foreign key creation when you install Moodle. I suggested he post here explaining how, but he has not yet.

4) I think module uninstall nowadays uses the install.xml files to drop things in a safer way.

5) Well, there is no such warning for developers who run with XHMTL headers turned off. I just end up fixing all the bugs everyone else introduce - or at least that is how it feels some times. However, if you write such a warning and check it in, I would not object намигнува
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од sam marshall -
Слика од Core developers Слика од Peer reviewers Слика од Plugin developers
install.xml stuff - not just when you install moodle, also needed when installing individual modules later. And I think it would be useful on a per-file basis that somebody has to turn on having checked it, because my guess is if you just turn that option on globally and install moodle, everything would break? Of course, I haven't tried it....

And смешко - XHTML headers isn't the default though, is it? It totally should be default when in developer debug mode. I think that would solve the problem better than any warning. (Ok except for those developers who use IE but really... there aren't any, are there?)

--sam

PS I just fixed a bunch of well-formedness and validity bugs in our code, AND in exasperation added a requirement about checking the sweary page with w3c validator to our handover process...

PPS I am liking the use of the word 'swear' as an actual swearword. It's swearing great!
Во одговорот до sam marshall

Re: Referential integrity in Moodle

од Dan Poltawski -

> And - XHTML headers isn't the default though, is it? It totally should be default when in developer debug mode. I think that would solve the problem better than any warning. (Ok except for those developers who use IE but really... there aren't any, are there?)


Sorry to drift off topic - I think the problem with XHTML errors for developers is displaying php errors inline conflciting with XHTML valdiation ( I'm guilty of both these crimes - along with user created XHTML errors annoying me ;))

Dan
Во одговорот до Dan Poltawski

Re: Referential integrity in Moodle

од sam marshall -
Слика од Core developers Слика од Peer reviewers Слика од Plugin developers
You're absolutely right, I had forgotten about that. I suppose there's no way to make php nicely collect up all the errors and then print them in a defined place in the footer or something... yeah, just kidding.

--sam


Во одговорот до sam marshall

Re: Referential integrity in Moodle

од Petr Skoda -
Слика од Core developers Слика од Documentation writers Слика од Peer reviewers Слика од Plugin developers
New messaging framework was supposed to handle that, right?
Во одговорот до Petr Skoda

Re: Referential integrity in Moodle

од Martín Langhoff -
Gentlemen, the _logs_ handle that. Anyone not tailing the logs in a separate window is missing out смешко

Во одговорот до Martín Langhoff

Re: Referential integrity in Moodle

од Petr Skoda -
Слика од Core developers Слика од Documentation writers Слика од Peer reviewers Слика од Plugin developers
hmm, how do you tail it if you do not have shell access? намигнува
Во одговорот до Martín Langhoff

Re: Referential integrity in Moodle

од Penny Leach -
Gentle what now? evil

Во одговорот до Penny Leach

Re: Referential integrity in Moodle

од sam marshall -
Слика од Core developers Слика од Peer reviewers Слика од Plugin developers
Don't worry Penny, I'm sure nobody intended to accuse you of being gentle.

As for shell access - OU developers generally don't have shell access to dev server or acceptance test server (and obviously not live), because all these servers are managed separately in the hope that we don't break them. We could probably get it if we really needed it, but it's not exactly convenient compared to seeing errors right there in the page... if only they could be safely located in the footer...

--sam
Во одговорот до sam marshall

Re: Referential integrity in Moodle

од Penny Leach -
I don't understand why you can't register an error handling function to catch all errors (except Fatal, you can't catch that), add them up somewhere , like a special global or something, and then make the footer print them (or error_log) them.

In Mahara, we have something similar, although the error handling function will print them to screen or apache log depending on config stuff, with a backtrace or not, depending on more config stuff.

Have a look here:

Во одговорот до Penny Leach

Re: Referential integrity in Moodle

од sam marshall -
Слика од Core developers Слика од Peer reviewers Слика од Plugin developers
That looks great.

If somebody put that in Moodle (either the simple plan described in Penny's post or a more full-featured version like the one in her code) then finally it should be possible to automatically turn on xhtml mode when in developer debug (which would mean removing a config option!!! wooooo! etc).

--sam
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
In lib\ddl\sql_generator.php there is a setting called $foreign_keys which defaults to false.

If you set this to true the install script will generate foreign key constraints. With MySQL this will only be preserved by a database that is using the InnoDB engine. If this were to run to completion it would then be possible to use the GPL MySQL workbench to reverse engineer an ERD from a database dump. This would not be a perfect ERD because some parts of Moodle are not really relational but it might be handy.

Where all this falls over is that I have just spent many hours trying to run the install with foreign_keys=true and it generates a 150 error indicating a broken foreign key relationship. I have tried turning off foreign key checking in MySQL or finding some way of capturing the DDL statements when run against MyISAm but to no avail.

But perhaps someone who understands this a bit more than me can take the idea to the fruition of a diagram (or just the DDL table creation statement would be handy).

Marcus
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
I have an example of a "reverse engineered" ER diagram of part of the moodle schema. You can see it at

http://www.jchq.net/moodlequiz.png

I got the sql from the mysql log and did a bit of hand arranging of the layout. The foreign key relationships come from the sql and you can see that some of them are missing a relationship, not sure why. This could be a handy way of auto generating a diagram for all/most of moodle.

Marcus
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
I have a nice hand-drawn diagram at Development:Quiz_database_structure which probably makes more of the structure apparent.

However, there are a lot of parts of Moodle missing such diagrams, and it would be wonderful if you had time to generate diagrams for some of the collections of tables on Development:Database_schema_introduction.

Also, what tool did you use to generate your diagram? I have yet to find an Open Source tool for drawing database diagrams that produce results that I am entirely happy with. And it would be much more helpful if we could upload database diagrams to Moodle Docs in a format that could later be updated with Open Source software, as well as in PNG format.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Samuli Karevaara -
Marcus' diagram looks like it's made with MySQL Workbench.
Во одговорот до Samuli Karevaara

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
Windows only, but less than 60 days to linux and mac releases if the "Please note that at this point only the Windows release is available. Linux and OS X releases will be available in 2008." bit means anything.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Samuli Karevaara -
Here they have instructions on how to build 5.1 alpha from the source (link to the source included in the article) for Ubuntu, but I don't think I'll bother, specially as it's "alpha". I'll try it in the "apt-get install mysql-workbench" state
Во одговорот до Samuli Karevaara

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
Yes, its MySQL Workbench, you can output as png or pdf or for other users of the product there the xml file to mess with.
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
I have just created a MySQL Workbench ER Diagram of the Moodle 2 database structure by "reverse entineering", i.e. Installing Moodle and then taking the statements from the log file, exporting them to a script and importing into MySQL workbench. Because I had the create Foreign Keys flag set to on (see previous post), it shows Primary Foreign Key relationships. If anyone is interested I can make this available and document how I did it.

Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од HJWUCGA INC. -
Yes please. That would be most helpful.

Thank you
Во одговорот до HJWUCGA INC.

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers

Screen shot attached and you can download the *.mwb (MySQL workbench) file from
www.jchq.net/moodle2erd.mwb

Screen shot shows a zoom in and the whole diagram, which as you will see would benefit from re-organisation to make the setup clearer. Note that Moodle 2 is still in development and may change before release.

I have put the Moodle logo in the diagram, is that OK with everyone?

Disclaimer, ... I'm just a fan, none of this is in any way "official".

Прилог moodle_erd_clip3.png
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Subodh Iyengar -
This is awesome..
Just what I needed to complete my project.
We tried before to do something like this, but failed miserably.
Thanks a lot.
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Alex Büchner -
Marcus,

this is great. Can you produce a huge JPG of the full diagram? Would love to print it on A1 or A2.

Thanks
Во одговорот до Alex Büchner

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
Have you got a very large piece of paper (A1)?
See
http://www.jchq.net/bigmoodle2erd.png

It did a better job of fixing overlaps than I was expecting, but it would be good to divide it into task specific sections.

Marcus
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Marcus Green -
Слика од Core developers Слика од Particularly helpful Moodlers Слика од Plugin developers Слика од Testers
Here is an example/attempt at isolating one part of the schema, the quiz related tables.

Прилог Quiz.png
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
I think Development:Quiz_database_structure and Development:Question_database_structure show the groupings of these tables even better, but then I would think that, I drew them.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Frank Ralf -
I added all your links to Database FAQ#b.29 Moodle database so don't quarrel намигнува
Во одговорот до Frank Ralf

Re: Referential integrity in Moodle

од Tim Hunt -
Слика од Core developers Слика од Documentation writers Слика од Particularly helpful Moodlers Слика од Peer reviewers Слика од Plugin developers
Not sure about quiz-specific links there. I think it should be enough to link to Development:Database_schema_introduction.

By the way, that page divides the list of database tables up into logical chunks. Those would be good sets of tables to use when creating smaller diagrams. And it would be good to have diagrams added to that page. In fact any improvement to that page at all would be great.
Во одговорот до Tim Hunt

Re: Referential integrity in Moodle

од Frank Ralf -
Removed the quiz-specific links from Database FAQ as they are indeed linked from Development:Database schema introduction, which I only noticed now.
Во одговорот до Frank Ralf

Re: Referential integrity in Moodle

од Alberto Gil Fernández -
Hey Marcus!

Would be so kind of you to post a link to the .sql script you had used to create the ER model.

Thank you
Во одговорот до Marcus Green

Re: Referential integrity in Moodle

од zakieh hashemi -

hi markus ....

can i have a complete erd of the moodle 2 database plss ... i need it so much if u can help me ...