Important: Separating Moodle and DMS

Important: Separating Moodle and DMS

by John Papaioannou -
Number of replies: 19
Hi everyone,

I 've been hacking around in the MyDMS code when I got the thought to lose some "redundant" database configuration. This rang a warning bell (we don't want to be limited as to the where the DMS database would be; it's not necessary that it will be the main Moodle database) because of things that have been said in discussions on this matter.

So I sat down and thought about it. The scenarios I can think of for interoperability between the DMS and Moodle are:

  1. DMS is ran as a Moodle component, and reads its data from tables in the same Moodle database. This is the simplest case. The database can of course be located on a different server; Moodle provides support for this out of the box.
  2. DMS is ran as a Moodle component as before, but the database used is not the Moodle database. There are three possibilities:
    1. The "other" database is part of another Moodle installation; we want to centralize our file repository, so we have "satellite" Moodlesites that depend on it.
    2. The "other" database is a subset of the database of this installation (for example, when using a more powerful server to host multimedia content, place the DMS tables and BLOBs there).
    3. The "other" database is a full-fledged MyDMS installation, and we just want to leech its information and make it accessible through Moodle.
  3. MyDMS runs stand-alone, and reads its data from the DMS database of some Moodle installation (this is scenario 2.3 reversed).

Important notes for the above scenarios:

Scenario 1 is of course the most obvious. We 're in the process of implementing it.

Scenario 2.1 presents some trouble: files can be organized and the access controlled by course, by user, by group etc. What if the "other" Moodlesite differs from ours? What is "our" DMS supposed to do in this case?

Scenario 2.2 does not present many technical difficulties, since we can do as we please with the "other" database. The only problem is that Moodle itself will have to be modified if we are to support this option (the "other" db needs to be kept current, so it must be updated by Moodle when courses/users are created, deleted etc), resulting in a complexity increase just for this. Would that be justified?

Scenario 2.3 presents a whole set of problems we also encounter in scenario 2.1. We aren't allowed to do "as we please" with the other database, so how do we handle permissions, organization etc etc when the "other" MyDMS installation knows nothing about courses, groups, etc? You could argue that in theory, things in the "other" installation will always be kept in conformance to some standards that we can agree on, but that's not possible in practice. We need to provide for erroneous cases anyway, so the standard might as well not exist.

Scenario 3, in order to be implemented, requires that we:
a) keep all tables, even redundant, of the MyDMS schema and painstakingly maintain constant synchronization between them and the normal Moodle tables (this will require changes in the Moodle core and, as any programmer knows, is asking for trouble),
b) be constrained in what we do with group access rights, roles etc by the functionality we can squeeze out of the existing MyDMS schema.
Not a good prospect, overall.

It should be clear by now that all scenarios except 1 are problematic in some way. However, instead of just going ahead and customizing DMS for scenario 1, it's time for me to pop the big questions:

1. Which of the above scenarios do you think that we have to support? How do you propose to support them? The "how" is especially important, though people without technical backgrounds will understandably not be able to answer it. Please document your opinion as best you can.

2. Is there some case I have forgotten or misunderstood? (these scenarios are based on my reading of the older discussions).

I cannot stress how deeply these matters will affect the development of the DMS, so I urge everyone to express their views on this subject. It will be much easier to make a good decision now that to change a dubious decision once it has been made.

Regards,
Jon
Average of ratings: -
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Good thinking and well laid out, Jon.

My initial thoughts are that just these two should be useful enough for now...

1)  Obviously it should work internally by default.

2.2)   Except that the users, groups stuff should be read DIRECTLY from the Moodle database (just as in 1) even though the actual data and DMS scripts are stored in an external database.

My thinking for the steps to move from internal to external would be something like:
  1. copy the DMS scripts from inside Moodle to the external location
  2. configure the DMS so it knows about
  1. it's own local database for files data
  2. the (now remote) Moodle database for all the other stuff

Cheers,
Martin

P.S.  Scenario 4 is ditching all of this with a peer-to-peer solution.  wink  wink  Joke!
Average of ratings: -
In reply to Martin Dougiamas

Re: Important: Separating Moodle and DMS

by John Papaioannou -
Martin,

That's not what my 2.2 scenario was. You are talking about something more like scenario 3 (standalone MyDMS). The main difference between these is that in one case we get to use all the standard Moodle libs, while in the other we don't have them. That's going to be a pain to maintain.

2.2 could be implemented as "read users etc from the Moodle database" (either local as is usual or possibly remote) and "read and write files and bookkeeping info to remote database" (another machine entirely). Is this the desired setup? Thankfully, it's the about the only one which doesn't give me nightmares...

Come to think of it, it does have one major disadvantage: it uses the Moodle web server as an intermediate station between the "remote" DMS file storage and the user, thus possibly defeating the whole idea of using a powerful file storage server. So you 're most likely talking about what I labeled scenario 3?

Hell... maybe we should indeed see if we can make BitTorrent a module. thoughtfulwink

Jon
Average of ratings: -
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Yes, Scenario 3, quite right, sorry! I made the mistake of reading too quickly (this is why "Read carefully" is next to every forum posting field! smile) and mis-mapped my preconceptions onto your words.

The Moodle libs situation is not a big problem if the external DMS just has a copy of the whole lib folder with it (and perhaps some checks in the main code to check the DMS version against the Moodle version and remind the admin when upgrades become necessary).

Average of ratings: -
In reply to Martin Dougiamas

Re: Important: Separating Moodle and DMS

by John Papaioannou -
I see... so, we need to plan ahead for this. But let's wait some more in case more sensible ideas come our way.

Cheers!
Average of ratings: -
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by Ger Tielemans -
Don't we need a simple user model where you are free to fill the DMS from outside, but when a Moodle uses a fill delete or replace is forbidden until Moodle gives it free?
Average of ratings: -
In reply to Martin Dougiamas

Re: Important: Separating Moodle and DMS

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

Why would we want scenario 3?

mike

Average of ratings: -
In reply to Mike Churchward

Re: Important: Separating Moodle and DMS

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Well, flexibility, really. For larger installations this offers an option to split the bandwidth and CPU across two machines without the need for fancy clustering schemes.

One of my customers, Paul Duff, was most keen on this, I've just subscribed him to this new forum in case he wants to talk about this more.
Average of ratings: -
In reply to Martin Dougiamas

Re: Important: Separating Moodle and DMS

by Paul Duff -

Hi All

Thanks for the 'heads-up' Martin.  I've watched this thread with interest, and am really pleased that someone is starting to tackle the DMS feature.  I've resisted the temptation to become too involved to this point, prefering to see how discussions evolve - the reason is that I'm very conscious of the fact that we are a large scale user, probably with slightly different view of the world - also, our view may not be the view of other large institutions.  Please accept that I'm not proposing to shape or skew development in favour of our model, but I'd be most grateful if the following were considered before commiting to a particular course.

1. Content

Separate the content from Moodle - this is fundamental for us.  The content is.....well, the content.  A mixture of home-grown and 3rd party product.  Theres lots of it (with more to come) currently 16Gb in our case, and thats without any volume of video/audio content.  Additionally, SCORM compliance is in the offing, and believe me, a lot of that material is biiiiiig.  Technically, the more servers we can spread the installation over, the easier is is to overcome potential performance issues - the ideal might be to accomodate the possibility of multiple DMS instances.  At a business level, large institutions are especially reliant on open standards and flexible solutions, and it's my job to ensure that we keep our options open.  We're on our third MLE/VLE, and keeping the content separate has made the transition from one product to another relatively painless. 

2.  Security

Ideally, we'd see the need to accomodate three main areas of content within one or more DMS:

a)  A public area which could be read by anyone
b)  A major single content area which technically could be read by any registered Moodle user (or possibly from an identified Moodle instance(s))
c)  A very small area of content which could be read only by certain individuals or courses to accomodate material like Cisco Academy stuff where we are contractually bound to present only to those enrolled on certain courses.

In all cases, the rights to write to each area would need to be limited to validated individuals or roles authenticated against the Moodle database.  The biggest area of content in this scenario would be item b, and for that content, to validate each read at user or course level might be an un-necessary overhead.  A simpler approach might be merely to confirm in some way that the calling host was actually a pre-defined Moodle instance(s) - thus technically, anyone registered as a Moodle user could read content, but only if pointed at this content from a legitimate Moodle resource link.  In the end, the content store will hold thousands of items or objects which are unordered and many unrelated - Moodle is the glue which brings order to this chaos, and does it very well.

3.  Searching the stores

We think that before authoring fresh material (and possible re-inventing the wheel), teachers should be able to search the stores for existing material.  This ability to search should I suspect also be limited to individuals or roles - certainly, there would seem to be little need for students to search the store.  I can also forsee a scenario where not all teachers would be authors.  Some institutions might wish to have materials authored/published by a central or core team, but allow other teachers to search the store to identify existing material to point at.  One day, Moodle itself might accomodate the notion of templated or re-usable course definitions, but that's another story (we'd certainly be willing to coontribute fiancially to that), but re-usable material is a good first step - we'd really like to avoid multiple copies of the same material and any onerous additional administrative overhead associated with the maintenance of read rights.

As I write this, I'm wondering whether the DMS should really be fully integrated with Moodle, or whether it might be better viewed as a separate facility which is Moodle compliant or aware.  This submission is certainly not exhaustive - I'm sure that many will have other requirements.  The suggestions above are simply a list of key elements which come to mind.  I also accept that some might prefer a fully rights based scenario for both read and write - thankfully, others will determine the direction development takes, but I would suggest that everyones best interests are served by making Moodle as flexible as possible. 

In the end, if for conceptual or programming reasons, some or even none of the above can be met, that doesn't alter the fact that Moodle is still the best game in town.

Regards

Paul

Average of ratings: -
In reply to Paul Duff

Re: Important: Separating Moodle and DMS

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Paul -

Your concepts are interesting and very valid. I won't add anything to your 'Security' and 'Searching the Stores' paragraphs, since they are the very reason I want to implement this feature.

The 'Content' paragraph raises some concerns though.

Separating the content from Moodle is not quite as simple as MyDMS. While providing CMS and LMS application server features, much of the content is integrated quite tightly into Moodle. This is its strength and (I fear) its weakness.

Certainly, a DMS feature will make it easier to separate document resources from the application. This is fundamental. But forums, quizzes, exercises, etc., I'm not sure how we can move those out of Moodle?

Unless... the DMS becomes the core content management system for the entire application?... Martin, is this what you were alluding to when you initially referred to the DMS as a Content Management System? (sorry, I'm thinking as I type now).

Hmmm. Hmmm. I need to think again.

mike

Average of ratings: -
In reply to Martin Dougiamas

Re: Important: Separating Moodle and DMS

by Jason Cole -
There's something bothering me about the whole approach so far. I wonder if we're reinventing the wheel and causing ourselves longer term headaches by ignoring WebDAV and products like Subversion. If the only way to access the filesystem (or at least have the DMS recognize that there are changes) is through a web interface, we're limiting the useability of the system. For example, my university is developing a SAN that I'd like to be able to tie in to moodle DMS. I can't do that if there's a database in the way, but I can do it with WebDAV as the SAN system should have that protocol enabled.

Everyone who has or wants NAS or SAN solutions is going to encounter the same problem. The only way to use MyDMS is through the web interface, there's no client side integration.

The option if we want to stick with MyDMS is to write a sweeper script that runs on a cron schedule that would look for additions and changes in the file system and write those changes to the DB.

I'd suggest we take another look at Subversion and a PHP webDAV front end for it.

I'm also working on diagrams for a straight MyDMS implementation too.
Average of ratings: Useful (1)
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

Okay. I'm going to step away from the code for a bit, and draw up a conceptual design for myDMS. I'll focus on the user point of view, teacher point of view, administrator point of view and back-end.

mike

Average of ratings: -
In reply to Mike Churchward

Re: Important: Separating Moodle and DMS

by John Papaioannou -
Very good idea... I 've been messing around with the code and also came to the conclusion that no real work can be done until we have what you propose to do.

Until then, for myself I 'm going to make only porting changes, nothing that commits to a specific idea or policy.

Jon
Average of ratings: -
In reply to Mike Churchward

Re: Important: Separating Moodle and DMS

by Jason Cole -
Here are some architecture diagrams I've ginned up. As a potential very large user, I'm interested in how the solution we come up with will scale. Seperating DB's adds a level of complexity that I'm not sure is warranted.

Diagram moodle-dms-int shows my conception of integrating the MyDMS db with the moodle db. We could always add a seperate dms app server (lower right) that would use the moodle db, but just have the functionality subset of mydms.

Diagram moodle-dms_synch shows the dms db seperate from the moodle db. As the moodle courses will still need to access the dms db for file pointers and metadata.

I'm nervous about synching the users in the two dbs. It seems like a scalability and support nightmare waiting to happen.
Attachment Moodle-DMS_synch.gif
Average of ratings: Useful (1)
In reply to Jason Cole

Re: Important: Separating Moodle and DMS

by John Papaioannou -
Comment on the diagrams:

Separated DBs

First of all, the title of this topic doesn't imply that I think separating the DBs will be a good idea. It's just the fact that I wanted to put the matter on the table for discussion, because we can't go on coding unless we have a very clear idea of what we want to do.

Your diagram implies (to me at least) that these two databases will be in different locations. If that's correct, we need to provide for scenario 2.2. I 'm just stating the fact so that it's clear to everyone who is going to get involved with the development, and I agree that supporting this kind of separation is probably going to cause more problems that it solves.

Integrated DBs

That looks like the main case we want to support. IIRC, Martin thinks so, too.

Jon
Average of ratings: -
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

So we really thinks that the DMS application could be so large, it would need its own server? Okay, I'll accept that (at least as an option). I'm thinking if we want to go this route though, we will need to look at Moodle 2.0 (or 3.0) and redesign the architecture from the ground up.

I'm not against that (in fact I'd love to be involved). Maybe we can plan a preliminary roll-in of DMS for the current Moodle, that doesn't involve that level of separation, and plan for the other in a separate development branch.

Is there a developer's IRC channel?

mike

Average of ratings: -
In reply to Mike Churchward

Re: Important: Separating Moodle and DMS

by Paul Duff -

Hi Mike

Two outa three ain't bad!!  I must confess that my view of 'content' was largely limited to documents (and SCORM compliant LOs) on the assumption that the D stood for those, although the notion of learners uploading files into individually secure areas did cross my mind. I suppose exactly WHAT resources the DMS is intended to house is very germain to the discussion.

I guess the main purpose of my submission was to flag up the need to design IN the possibility to have the DMS either on the same server as Moodle, OR optionally on one or more servers for those with larger amounts of content - not for storage reasons (that's pretty cheap these days) but really because more content most likely means more activity, and that requires processing power and bandwidth, particularly with the ease with which Moodle now accommodates audio & video content - all on one box pretty well limits expansion to clustering.

In truth, I was hoping that my submission might just garner support or discussion from those representing larger institutions but that doesn't seem to have happened much - I guess that means that my college isn't representative of a core of users...........or I'm way off mark. 

I still think that scalability should be taken into account at the design stage, but do understand that Moodle developers must cater to the needs of the majority of its users.  Can't win 'em all............

Regardless, I'll be interested to watch how this discussion evolves.

Regards

Paul

Average of ratings: -
In reply to Paul Duff

Re: Important: Separating Moodle and DMS

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

I'm not ruling it out at all. I've just had to readjust my thinking. It looks like what we need is for the DMS to be a totally separate front end that interfaces with Moodle. And from my efforts to spec it out, that looks like the best way anyway.

mike

Average of ratings: -
In reply to John Papaioannou

Re: Important: Separating Moodle and DMS

by John Papaioannou -

In view of all this, I propose that when coding, we use the $settings->_moodleInterface var I added in CVS to make this extensible. We can have if blocks for this where an assumption would otherwise be needed.

Small example (taken from inc.Language.php, function getMLText):

if (!isset($text[$key])) {

        // Let's see if we can get it from Moodle...
        // VERY conservative programming here!

        if($settings->_moodleInterface == DMS_INTERNAL && function_exists('get_string')) {
            return get_string($key);
        }

        return 'Error getting Text: '.$key.' ('.$settings->_language.')';

    }

Instead of losing the getMLText entirely (after all, we wouldn't need that if we 're running inside Moodle) we can keep it and just change its behavior to call get_string() if we ARE inside Moodle.

Note that the way it's written now, fetching strings is deferred to get_string() only if the string isn't found, while normally get_string() should take precedence. I left it that way so that I didn't need to concern myself with the language files yet.

In the future, when deciding what could be done for hosting content on a different server, $settings->_moodleInterface can help us extend this while letting the code work as before in normal Moodle installations.

Jon

Average of ratings: -