Central Moodle instance sync to local instance - updated

Central Moodle instance sync to local instance - updated

by Paa Kwesi -
Number of replies: 14

Hi moodlers,


I'm setting up a new Moodle installation and i want it to have these features.


1. Central Moodle instance (e,g. Cloud Server). Where all courses would be created here.


2. LAN would be set up at different locations (for the start we want to test with 3 locations, but more would be added if everything goes well. These local hubs should be able to communicate with the Cloud Server to sync courses and course materials created on the Cloud Server. 


3. Users would be created on the various LAN sites and would be take the course accordingly without the need to connect to the internet, but access the material via their local network. 


The diagram below is a pictorial view of the required topology.


I will very much appreciate your contributions.

Thanks in advance.


Attachment Topology.png
Average of ratings: -
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Pretty diagram... but how is this any different from the last time you asked this question (https://moodle.org/mod/forum/discuss.php?d=345932)?

Average of ratings: -
In reply to Howard Miller

Re: Central Moodle instance sync to local instance - updated

by Paa Kwesi -

Thanks Miller,


Well the question is the same, except that this one sheds more light on the scenario and explains better. Sorry to break the rules tho. Was hoping to give a better explanation so i get more suggestions on the solution to this issue. 


Thanks for the compliment too..

Average of ratings: -
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

So, if the hubs have the necessary internet connection, why not just extend that to the students' laptops?  

The biggest problem I see with your scenario is that you have three different locations that would want to sync with one server.  That would cause issues in most situations, irregardless of the fact that Moodle just does not offer this functionality.  

If you are dealing with a bandwidth issue.  I think your best solution would be to host three separate Moodles, one at each hub.  You could then mirror the three installations in the cloud if you needed to for course creation etc.  Then just sync the three installations daily or weekly - whatever you need to do.

Average of ratings: Useful (1)
In reply to Emma Richardson

Re: Central Moodle instance sync to local instance - updated

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Don't forget that when 'syncing' (I take this to mean making a complete copy of the site in this case) you should have your site offline or (at least) in maintenance mode. What you don't want is for the database to get out of sync with the file area. 

As previously discussed, there's no mechanism for true synchronisation of Moodle sites and it would be *very* difficult (I don't want to to say impossible but getting on that way) to achieve. 

Average of ratings: -
In reply to Emma Richardson

Re: Central Moodle instance sync to local instance - updated

by Paa Kwesi -

Thanks Emma for your suggestion. I appreciate it.


However can u explain what you mean by 'mirror the installations'.


Also can the 3 installations be synced as you stated in your last sentence.  


Thanks.


Average of ratings: -
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

I think ideally you would have three separate installations that were then duplicated to the hub server.  Ideally, as Howard mentioned, you should take everything offline (at night for example) and then run a sync utility between the two instances.  Once set up, I would think that you could script rsync or some other utility for the moodle and moodledata folders and there is surely some sync utility for whatever database you use that could be scripted too.  

As mentioned several times, there are always risks with this sort of setup so it would be wise to backup both sides prior to the syncing process to be safe.  Chances are you might run into some caching issues between the two instances but that could be resolved with a purge of the caches after syncing.

Disclaimer(!!) - I have not run a setup like this myself.  But I have migrated a lot of sites though and after initial copy, I have been successful using rsync to update all the files prior to going live with the new site (makes my downtime so much shorter!).

Average of ratings: -
In reply to Emma Richardson

Re: Central Moodle instance sync to local instance - updated

by Ken Task -
Picture of Particularly helpful Moodlers

Migration isn't the same as a sync.   The files in moodledata can be sync'd easy enough, but what of the data in the DB that's changed.    Having said that, however ... this info might help:

http://dba.stackexchange.com/questions/65351/sync-two-mysql-databases-in-two-different-locations

which means those circuits between LAN locations and HUB better not ever develop hickups!

Think @Paa needs to clarify ... sync isn't going both ways is it?  What you want is new courses developed at the HUB to appear in the LAN instances, is that right?

From what I've understood about the proposal, courses are developed on the HUB server and the goal is to have those new courses appear in all the LAN servers.

Now *if* the sync desired is going from each of the LAN servers into a single server at the HUB that's different.   Moodle has no DB utility, of which I am aware, that dumps the days activity to an sql that could be imported into a 'semi mirrored' server.    The only way possible would be those suggested in the link provided.

Not that this would happen in posters case, but it could ... I live in South Texas and have had to assist a school district with a user issue.   Two students have *exactly* the same first, middle, and last name but they are not related and don't go to the same school.   Easy enough to solve ... user name and email address for one had to be generated as a special case ... no conflict.

Anyhoo ... it will take some experimentation me thinks ... and there will be un-intended hickups that might involve only ONE of the LAN servers.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Central Moodle instance sync to local instance - updated

by Paa Kwesi -

Hello Ken, 

Thanks so much for the info... to clarify the question better


1. The sync isn't going both ways.. And no data would be synced back to the hub.

However the LANs should be able to sync the courses that would be created on the Hub. This also means the LANs should be able to sync any update to the courses on the Hub.


Average of ratings: -
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Ken Task -
Picture of Particularly helpful Moodlers

Am pretty sure the course (really no user backups) could be 'pushed' to the LAN servers from the HUB instance where the courses are developed.

It's the second part ... that's still unclear to me ...

"This also means the LANs should be able to sync any update to the courses on the Hub."

So ... let's say a course called 'Moodle Basics' (MB for short) is developed and 'pushed' to the LAN servers.    A week later (or sometime later), an addition is made to MB.   It is desired that the addition now be added to the LAN MB courses?

There is a Moodle work-around to do that but it would mean a human located at the LAN servers would have to be involved ... basically a no user backup of MB that included only the newly added content.   That backup made available or pushed to the LAN servers.   The admin at the LAN server would have to restore that to the MB course and re-order/place the additions in the proper section etc. of the course.

Think it best to make sure that whatever is contained in a course to be pushed is really complete ... no new addtions needed.

If neither of the above acceptable, then think that leaves only LTI Tool on the LAN be added to the MB course that points to an LTI resource ... just the addition ... on the HUB.   That would mean the LAN instances would be attempting to access content on the HUB - so that kinda rubs against what the overall design does.

Think there still would be need of a human at the LAN servers who have fairly good knowledge of how to do some things on the LAN servers.

To achieve total automation between LAN's and the HUB would involve DB replicaton setup between LAN's and possibly separate DB's (one for each LAN) on the DB server located at the HUB.   This again means traffic on the upstream circuits.

Maybe if you could disclose some details about  those upstream circuits ...   IF, for example the connections from the LAN's are 56KB  dry pair then or less than a T1 or a T1 with sporadic outages etc.  think the setup of all of this would be in need of constant issues.

'spirit of sharing', Ken

 



Average of ratings: Useful (1)
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Ken Task -
Picture of Particularly helpful Moodlers

Interesting ... but full  of issues *IF* the intent is for the LAN Moodles to 'sync' data back to the HUB server.   That's due to the nature of Moodle being a dynamic application *requiring* a database and the new data contained in the moodledata directory of each site.

However, pushing new courses to LAN servers could be scripted ... me thinks.

Briefly ..

HUB server uses CLI scripts that already exist to create no user backups of new courses and saves the backups to a directory outside of Moodle ... a 'coursepickup' directory if you will.

LAN servers are set up with an rsync script that syncs the 'coursepickups' from HUB to a directory outside of Moodle in each LAN server called 'toinstall' - that's a one way sync ... HUB -> LAN's.

Upon completion of the syncs, the LAN server executes a 'moosh' script to install the courses and enroll users from the backup files that are in 'toinstall'.

Upon completion of all the above the 'toinstall' diretory is cleaned up ... and on the HUB server the 'coursepickups' is also cleaned up.

Whole process repeated every week ... but would think that something daily might prove to be too much for networking issues ... am thinking the only reason one would build such a setup is limited circuits upstream (1 T1 could get easily saturated with traffic or the circuits are not reliable (drop packets/outages in route, etc.

We've not mentioned platform upon which the systems will run, but would highly recommend a LTS Linux of some flavor.

Can see potential for real issues when/if, for example, there is a networking issue between one of the LAN locations and the HUB server ... where LAN A and B were able to complete but LAN C couldn't due to electrical issues/network issues/other.

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Ken Task -
Picture of Particularly helpful Moodlers

In the description of this project, it was originally asked if there was a way
to 'push' new courses created on a HUB instance of Moodle to a LAN moodle.
Suggestion made by me was totally scripted and done in stages.  It also
would be one way ... HUB -> LAN.   In a discussion with the poster seeking additional information (a Google Chat), it was disclosed that the description was
a little in-accurate and what was desired was a two-way sync.
New courses developed on the HUB instance would automagically appear on the LAN
instances, but also, the student use of those courses would be sync'd back
to the HUB instance.

Have been digging into this a little more.  It is possible to sync the DB's:

The way to accomplish it ... master/master setups of the DB servers and the DB for each LAN ... The HUB server having all instances of the LAN servers in subdirectories - each with their own code directory and moodledata directory.

HUB LAN A DB -> LAN A DB
LAN A DB -> HUB LAN A DB

For an idea:
https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication

The catch 22, however, is how to sync the moodledata directories - actually only filedir

Possible solutions:
https://www.cis.upenn.edu/~bcpierce/unison/
https://github.com/dooblem/bsync

And the timing between the DB syncs (which happen as soon as one or the other does anything with the DB) with the data directory syncs?

Another catch 22 ... would raise it's ugly head (maybe) when and upgrade
to moodle code needed to be performed.  All sites would need to be same versions of
code and any/all plugins/addons would have to come from a 'master' me thinks.

catch 22 #3 ... the $CFG->wwwroot variable in config of Moodle.

Possible solution ... creators of the content located at the HUB would have to use 'stealth' settings in their host file to access the LAN A HUB server by a FQDN.  Once their work was done on the LAN A HUB server, to check the results of their work appearing in LAN A site, the removal of the 'stealth' entry in host file of the workstation would need be reverted - commented out.

With such complication administrators of the systems would have to be on top of any hickups and the connectivity of all the servers to the wider internet needs constant monitoring.   Plan for a hickup and how to recover from that hickup.

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Central Moodle instance sync to local instance - updated

by Paa Kwesi -
Hi Ken,


Thanks a lot. My team have benefited a lot from your input.


Best regards..

Average of ratings: -
In reply to Paa Kwesi

Re: Central Moodle instance sync to local instance - updated

by Sys Admin -

Hello Paa Kwesi,


Sorry for the off topic response. I was just curious what tool you used to create this beautiful diagram.


Thanks!

Average of ratings: -
In reply to Sys Admin

Re: Central Moodle instance sync to local instance - updated

by Paa Kwesi -

Hi Sys Admin,


Thanks for the complement. The diagram was designed by a colleague in my team. The tool that was used is Adobe Photoshop CC.


Thanks..


Average of ratings: -