Server Mirroring Question

Server Mirroring Question

John Smith -
Atsakymų skaičius: 7

Hey everyone,

I am looking into a solution but would appreciate your insight before I start implementing it, since the last time I went for it directly without consulting first, I wasted a lot of time on something that was technically not feasible. My issue is similar to this thread (but not that same): https://moodle.org/mod/forum/discuss.php?d=62917

Here's a summary:


Use case:
- Limited connectivity in a school (bandwidth is available, but speed is very limited)
- Students are using tablets and require internet access for 2 reasons: (1) download content on the moodle mobile app for offline use (2) to use moodle during classroom hours

Solution (potentially):

  • Create an on-premise mirror server that mirrors the content on the remote server (we are primarily interested in H5P + audio files)
  • This server will serve content from the mirror server (over the local school network), and forward all other requests to the origin server. It will not be a full duplicate of my original moodle server
  • This server will use Rsync to remain synced to the origin server. Content changes are made approximately once or twice a week (few file changes, nothing too complex, so I think Rsync should suffice)

Some hypothetical questions:
- Does this method, in theory, work? I am primarily concerned about the location of this content that will be mirrored. I'm not familiar enough with how/where this content exists (moodledata or default moodle directories - or both?). Also not sure where/how I should store this content on the mirror server. I think a full clone of the moodledata may cause some issues, but I'm not very sure.

- Will this solution also have an effect on how quick this mirrored content loads when simply browsing through lessons on the mobile app and web version? Or will this only have an effect on "download requests"?

Any additional notes/considerations are appreciated. 

Įvertinimų vidurkis: -
Atsakymas į John Smith

Re: Server Mirroring Question

Ken Task -
Particularly helpful Moodlers paveikslėlis

Qualifications for this response ... none - am not a developer - thus think in terms of existing and least.

Brief description (Note: I have done something like this):

In your content development server, create the new content.   After checking that new content (it works), do a backup of just that new content to a moodledata fileystemrepo called 'newcontent'.  That will be a small .mbz file.

One then rsync's the .mbz files in 'newcontent' to the various schools - A, B, C, D to their matching moodledata filesystem repo 'newcontent'.   Do that on a weekend.

On Monday AM - school admins know that there should be new course content to restore to existing courses.   They do that manually - that's not a guruish thing to do.

My 2 cents! besišypsantis

'SoS', Ken

Įvertinimų vidurkis:Useful (2)
Atsakymas į Ken Task

Re: Server Mirroring Question

John Smith -
Hey Ken, thanks for taking the time.

- The problem is that the content is too large, there are content for KGs up until the 9th grade so will this method also work in this case?
- After I have the .mbz file, I didn't really understand where this file should go in the on-premise server. If we restore this .mbz course file aren't we just copying the entire course to that on-premise server? So in other words we are creating a full server clone that is just physically closer to the school? (also curious about your take on Main/helper servers deployed around different schools. We think we should avoid this due to security and maintenance reasons, but are not completely ruling it out just yet)

We only want to feed the on-premise server content the such as images, mp3, h5p and allow students of that school to get served from it (while the rest of the requests are forwarded to the origin server). We tried to do this with caching, but are unable to get NGINX to cache h5p related requests for some odd reason. So we thought we opt for this other method.

P.S - the content changes are basically just some corrections that are made by our teachers to some courses, so they are not as frequent or significant, so as you said this can be done manually every X amount of time.
Įvertinimų vidurkis: -
Atsakymas į John Smith

Re: Server Mirroring Question

Visvanath Ratnaweera -
Particularly helpful Moodlers paveikslėlis Translators paveikslėlis
Hi

I saw "Server Mirroring" in the subject line and wanted to say there is no such thing: Moodle architecture combined with the underlying LAMP platform is not capable of running two "hot" copies of Moodle. Then I read your post in detail and understand what you want is more "Regularly pushing course content from course development Moodle to production Moodle".

You have already answered what "regularly" means, "Content changes are made approximately once or twice a week". So no big obstacle. The next question is what the "Content" means. You said, "few file changes, nothing too complex". So you are talking of static content, what Moodle calls Resources? Then the procedure Ken explained should work. You shouldn't forget, the strength of Moodle is in its collection of interactive elements, called Activities. There the things get quickly complicated. Can you update an MCQ test already conducted, or an assignment students have already submitted? Most likely not! Then there are whole lot more paraphernalia, no point in going in to details unless you use them. So you need to know what else you will need.

All that said, are you aware of the off-line capabilities of the Moodle Mobile App. I has incredible features nobody expect, like answering a MCS at home (without Internet) and once in the class (connected to the Moodle site) the MCQ results get uploaded to Moodle and graded! Same with forums. Not to mention simple (office) files. Explore them. That could compensate the bandwidth problem the school has.

I must mention another community product which makes a turn-key solution for exactly this szenario. It is called MoodleBox. Note that it is a separate project with its own community.
Įvertinimų vidurkis:Useful (1)
Atsakymas į Visvanath Ratnaweera

Re: Server Mirroring Question

John Smith -
Hey Visvanath, thanks for your response.

1) Just to make sure we are on the same page, our main goal is to provide a faster experience to students in schools with poor internet connection. The issue they face is the loading (and downloading) of the content and static files, so we figured that if we can host this content on school premises in a server, and serve all the requests related to viewing and downloading them. Btw we rely heavily on the offline capabilities of moodle but are limited by the poor connectivity and generally above-average file sizes. That's why a solution like that could help students browse and download all and any content they need much faster.

2) Yes, thanks for the clarification, the content I am referring to are the Resources and some activities (mostly H5P activity, Assignments, Forum and Quiz). Both of these types of content are not frequently updated (they are only modified if there were some mistakes/minor issues). These changes can either be manually synced physically or through Rsync if that works. But my main concern is whether Moodle architecture could allow us to serve these files/resources from one server, and all other requests from another server.
Įvertinimų vidurkis: -
Atsakymas į John Smith

Re: Server Mirroring Question

Visvanath Ratnaweera -
Particularly helpful Moodlers paveikslėlis Translators paveikslėlis
Hi

> Just to make sure we are on the same page,

Very good idea. Currently it is not the case, I fear.

> our main goal is to provide a faster experience to students in schools with poor internet connection

You started by saying:
>>> Limited connectivity in a school (bandwidth is available, but speed is very limited)

Moodle won't make a low speed connection faster! What we all expect is to make the data transfer more efficient. Maybe, that is what you meant by "faster experience".

> The issue they face is the loading (and downloading) of the content and static files, so we figured that if we can host this content on school premises in a server, and serve all the requests related to viewing and downloading them

Kind of obvious, isn't it. I assume the network within the school premises (LAN) is good.

> Btw we rely heavily on the offline capabilities of moodle but are limited by the poor connectivity and generally above-average file sizes

Obviously, if those files are served through that slow Internet link of the school.

> That's why a solution like that could help students browse and download all and any content they need much faster.

Again, What we try is to make the data transfer more efficient. It will "feel faster" because the same data won't be transferred again and again. Think of a link to YouTube against uploading the mp4 to your Moodle.

> the content I am referring to are the Resources and some activities (mostly H5P activity, Assignments, Forum and Quiz)

Today morning they were "few file changes, nothing too complex". Now which is which?

From your questions I gather that you don't understand where Moodle stores its data. It is no easy question either.
- a file: Uploaded to a place called moodledata/filedir. The database stores all the administration information about the file

- MC questions and quizzes: They are in the database alone, unless you have embedded media files in them. Then the are handled as files as in item 1.

- H5P. I don't know, To be more accurate, I am confused about them. A long ago there were in h5p.org, the Moodle just link them. Then came a version where Moodle saves H5P in its own storage. That idea then got a sibling which make them go to the Content bank. I hope somebody who is not confused by all that will tell us.

As I said the first time, the success will strongly depend on the type of resource/activity. Better, you make a list of resources/activities you want to push, then for each discuss individually which type of "pushing" is possible. To remind the previous example, it is not goo do change a MCQ already taken by the students.

You need to study Course backup, Course restore, Import course data, Export questions, Import questions (if the MCQs are relevant). And also the File system repository among other things. And also consult the H5P forum and the Moodle for mobile forums.

P.S. I don't think Moodle developers can help you at the present stage. So the discussion is better suited for the General help forum, I would say. In the past similar discussions ran in the Hardware and performance forum. From the discussion so far, I don't think you have to dig deep.
mirkt
Įvertinimų vidurkis: -
Atsakymas į John Smith

Re: Server Mirroring Question

Ken Task -
Particularly helpful Moodlers paveikslėlis

Define:
"schools with poor internet connection"
in technical/network terms.

There is a term called 'last mile' ... that's the connectivity between the server to
the browser requesting the content.

Student A - uses a local telco connection to the internet.
The server they are going to at the school might be on that same telco network but then again it might be on another network.
On internet, one does not go straight from point A - the students location - to point B - the server unless they are one the same TCP/IP network - probably a 10. wide area network.

So another example: Student A is connected to Telco called 'FastInternet' via their computer which goes through a home router to upstream 'FastInternet'. The local school server is hosted on another network called InternetSchools'.   They are not the same network.
For Student A to get to the server there are other routers - 10 hops away - which is really good, BTW, to get to the moodle server.
The more hops the slower it is.
You, as the moodle server admin, don't control the networking layer of that.

How you could see this:
From a computer where Student a is connected, use a terminal program that can to a 'trace route' and point trace route to the IP address of the Moodle server.
Also, from your server, do a traceroute from the server to the IP address seen by
the moodle server for Student connection.
Those trace routes happen in milliseconds and each packet of info has a time to live built into the packet.  If a packet doesn't reach destination within 30 tries, the packets kill themselves - thus breaking the connectivity between Students location and the Moodle server they are trying to go to.

Moodle content is comprised of the .php scripts that point to files contained there in.
Example: link to an .mp3 audio file in a course.
That .mp3 file is stored in a sea of files moodledata/filedir/ but not by it's humanly recognizable name - cool.mp3 - but rather a content hash value which looks like
sa34dbwerwf334fdsasdjedaghasdfas
a bunch of alphanumeric characters.
Taking the example above:
In moodledata/filedir/ there is a sa directory and inside it there is a 34 subdirectory
and inside that resides a file named for the contenthash: sa34dbwerwf334fdsasdjedaghasdfas
Moodle users see the link to it as cool.mp3 but that name and file extensions comes from  the meta data - the same row in the database that has the contenthash value - a humanly recognizable name ... cool.mp3.

H5P activity is a different beast ... H5P has modules to itself ... like little players
for content types.  The entire activity has an .h5p file name extension, but contained there in could be a link to a 'cool.mp3' audio or the 'cool.mp3' file is included in the .hp5 file itself.

Rsync does files.  It does NOT do specific rows in a table of a database.

Relatively addressed links to large files is possible with moodle, but one has to use file system repo setup in moodle.

Your content development server has a directory outside of moodledata/filedir/ we'll all 'multimedia'.  'multimedia' has sub folders/directories ... audio, video, etc.

Inside those related files ...  all .mp3 files are audios and go into the multimedia/audio/ directory by humanly recognizable names ... don't use spaces/caps etc in the file names - just alphanumeric characters.

In Moodle to provide a link to that file, would be done via Moodle's file system repo which is pointed to '/var/www/multimeais' with subdirectories of audio, video, etc.
The link built in the moodle interface - resource is the file system repo drilled down to the subdirectories and specific file name - important that you use 'alias'  That prevents the multimedia file from being copied into moodledata/filedir/ It then is a relatively address link to a file.

On you school moodle servers ... they would have the same structure ... a '/var/www/multimedia/' directory with same sub directories.   The course backups make then hopefully would in include the alias to the humanly recognized file in the file system repo on the school server.

NOTE: that will speed up access for students ONLY ... ONLY when the are at school. NOT the 'last mile'!

You still do specific content links in backups of courses and add those to existing courses, but you also rsync the /var/www/multimedia/ directory progressively ... meaning you rsync only new multimedia files that don't exist in the school multimedia directory.

Mind boggling isn't it! besišypsantis

If you can find info, check out how Discovery Education (DE) works with local servers - and 'last mile'.  What I've tried to describe above is kinda what DE does.

'SoS', Ken

Įvertinimų vidurkis: -
Atsakymas į Ken Task

Re: Server Mirroring Question

Visvanath Ratnaweera -
Particularly helpful Moodlers paveikslėlis Translators paveikslėlis
Hi John
Cc Ken and others

May be you are overwhelmed by the information overload. That is Moodle, that is community.
wink

Here is another approach which you could adapt. Write an old-style functional specification paper. Here is a guess, say an educated guess based on old discussions in the Hardware and performance forum and the MoodleBox support forum. Consider this a template, edit it and reuse as you want, it is not copy-righted.

0. Introduction
School X is a (brick-and-mortar) school located in a remote place. The kids commute to learn (in-person). The school also has teachers who commute, i.e. physically present during school hours. There is also an administrative staff (the principal and assistants). The administration wants to augment this in-person teaching with a web-based teaching component, aka blended learning.

1. What is already there or immediately affordable:
1.a Content creators outside the school location, all have Internet and prepared to collaborate on a "master" Moodle server in the Internet making and improving the courses

1.b The school has electricity

1.c An in-premise (Moodle) server and network switch(es) and wireless access point(s)

1.d Some local IT staff, who are capable of doing some basic IT stuff like looking after a server, do Moodle site administration, provide user support.

1.e The school and/or kids own Mobile devices (smartphones or tablets)
 
1.f A slow, unreliable, expensive Internet connection

1.g Teachers who are willing to try blended learning.


2. Challenges
2.a The kids don't have Internet at home.

2.b The school Internet is insufficient and too expensive to provide Internet to the kids.

3. Solution
3.a Kids don't need Internet! They interact only with the on-site server, when at school. When they go home the content is still available in the Moodle Mobile App to do their homework, also without Internet.

3.b To download and update the content from the master Moodle in the Internet, the school Internet connection is used sporadically - weekly to nightly.

4. How to download and update the content from the master Moodle.
(This needs to be filled. Here is just a start.)
4.1 Download a new course
The master Moodle has automatic course backups activated. The backups go to its File System Repository. The school Moodle pulls (e.g. rsync) the master repository to its File System Repository. So the site admin can create a new course from the mbz file thus downloaded.

4.2 Upgrade an existing course
Once a course has started at school it can be technically fully rewritten with a newer version, but is never a good idea, since many local work/interactions get lost. So this needs to be handled case by case.

4.2.1 A File resource
- New file resource: Backup the "course" selecting only the File resource. Transfer the backup to the school server as described in 4.1. The local site admin restores the course in to the running course selecting the merge option.

- Update an existing file resource: The method for New file should work.

- Delete a File resource: Has to be done at the school server end.

4.2.2. A Folder resource
(pl. complete)

4.2.3 A Page resource
(pl. complete)

4.2.4 A Book resource
(pl. complete)

4.2.5 An Assignment activity
(pl. complete)

4.2.5 A Forum activity
(pl. complete)

4.2.6 A quiz
(pl. complete)

4.2.6 A question category
(pl. complete)

4.2.7 An H5P activity
(pl. complete)

and so on.
Įvertinimų vidurkis: -