Periodically transferring information from a production moodle instance to a moodle instance with historical information

Periodically transferring information from a production moodle instance to a moodle instance with historical information

by CARLOS MARIO JARAMILLO OLAYA -
Number of replies: 9

Cordial greetings to the entire community, in advance I thank you for your enormous work in giving birth to the millions of employees who bet every day on Moodle in the transformation and enrichment of education in our institutions. Now, I have the following need and I would like you to be able to guide me technically how I can meet it:

1) A Moodle instance is required that will contain purely historical information about the institution.

2) There is a production Moodle instance that has data retention policies of 2 years, that means that when 2 years of information is completed, the oldest year must be transferred to the historical Moodle instance.

Is there an automatic mechanism that can do what is required? or if it is through scripts or things so I must investigate or take into account to carry this out?

Additional info: the 2 instances of Moodle have the same version which is 3.5. Each instance is in different places (they do not share the same server).

I remain attentive to any help to fulfill this requirement. Thank you very much.

Average of ratings: -
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Visit the Moodle docs about "migration."

Also, there has been some similar discussions on this topic on Moodle.org.

Some of what you want to do depend upon if you want to "accumulate" courses in your backup, or simply have backups that can be opened.

Average of ratings: -
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

You could run an https://docs.moodle.org/35/en/Automated_course_backup, before that giving it a target directory under Site administration > Courses > Backups > Automated backup setup: Save to. Also tick Use course name in backup filename.

Now you need a list of the course IDs of the courses to be archived to transfer those backups to the historical server. Once with the help of Moodle Shell https://moosh-online.com/ you can restore them in to the Moodle.

A side-effect, you can also archive the bundle of Moodle backups, for a second line of security.

Question: The historical server is planned to run forever? Won't it be full one day?

N.B. Saw that you've posted the same question in the Spanish forum: https://moodle.org/mod/forum/discuss.php?d=412530. It would be good, if you put a note there saying your problem is being followed up here, in the Engl. forum.
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by CARLOS MARIO JARAMILLO OLAYA -
Thank you very much for your reply. I want to add to the guidelines of the requirements that in our institution there are legacy systems with historical information that have more than 20 years of information and at the moment the administration does not plan to limit this. Also keep in mind that the historical moodle instance is always online for consultation.
Therefore, the big questions that came to my head when addressing this requirement is how to merge the historical moodledata with the fragments that arrive each year from the production moodle? How to merge the data from the production database with the historical database?
Average of ratings: -
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Ken Task -
Picture of Particularly helpful Moodlers

Moodle 1.0 was released 20 Aug 2002.
For you to have 20 years (an entire generation) of data (in a moodle) will be reached in 2022.

Has your site been marched from version 1.0 to current version?
That's 18 years of data on users in Moodle!!!   Really?

Had a corp client that mistakenly thought they had 10 years worth of student data.

But, they forgot they did things like auto-un-enroll and allow accounts to expire due to lack of use.   I found one user in the mdl_user table that was first created 8 years ago ... but no data for that user ... anywhere.

You might check your moodle's user table ... find the oldest account.
See what data you have on that user.

Even if you cranked up a new moodle every 2 years - that's 10 versions of moodle on very old machines which better be accessible only to internal network and nothing from outside.

There is a major uni on the west coast of US.   They do new moodle every year - what ever is the highest version of moodle.   New server too ... something like ol2014.unidomain, ol2015.unidomain, ol2016.unidomain.

Teachers still have access to their courses (non-edit) ... students that were enrolled still have access but read only.

Unless I totally mis-understanding, the suggestion to archive course backups (full user) might be best approach ... course you'd have to have the ability to restore.

Dunno!!!   Can see issues.

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by CARLOS MARIO JARAMILLO OLAYA -
Thank you very much for your answer, but unfortunately I think I did not make myself understood, because when I talk about legacy systems I am not talking about moodle, but about any other, and I do not agree that a new instance of moodle is created every year. In our institution we have been working with moodle for 10 years and we cannot eliminate the historical information, that is why I am requesting the help of the ideas of all of you to be able to have a stable ecosystem between a moodle instance for historical info and a production instance that each year, the oldest year is transferred to the historical instance. Happy day.
Average of ratings: -
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
I am a little puzzled. If you want to archive all courses into one backup Moodle, and if this will happen for at least 20 years, the space required for your backup Moodle will be large. You mention that you plan to remove courses from your production Moodle after they are a little old, for retention. But why not simply keep all courses in your production Moodle, and periodically make a full backup of your production Moodle? The backup can also be restored on a server so that you can access old courses, as needed.
Average of ratings: -
In reply to Rick Jerz

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by CARLOS MARIO JARAMILLO OLAYA -
Thank you very much for your reply. Starting in 2020, we had the production instance that way, that is, all the information in one place, but with this pandemic situation, the entire institution focused on virtuality and the directives made the decision to leave the production instance as lightest possible to have the best performance for the 32k users being served. Again, I thank you for your contributions, but there is already a decision made on what to do, so I am only asking for your support in the technical part to try to achieve it. Thank you very much.
Average of ratings: -
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Technical advice to drive in to a corner? Not necessary. Just close your eyes!
Average of ratings: Useful (1)
In reply to CARLOS MARIO JARAMILLO OLAYA

Re: Periodically transferring information from a production moodle instance to a moodle instance with historical information

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Okay.

Based upon what you have said, I would:

1) Migrate the current production moodle to a backup server.  This could even be the same server.  You could adopt a good naming convention, such as moodle_old for your database, moodle code, and moodledata items.

2) Remove current courses from the backup.

3) Remove old courses from the production moodle.

4) From this point forward, make a course backup, with students, each semester. Restore this backup into the backup moodle.

5) When you upgrade your production Moodle, also upgrade your backup Moodle.  Try to keep your backup Moodle "in sync" with your production Moodle.

Keeping the two Moodles "in sync" can become a bit challenging as time goes on.  For example, if you remove a plugin from your production Moodle, that plugin might be needed for an old course.

I happen to have several backups, one called moodle19, and one called moodle29.  I can get to any course that I created under these moodle versions.  And I also have a moodle3 that mirrors my current production Moodle.  My thoughts are that eventually, I will begin a new moodle4, probably next year when Moodle 4.0 is released.  Since my moodles are small, these backups are on a MAMP installation.  1.9 and 2.9 require PHP 5.6, whereas 3.0 requires PHP 7.  But I can easily switch between these PHP versions within MAMP, when needed.

Average of ratings: Useful (1)