Legacy - an experiment in converting a Legacy course into NON-legacy course.

Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Ken Task -
Number of replies: 6
Picture of Particularly helpful Moodlers

This not really a request for assistance or help ... but the results of an experiment in eradicating legacy from a course.  * do not do this on production server * ... tinker with it first.

Took a backup of a course (1.7gig) from a site that began as a 1.9.highest and
has been marched through series 2 to current version 3.0.hghest via git.

Made a no user no quiz backup and downloaded.

Un-archived the .mbz

Edited only the files.xml file in a text editor

Search for the tag for legacy replaced with draft.

Saved the files.xml file

Rebuilt the .mbz file.

SCP'd the course to a filesystem repo on another Linux server running Moodle 3.2.highest.  File system repo already mapped and linked in the 3.2

Restored the course via moosh. Executing moosh in the moodle32 code directory:

moosh course-restore /var/www/unirepo/backup-nolegacy-20170405-1920-nu.mbz 1

[root@sos moodle32]# moosh course-restore /var/www/unirepo/backup-nolegacy-20170405-1920-nu.mbz 1
Creating new course to restore backup
Restoring (new course id,shortname,destination category): 9,RJMC,1
New course ID for 'RJMC': 9 in category 1

1 above was the category.

* Course restored * - no errors.

Logged onto the Moodle and inspected the course via GUI.

Legacy was on ... Turned off Legacy

Results:
img missing from Topic descriptions - they were small icon type images in front of Topic descriptions.  (P:off might work on those - pics provided)

The real bottom line ....
ALL the other file resources which were link to files stayed/displayed
Could be opened/viewed/downloaded.

Course no longer in legacy!!!!

This was only one course ... think how the courses were built originally
would be a factor ... so can't say this is 100% ...
*BUT* 'lions' share of eradicating legacy from this course was accomplished and what this non-programmer did could be replicated by anyone not afraid of the command line.

Thanks to Justin Hunt and his 'poof' ... got the idea from inspecting the code of that addon.

Some images attached cause the descriptions above brief (only three ... that's max)

'spirit of sharing', Ken

Attachment legacytopicimgs.png
Attachment missingicons.png
Attachment mooshatwork.png
Average of ratings: Useful (1)
In reply to Ken Task

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Ken Task -
Picture of Particularly helpful Moodlers

Just to follow up ... moosh also has 'switch' to see course info. 

Used Moosh to compare the two courses ... course with legacy on one site ... course without legacy on the other site.


The test done with a No User, No Quiz backup of the legacy course.  Because the original course used the old certificates module and SimpleCertificates, a NO USER backup didn't include those certs.

The important items related to legacy files are noted via ****

From original site with Legacy:

Course ID: 25
No of contexts: 139
Context by level:
    CONTEXT_MODULE (70):    135
    CONTEXT_BLOCK (80):    4
Context by module:
    forum (5):    1
    label (9):    32
    quiz (12):    10
    resource (13):    85
    certificate (18):    1
    folder (21):    1
    page (24):    3
    simplecertificate (28):    1
     (29):    1
Number of role capability overwrites: 0
Role capability overwrites by context:
Number of local role assignments: 0
Locally assigned roles by context:
Number of enrolled users: 25
Users enrolled by role:
    manager (1):    25
Number of groups: 0
Group statistics:
    Min number of members in a group:    0
    Max number of members in a group:    0
    Avg number of members in a group:    0
Course modinfo size: 253459
Number of sections: 21
Section statistics:
    Sections visible:    21
    Sections hidden:    0
    Min number of modules in a section:    0
    Max number of modules in a section:    19
    Avg number of modules in a section:    6
Number of grades: 143
Number of log entries: 2923
**** Number of files: 239
Total file size: 343768444
Cache build time: 1.2794678211212

From site where course restored (no user no quiz):

Course ID: 9
No of contexts: 126
Context by level:
    CONTEXT_MODULE (70):    122
    CONTEXT_BLOCK (80):    4
Context by module:
    folder (8):    1
    forum (9):    1
    label (12):    32
    page (15):    3
    resource (17):    85
Number of role capability overwrites: 0
Role capability overwrites by context:
Number of local role assignments: 0
Locally assigned roles by context:
Number of enrolled users: 0
Users enrolled by role:
Number of groups: 0
Group statistics:
    Min number of members in a group:    0
    Max number of members in a group:    0
    Avg number of members in a group:    0
Course modinfo size: 237778
Number of sections: 21
Section statistics:
    Sections visible:    21
    Sections hidden:    0
    Min number of modules in a section:    0
    Max number of modules in a section:    18
    Avg number of modules in a section:    5
Number of grades: 0
Number of log entries: 28
*** Number of files: 236
Total file size: 343214021
Cache build time: 0.23033213615417

**** Number of files difference: 3

Notes:

* original site had certificates and simple certificates and with a no user backup
those were not included in the backup thus those modules don't show in no legacy restore.

* think a lot depends upon how the original course was built.  Wouldn't expect 100%, but heck, if I could get the majority of files out of legacy, there is a gain.

* also note it doesn't reduce course size much.  Can only attribute that to hidden/unused links to resource files in the original course.

Then again ... considering the new file system ... is one jumping out of the pan and into the fire?

Only the course creator/Moodle Master could answer that one! ;)

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Ken Task -
Picture of Particularly helpful Moodlers

Videos and Readme for 'command line poof' (clipoof) ... a bash shell script that requires tar, gzip, sed, and wget.  Mac's have those as do Linux boxen.

https://drive.google.com/drive/folders/0B5gmU8YqbNJQVFN4eUNiNHp0dWs?usp=sharing

Can't claim that it's 100% because of the un-knowns in how a course was built and with what.  Only course designer would really know.   Hopefully, however, clipoof will so some 'heavy lifting' for ya! smile

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Andrew Normore -

Hi Ken, great guide you have here.

We are in a serious jam trying to move to a clean Moodle 3.3 install, coming from Moodle 2.6.

The problem is: restored courses linking to files are broken.

Earlier, you mention this:

"SCP'd the course to a filesystem repo on another Linux server running Moodle 3.2.highest.  File system repo already mapped and linked in the 3.2"

Are you suggesting that we must scp / rsync our /var/www/moodledata folder over to the new server?

Our Moodledata folder is 720gb from a decade of use, we want a clean start, but have hundreds of courses that would need file structure rebuilding.


Thoughts? smile

Average of ratings: -
In reply to Andrew Normore

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Ken Task -
Picture of Particularly helpful Moodlers

Ouch ... 2.6 to 3.3.  Did one of those last week ... not fun ... marched from 2.6 to 3.3 one version at a time.   Got there but still had legacy. :\

Don't think you an use the moodledata from a 2.6 to a fresh install of a 3.3 and rid the new site of legacy.

So the only route, I think, would be to set the 2.6 to run automated backups to a directory outside of moodledata/filedir/ ... a designated directory.   That way you can see files by humanly re-cognizable names.

Also, one would have to un-archive one of those backups to a test directory and see if files.xml has 'legacy' tags.   Bad news is ... it would mean all those backups that have legacy tags related to files would be restored to the 3.3 with legacy still in effect.

The only way I've found to rid backups of legacy is to edit files.xml and replace 'legacy' with 'draft'.    Then re-archive the backup and restore.   Still missed some files ... but a few is better than 100's and legacy was gone.

As far as getting them to the new server ... on the new, create a file system repo whose directory is outside of Moodle but in an area where apache can see the directory used for the rep;o ... I have CentOS so /var/www/unirepo is a good place.

Scp all the backups from the designated directory on the old system to /var/www/unirepo/ on the new (3.3).

In one course, setup the file system repo to use /var/www/unirepo/   Restore from there ... iF you are fast, one could restore 15 courses in an hour ... but would probably be brain dead. :|

However, on the new system install moosh and using moosh through a looping bash shell script one could restore all courses in /var/www/unirepo/ to the site.   Recently did that and it was intensive but restored about 140 courses in about 2 hours ... and, wasn't brain dead afterwards.

As far as what you are seeing ... broken file links ... might depend on how the courses used links to files when it was a 2.6 ... did the site have a file system setup and did they use 'alias' when linking?

If the broken file links are to things like dropbox, is new site setup appropriately with same dropbox location?

Think, if there wasn't a standard when the site was a 2.6 for linking to files, one might have to inspect every 2.6 course backup .... not sure how to do that!

'spirit of sharing', Ken

Average of ratings: -
In reply to Andrew Normore

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Andrew Normore -

BIG RED FLAG ON MOODLE 3.3

https://tracker.moodle.org/browse/MDLQA-10609

Ken, the issue you report "Legacy files not restoring in to course" is very similar to "External repositories not restoring in to course"

The difference being... one is an official moodle bug by the looks! Trying out Moodle 3.2 now to see if this bug is present.

Average of ratings: -
In reply to Andrew Normore

Re: Legacy - an experiment in converting a Legacy course into NON-legacy course.

by Ken Task -
Picture of Particularly helpful Moodlers

That link takes one to a entry for 3.3 QA.  3.3 is now: $release  = '3.3.2+ (Build: 20171027)' in the latest git pull.

The 3 files missing in my experiment were due to how the author of the course put an image next to a Section heading ... if I re-call correctly.  Author had created "icons" that had been used in other courses.   Author found the URL to the 3 images and provided URL's to them in other courses.  Just happened that the one course I choose for the 'clipoof' experiment was one of those.   Those 'icons' got lost due to course id changes ... contextual changes ... etc.,  when restoring the course after 'clipoof'.   All the other regularly linked files worked.   And legacy no longer required.

Mind you it was just one test backup that I tried 'clipoof' on.   Would imagine that other courses run through 'clipoof' might show different results.   Still though ... looking at 100's vs a fewer number is a gain.  Guess Legacy will be supported in Moodle forever so it's a matter of how much of a perfectionist one is.

Just last week I did a march from 2.6 to 3.3 and when arriving at destination version, discovered they had somehow deleted the admin block (customized theme).   Admin block included the course admin block.  No way to add the admin block back to courses that were using the clean or more theme, but everything worked with the new boost theme.     Looked at tables related to blocks and could never decipher how to get the admin block to appear for all courses regardless of theme chosen.

Had done a run of automated backups of the site right after arriving at 3.3 and saved to a directory outside of Moodle.

So, installed a fresh 3.3.highest then used moosh through a looping bash shell script to restore the majority of the courses.    In all courses, course admin menu appeared in any of the stock 3.3 themes ... boost, clean, and more.

Legacy files, however, were still there - in some courses.

The 'customer' was pressing to 'upgrade it' ... to some it's a matter of $ and 'as long as it works' ... ie, didn't want to pursue removing legacy from the site.   Granted don't think I've ever seen a Moodle site that was '100% perfect'.

'spirit of sharing', Ken


Average of ratings: -