Done with backup/restore

Done with backup/restore

by Henry Quinn -
Number of replies: 16

I am at my wits end with Moodle.  The backup/restore function simply doesn't work.  I don't care about the reasons anymore.  I've spent a lot of money upgrading disk space and virtual server resources to no avail.  I've upgraded the PHP versions and tried everything I can read about on here.

I also can't just upgrade Moodle to the latest version because I needed a lot of customization done by a "Moodle partner" from another country who has gone totally radio silent.  As far as I can tell, I'm running Moodle 3.4+ (build 20171228) although I can't find anywhere in Moodle that just says "You are running Moodle version x.xx".  If I just overwrite core PHP files with a new version, it'll destroy the thousands of dollars I've spend on this Moodle Partner, who hard coded the changes I needed into the core PHP files.  

So, is there a manual way to backup/restore courses?  I'm using lots of SCORM and it seems to be getting file sizes between 1-2GB in size.  



Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
What exactly are you trying to achieve?
Average of ratings: -
In reply to Emma Richardson

Re: Done with backup/restore

by Henry Quinn -
Trying to backup a course, then restore it. Using backup is a workaround for a non-existent "clone" function.

Since the built-in backup and restore functions are not reliable, I'm overall trying to clone a course. Up till now it's worked maybe 25% of the time and when it does finally work, its for unknown conditions. Last time it worked, I had tried and tried but failed... then I left it alone for days... and when I tried again, the restore function worked. The same thing for the Backup function. Totally unreliable.
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

Version ... Site Admin Menu, Notifications.

OR from command line:

cd /path/to/moodle/code/

fgrep '$release' version.php

So all attempts to backup a course fail?   Command line also?

cd /path/to/moodle/code/admin/cli/

php backup.php has help.

Probably best to save the backup to a designated directory rather than in the 'sea of files in moodledata/filedir/' so you can see them ... even un-compress them there so one can also see what might be affecting a restore.

So debugging shows nothing?   either backup or restore via Web interface?

Now the nitty gritty ... do you have any documentation provided by this 'radio silent MP' concerning customizations?

Even more 'nitty gritty' ... and this might make the MP break 'radio silence' .... who is the MP?

As far as a 'fix' that anyone in community might be able to provide, community does have an 'autopilot' ... so afraid you are gonna have to provide a lot more info! :|

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Done with backup/restore

by Henry Quinn -
"Version ... Site Admin Menu, Notifications."
Found it. Makes no sense to be there! Not intuitive in the slightest bit. Perhaps they need to have a section called "About" or "Version Info".
At the bottom it says: Moodle 3.4+ (Build: 20171228)

I'll try command line... it's on a hosting provider server so I need to find out if they have SSH interfaces.

Debugging shows nothing... it just seems to hang at some arbitrary % of completion. This happens with backup and restore. Occasionally they do work but it's totally random when it decides to work.

I asked the MP for documentation but they are more focused on getting code done as quickly as possible. I don't think I should give their name out since I made them sign an NDA and a contract.

I'm thinking the fix is that I'm going to have to write my own highly specialized system that takes input from SCORM. I've been learning PHP myself and it seems very similar to ASP. My code may not be pretty but in the long run it'll probably be a lot cheaper and more reliable than relying on a third party. I have no idea how to interface with SCORM but I'm confident I could write all the other functions I need myself. I'm not certain I will go this route yet... but is there anyone who could provide a simple PHP class (paid) that I can instantiate to contain SCORM lessons?
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

Hmmmm ... NDA. sad

From what you have described so far (really not much technical detail, but good enough for guesses), sounds like server may not have enough 'umph' for GUI backup/restore.  

So what are server specs .... like memory/space etc.   As well as set up ... all on one server?   Dedicated DB server?

If you find you have ssh access and can install software, consider installing moosh - a cli tool outside of Moodle code.   It has backup as well as restore.

And, just so you know, I admin a K12 server ... there have been courses as large as 130+ Gig .... a few 50Giggers and many are in the 20+ Gig range.   For those, can only do CLI backup.   This to say 1-2 gig ain't that large! ;)

Those large courses I mentioned were backed up ... over a weekend .... and with CLI only with &nohup!

How did you determine courses were of that size?

Sounds also like courses are SCORMS ... I hope they are HTML 5 compat and not still SWF.

Are you using file system repos for those SCORM's and linking via 'aliases'?

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Done with backup/restore

by Henry Quinn -
Good news. I was able to effect a backup using CLI. It wasn't verbose at all but it did report a success. I'm now attempting a GUI restore to a new course and it's stuck at 13.91% even after 30 minutes.  Incidentally, during the CLI backup, I couldn't specify my own backup directory since the server has some unique permissions applied but I was able to make it work using the defaults (specifying no command line switches for backup's destination). Hopefully restore works the same way.

I'm glad to know that you are working some BIG projects! I'm using SCORM modules because I can't get the same kind of high quality content without it.

I determined the course size based on the size of the backup files. Sometimes it freezes for hours but, if I return to the main page of Moodle, then navigate to the backups section, the .mbz file is magically there. I don't know if it actually finished writing the zip file correctly though. The earlier today backup was 1.9gb and the one I just did via CLI was also 1.9 GB.

Yes.. the SCORM courses are HTML5 compatible. It's really nice to have something using an industry standard that is universally supported.

"Are you using file system repos for those SCORM's and linking via 'aliases'?"
I really have no idea what this means smile 
I'm uploading ZIP files of the SCORM modules into a SCORM activity. I think Moodle is unzipping them and storing the files however it does it. The ZIPs are standard SCORM modules with the manifest file and all resources in the appropriate sub-directories. 

Here's the server info... I think there's about 70GB of disk space free.
===========================================
Server Information
Processor Information
Total processors: 40
Processor #1
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #2
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #3
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #4
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #5
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #6
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #7
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #8
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #9
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #10
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #11
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #12
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #13
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #14
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #15
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #16
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #17
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #18
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #19
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #20
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #21
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #22
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #23
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #24
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #25
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #26
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #27
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #28
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #29
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #30
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #31
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #32
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #33
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #34
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #35
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #36
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #37
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #38
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #39
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Processor #40
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Speed
2500.017 MHz
Cache
25600 KB
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

Good to hear you have a backup via command line.

Am willing to bet, you could designate a directory if you took enough time to find out. ;) 

When you first login via ssh, do a 'pwd' command.

That shows your location ... a path.  Might be like:

/home/customername/

You might then be able to use something like:

php backup.php --courseid=2 --destination=/home/customername/

Name of the backup file can be seen then ....

example: backup-moodle2-course-1-37-20191111-0158.mbz


as well as it's size ... like:

ls -l /home/customername/backup-moodle2-course-1-37-20191111-0158.mbz

How about some simple CLI commands to find info about server?

ssh into server.

Issue:

uname -an [ENTER]

See if you have access to 'top' ... whereis top [ENTER]

If that command shows a path like /usr/bin/top then issue

top [ENTER]

Copy and paste to a text file on your workstation the 'stuff' at the top of top output ... which is realtime info.

Shows memory usage.

Press q to quit top.

Diskspace ...

df -h

You didn't sign a NDA with provider did ya?  So where is server hosted?   With what provider?

'SoS', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: Done with backup/restore

by Henry Quinn -
What is the commandline "restore" function? I see there's a backup.php but the php file list includes nothing obvious for restore....
Here's all the files inside of \cli:
fix_orphaned_question_categories.php
../ install_database.php
alternative_component_cache.php install.php
automated_backups.php kill_all_sessions.php
backup.php maintenance.php
build_theme_css.php mysql_collation.php
cfg.php mysql_compressed_rows.php
check_database_schema.php mysql_engine.php
cron.php purge_caches.php
error_log reset_password.php
fix_course_sequence.php svgtool.php
fix_deleted_users.php upgrade.php

I'm hosting at Inmotionhosting.com.
=====================================
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/ploop14116p1 148G 77G 64G 55% /
devtmpfs 768M 0 768M 0% /dev
tmpfs 768M 0 768M 0% /dev/shm
tmpfs 768M 384K 768M 1% /run
tmpfs 768M 0 768M 0% /sys/fs/cgroup
=============================================
Here is what top shows:
top - 19:05:39 up 45 days, 23:13, 0 users, load average: 0.35, 0.30, 0.37Tasks: 76 total, 1 running, 74 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
KiB Mem : 3145728 total, 975028 free, 415656 used, 1755044 buff/cache
KiB Swap: 3145728 total, 2716532 free, 429196 used. 2541806 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
902 mysql 20 0 3437120 91356 6008 S 0.3 2.9 207:55.45 mysqld
1 root 20 0 195020 3040 1876 S 0.0 0.1 43:18.34 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrea+
3 root 20 0 0 0 0 S 0.0 0.0 0:23.47 khelpe+
122 dbus 20 0 58180 1328 828 S 0.0 0.0 17:34.22 dbus-d+
132 root 20 0 146232 516 36 S 0.0 0.0 0:06.90 pure-a+
139 root 20 0 74404 616 4 S 0.0 0.0 0:00.00 saslau+
140 root 20 0 74404 612 4 S 0.0 0.0 0:00.00 saslau+
168 root 20 0 126316 656 396 S 0.0 0.0 1:26.80 crond 169 root 20 0 110060 136 4 S 0.0 0.0 0:00.00 agetty
170 root 20 0 6468 132 4 S 0.0 0.0 0:00.00 agetty 171 root 20 0 25860 264 208 S 0.0 0.0 0:00.06 atd
598 root 20 0 45408 6096 2260 S 0.0 0.2 1:09.73 queuep+ 717 root 20 0 148768 356 148 S 0.0 0.0 0:08.84 pure-f+
718 root 20 0 24996 300 108 S 0.0 0.0 0:00.00 xinetd
719 root 20 0 16552 1032 720 S 0.0 0.0 2:06.99 dovecot
772 dovecot 20 0 10088 480 356 S 0.0 0.0 1:54.33 anvil
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

Moodle doesn't have a restore.php script.   But you are in luck cause moosh does!

Restore done via moosh

https://moodle.org/plugins/view.php?id=522
https://moosh-online.com/
https://moosh-online.com/commands/

From last link above:
course-restore

Restore course from path/to/backup.mbz to category or existing course.

Example 1: Restore backup.mbz into category with id=1
moosh course-restore backup.mbz 1

Example 2: Restore backup.mbz into existing course with id=3
moosh course-restore -e backup.mbz 3

One line script looks like:

moosh -n course-restore /path/to/backupmbzfile/to/restore/restorethisone.mbz 20

The 20 on the end is a category.  Category on your site should be hidden ... like a category called 'staging' and it's ID number is 20

Did this very thing for a gentleman who does things pretty much like you want to do them.

PM me and I'll give you his contact EMail address.  Think he can vouch that it works and works without error/fuss/muss.

'SoS', Ken



Average of ratings: Useful (1)
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

What's your 'workflow'?  And reason for backing up courses only to turn around and restore them.

That for starting basically the same course only for different students?

One could make a course template that contains no students, back it up to a location outside of Moodledata, and then use moosh's restore command to restore that template course to a new course(s) in a category.  Minor changes to those template courses would be needed, but that script using moosh could restore multiple templates much faster.   Did make something like that for a user some time back.

In looking at your previous postings, appears your courses are 100% scorm - all activities (except for certificate) are SCORM's.

OR ... ever hear of 'reset'?

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Done with backup/restore

by Henry Quinn -
What I envisioned is to create a course then clone it for each different audience group. The audience groups must have absolutely no interaction whatsoever. Also, I have a series of custom reports dependent on each audience group getting their own course in Moodle. So when I get a new audience group, I backup then restore an existing course into a new one, then I make slight modifications for the new audience group.

Yes... correct. The courses are basically all scorm. I'm using "labels", "SCORM containers", and "Simple certificates" with a few graphics thrown in to make navigation more user friendly. In fact, I had to have the Moodle Partner make GUI modifications since I was having so many usability issues with audiences.

Reset what? I'm not sure what you mean precisely...

Do you know of anyone who could create some SCORM classes (in the programming context of the word) that I could include in PHP code that I write myself?
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Colin Fraser -
Picture of Documentation writers Picture of Testers
Sorry guys, I have been a little busy, and Ken, you may be able to supply more information here than I can, or I may be on the wrong track, but Keanu, have you considered using Meta Courses? I've never tried them with a completely SCORM based course, but they should work. This way, you only need to create new courses with materials and resources when you are tweaking a course, or intrducing a new course, if you have the original SCORM files, of course.
Essentially, the meta course is a parent and Users are in a child course. Each child course is completely separate from each other child course and you can have any number of child courses linking to one parent course. More information about Meta Courses can be found in Moodle Docs.
If it's workable, this should save a lot of messing about and there is no reason to think it's not workable in this circumstance. 
Setting it up is not difficult, maintaining it should be simpler, saving you a lot of time and no small amount of anxiety. 
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Colin Fraser -
Picture of Documentation writers Picture of Testers
It has been a while since I have written any SCOM modules, but I still have a program to do so. Have to reinstall it and see if it works. I am going to be busy for the next six to eight weeks, so if you're in no real hurry, perhaps we can talk about what you want.
Average of ratings: -
In reply to Henry Quinn

Re: Done with backup/restore

by Henry Quinn -
Ken, I want to give you my sincere thank you for helping solve this most daunting problem. I was very very impressed by the timeliness and "resolution" of your replies. One of the biggest obstacles of open-source software adoption is the lack of good documentation and support. One of the better examples are Linux Mint, Ubuntu Linux, and the documentation found at Digital Ocean. Like those, you were very detailed to always answer the who, what, when, and where. I appreciate your detailed answers in that respect. I was able to connect the dots only because of this assistance.

For the benefit of others, if you have large SCORM modules or large courses in general, don't bother with the Moodle web GUI. Go straight to command line as Ken is doing. There is a backup.php script and via another plugin called "moosh", a restore PHP script. Commandline overcomes the execution limitations imposed on web CGIs. Moodle will error out in countless variations which are not verbose. It will just hang. But, if you use these commandline functions, that will not happen. Without Ken's mentioning of this, I could not have accomplished what I needed to do.

Ken, you ought to be making design choices in Moodle! Your expertise is self evident and I can tell you've been in the education industry for quite some time.
Average of ratings: Useful (2)
In reply to Henry Quinn

Re: Done with backup/restore

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for kind comments.  And congrats on moving forward! ;)

One of the best things about being able to actually talk (we live in close proximity - about 30 minutes away) is exchange of information.   Got a lesson,advice, and first hand experience with a thing called Signal.   Thanks!

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Done with backup/restore

by Henry Quinn -
I couldn't have done it without you. This was all you!

I can't wait to continue to share more Linux and technology knowledge! Did you every do electronics? Remember those Radio Shack electronics books written by Forrest M. Mimms III? He lives here too... although I've never mustered the courage to meet a childhood favorite author.

If you're interested in aviation, you should volunteer to get an ADS-B receiver that you can mount on your roof. They give you a free subscription to FlightAware that lets you see and interrogate aircraft anywhere. The reason they give it to you for free (the equipment) is because it augments their network. If you're into Ham radio or anything similar, ADS-B is in the same realm of interest.

I'm so glad that you have so much Linux knowledge... I've only recently begun using it so we have much to talk about. If you're interested in RAID, I can help you with MDADM.
Average of ratings: -