How to I bulk delete Automated backups

How to I bulk delete Automated backups

由Richard Ackland發表於
Number of replies: 61

Using Moodle 2.2.2

Hi, in additional to our standard server backup procedures. I run an automated coursebackup every night.

The backups were being saved to an attached drive.

I have it set to keep backups for 14 days.

All was good until the attached drived got too full - then the backups started being saved into the Moodle Data folder. Now our server drive is full. (no warning given)

It seems that the automated backups were not being deleted.

I have changed the files kept to 1, but only 1 day seemed to be deleted after this.

I now how hundreds of backup files that I can only seem to delete one by one, when I go into each individual course ad go to restore.

As a full time teacher - who runs our e-learning as a side job - I havent got time to do this as it takes ages.

Is there a cron or script I can run to do this quickly. 

regards

 

I am desperate!

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Before anyone could give a targeted response, a little more information would be required ... like operating system and 'attached drive?'.  Would make a difference.

As far as 'direct control' over files in a 2.x, that's one thing removed from Moodle UI for admin/teachers - no interface for direct interaction due to nature  of the new file system (bulk especially).  Now, it's a combination of DB and operating system (manual labor) when it comes to a situation like yours. 傷心

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由Richard Ackland發表於

Ok... Have updated to moodle 2.2.7 it meeded doing anyway.

Using a MAMP server.

The attached drive in this case was a USB. We basically did course backups, so when the odd user, trashed their course - we could get it back quickly - it happens.

Basically the backup up usb drive got full so instead of stopping the files started to save in the DATA_FOLDER.

Looking at the backup logs everything looked fine. It wasn't until ouyr server started complaining that its 1TB hard drive was full I knew something was up.

I have run a new backup and told it to keep just 1 file - hopeing that these would delete all the old ones. It just deletes one file, before adding the new one, so the 20 or so old backups remain.

This is in well over 150 courses.

And each file has to be deleted individually.

This is not good

thanks for replying btw

 

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Following deals only with the backup files that reside outside of Moodle.  The backup files in Moodle involve db manipulation as well as manual manipulation of the 'filedir' (not recommended).

The backup files have a date/time stamp in file name.

example:

backup-moodle2-activity-40-presenter40-20121025-1328.mbz

Using MacOS Finder and also via terminal, one could find all file names that contain the YYYYMM of old backups. Then remove.

From terminal and located in the folder/directory in which the backups were located, the following would delete all files having 201210 in their filename:

sudo rm -f *201210* [ENTER]

Personal opinion when it comes to backups ... any automated removal of backups that have been archived needs to be fool proof.  Using a single attached device for such backups just requires monitoring once a month/every two weeks.  It has been my unforunate 'luck' that the day I remove, is the day I need a backup.  Maybe your luck is better! ;) 2 cent opinion. 

'spirit of sharing', Ken

 

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Mike Kelly發表於

I'm running into this problem as well. Here's some background:

Moodle version: 2.3.6

OS: Linux (CentOS 5.6)

Our automated backups were original configured to be stored in the MoodleData directory. A week or so ago we started running into space issues w/our Moodledata directory during our automated backups. So, we decided to move backups to their own filesystem.

Now automated backups are happily running to our new partition (/var/moodlebackups, let's say). But our moodledata directory is still quite full. Turns out that all the old automated backups from before the switch are still sitting in Moodledata. I'd really like to make these go away.

We've got over 2,000 courses on the system and there's no way I'm going to do this manually. I realize that the new Moodle 2.x filesystem design uses a file's hash as its filename, and stores all the metadata in the database. So, it seems like we'll need some kind of a script to automate this cleanup.

The question is: Has anybody written this script already? If not, are there any pointers to doing so?

Second question: Is this not a bug in automated backup's behaviour, or at the very least an incomplete feature? When you switch from one backup location to another, there should be *some* way to clean out those old backups.

評比平均分數： -
In reply to Mike Kelly

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@Mike K.

Funny you should ask ... (actually not) cause I've been working on something you might be able to use ... bash commandline script to find the backup files contained in filedir and copy them to an ./extracted/ directory.  Does work.  Have used on a CentOS 5.9 with 2.3.x MDL. 

'Discalmer' ... nope ... spelled it right ... not reponsible for problems using this might create.  ** Don't run without inspecting!!!! **

Could change the cp to rm.  That **would require** editing DB tables as well.  But the command used to find those records is in the script.

Designed to run as root user in the data directory itself ... feel free to edit/change/whatever.  And if you do improve/change for a purpose ... share back!

See attached .txt file.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由Mike Kelly發表於

Thanks Ken, much appreciated. I had a look at the script and it's similar to what I had in mind - except filtering for backups in the "automated" file area, etc. I'll be sure to post my changes here as well.

Regarding the database - do you know if simply removing the desired entries from the mdl_files table is enough? Or are there any other tables that I'd need to look at in order to clean things up properly once the files are removed from the filesystem?

 

評比平均分數： -
In reply to Mike Kelly

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Welcome.

mdl_backup_courses, mdl_backup_controllers

are two that do contain references to course ID's but am not at all certain how manual removal of records would affect a site and any backup process.

Guess the only way to know for sure is install a sandbox Moodle, create a course and back it up, then go through whatever process to remove not only the files from filedir but also tables.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由Mike Kelly發表於

Thanks Ken.

After a bit of digging, I've been able to patch together a little PHP script that uses the Moodle File API to identify & remove automated backups. I'm using the same method that the backup script uses to clear out old backups, so Moodle takes care of all the db details.

Just running a few more tests today - when I'm done, I'll post the script here.

評比平均分數： -
In reply to Mike Kelly

Re: How to I bulk delete Automated backups

由Carol Booth發表於

Hi Mike

this sounds like exactly what i need!  Did you get it working?  I changed my course backup location and ended up with MANY backups - I'm not even sure where they all are.  I think some have gone back into the course backup areas perhaps because of a problem writing to the new backup location.

Carol

評比平均分數： -
In reply to Carol Booth

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

This is what I run on my Debian server in terminal.  It removes any file over 1 day old.

/path to backup folder(fill in with appropriate path)/* -mtime +1 -exec rm {} \;

Works beautifully - I have been fighting a hard drive that ran out of space for a few weeks now.  Trying to get through to the end of semester before I move to a different server and so I run this every couple of days.  I also sync my backup folder with an off-server location which helps.

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由Will Pandle發表於

Hi Mike

did you ever come right with the script you were referring to on the post of the : Friday, 12 April 2013, 10:21 PM.

 

Thanks

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片
Emma

Didn't you mean 'find'?

$ find /path/to/backup -mtime +1 -exec rm {} \;
評比平均分數： -
In reply to Visvanath Ratnaweera

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

No - I ran this from terminal if I remember correctly - takes you to the folder and then the rm command deletes the appropriate files.  Luckily I don't need this anymore - got migrated finally to a different server.

評比平均分數： -
In reply to Mike Kelly

Re: How to I bulk delete Automated backups

由curt bixel發表於

I think I may have a similar problem to which I need a solution.  I am running moodle 2.4 on a virtual private server at Inmotion hosting.  I am running into problems as my server is full.  This was a surprise, as I am only hosting 2 classes. 

I investigated the problem, and it seems that the "eraser" is not deleting old backups that have been "deleted" through moodle. 

I think that the problem is that when I originally installed moodle, I named the moodle data file "moo" instead of allowing the default "moodle_data."  (don't ask why I made this choice, it is a long story)

The automated backups were working fine until I ran out of space. 

I thought a possible solution might be to go to the automated backup setup and "specify the directory" for the automated backups, to help the "eraser" find them.  I searched through the moodle data file hoping to find a single directory where the backups were stored, copy the path, and then enter that in the automated backup settings. 

Unfortunately, it appears that the backup files are being stored all over the place, making it difficult to figure out to what to set the "specified directory for automated backups."

I can find the backup files in the database, but I am not sure what would happen if I simply deleted them.  I don't know how moodle might react if a database file were simply deleted.  Right now, I have a small file, 50KB, that I am using to overwrite the backups.  Moodle seems fine with this, as there is still a file with the correct name in the data base in casemoodle should ever look for it. 

Another option would be to help the "eraser" find the files be leaving the automated backup settings at default, and then just rename my database to the default, which I think is "moodle_data".  I could then go into the "config.php" file and change the path to the data file, but I am not sure if there might be any other place where a change would need to be made. 

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

You have a couple of problems here.  First, I would be very nervous about changing the name of the database - is that what you meant or did you mean your data folder?  As long as Moodle knows where your data is, I don't think the name of the folder is the issue.

Even if you find all your files in one place, at this point, I do not think the eraser will just automatically delete the old files.  It seems to just start from the day it is set up and pretty much ignores anything existing in the folder.  

You might start by looking at the temp folder and the trashdir folder.  If you say you have deleted the files within moodle, maybe it is these folders that are not clearing out correctly.

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由curt bixel發表於

Emma, you are right, I meant the name of my moodledata folder.  I am a bit scared of the database.

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@Curt

In many respects, the new file system is a mess when it comes to administration and in situations like these.
Backups are actually stored in moodledata/filedir/
and not by their humanly re-cognizable name such as:

backup-moodle2-activity-40-presenter40-20121025-1328.mbz

but rather a hash.  The backup name above is actually:

7aef21f74c9e5e0a90d481cf3c462711ca488822

which is also includes the path to the file … once deciphered

./filedir/7a/ef/7aef21f74c9e5e0a90d481cf3c462711ca488822

This file (just above) is actually the backup-___.mbz file shown as example above.

So cleaning up issues like this involves not only the physical removing of files … (very carefully) but also cleaning up of the mdl_files table.  NOT an easy undertaking.

To see how many ya got, try this query (could add other fields to the select parameter to get more info ... like size, etc.):

select filename from `mdl_files` where (`mimetype` like "application/vnd.moodle.backup" and `filename` not like ".");


Think I'd try this first … reset automated backups to keep only ONE (evidently you current have save multiples).

Leave it to Course backup filearea - that means filedir and the hash file naming stuff mentioned above.

Schedule it for all days (Sun, Mon, Tue, etc.)

Active: set to manual - must run via command line or via browser then.

Also set the time to something very close to the present … slightly in the future … so that when you run cron (and you'll have to do that multiple times to catch it just right) you can see it 'kick in'.

** Change or Add something to the courses.  Add a hidden page with a comment in it as to when you last backed up or something.  Reason for this, version 2.4.x skips courses that have not been changed - automagically (unless told to do otherwise).  We want to force it to re-cognize the new 1 only backup (thus should remove the others). Check the options in automated backups for skip hidden courses (default is yes) and Skip courses not modified since - set lower - 1 day is lowest.

If you have command line access,  there is a script one can run that pertains to automated backups ONLY.

cd /moodlecodedirectory/admin/cli/

php automated_backups.php

IF not, you're stuck with running cron.php from a browser.

When you run cron, make note of the backups and also make note of the cleaning of the trashdir.

Another factor here … one doesn't control the cleaning of the trashier.  Could be much of your space is actually in the trashier.

After you get only ONE backup - don't forget to re-set all those changes to backups/cron, etc. you have done to something sensible and desired.

'spirit of sharing', Ken

評比平均分數：Useful (1)
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

Thanks for the help so far.  In fact, before I read the posts above, I took the following steps:

  1. renamed my moodle data directory from "moo" back to the default of "moodledata"
  2. edited the config.php file so that it would look for the data directory under the name of "moodledata" instead of "moo"
  3. Restarted my server.
  4. Set cron to run every minute (I have cPanel access)
  5. Let cron run a few times.
  6. Set cron back to run once twice per hour.

I was sort of hoping the "eraser" would now be able to find the older backup files and delete them, and I would be able to watch the size of my moodledata folder begin to shrink.  That does not seem to be happening.

Ken, thanks for the tip on how to find the old backups.  I have tried creating a very, very, small file, 75 KB, naming it the same name as the backup, and then saving it over the top of the backup.  I am able to see immediately the size of my moodledata folder shink an amount equal to the size of the old backup file. 

What would happen if I simply deleted the old backup file.  I get the impression that this might create some serious problems. 

I will now read your posts above once again, more carefully, and see if that can give me some more ideas.  微笑

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由curt bixel發表於

Well, now it seems to have created a folder called "backup" in my temp folder, and placing the backups neatly there.  It is even naming them things like, "backup.mbz" 

Go figure.  I wonder why having my moodle data file named "moo" instead of "moodledata" resulted in in spraying the backup files all over the place.

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Unless you set the autobackups to save 'Course backup area' AND to include a specified area AND purposely put the path to the moodledata/temp/backup/ directory am afraid they won't be there in say 4 days.  A temp directory is usually just that ... temporary and will be cleaned out sooner or later.  Actually, if it's working correctly, one might see the .mbz files there for a little while until the system copies the .mbz to the designated directory.  Once it has done that, it is removed.

Your replacing of those hashnamed files with small any K size files doesn't match the meta data concerning backups in the tables of the Moodle DB.  While you have been successful in reducing the space, you still have a problem should you ever need to restore a backup.  Your only alternative ... a site backup and restore - which is way too much to get a little thing back into a course and might prove to be difficult to do on a remotely hosted system.

No offense ... but much more tinkering like this, Curt, and one should expect a total meltdown.  And no one will be able to help.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

If I understand correctly, the standard procedure for storing backup files would be for moodle to place the backup file in the temp/backup directory for a while, and then give them a hash type name and stick them somewhere in the filedir folder.  This seems like a strange way to handle things, but that would explain why all my backups seem to be randomly sprayed around in the moodledata folder in different subfolders. 

If I have this correct, then the backups made today will remain in the temp folder for a short while, then they too will be moved to some random folder in the filedir????

I fully understand your advice to avoid messing around in the file system like this.  I am trying to find the best way to solve this problem, and, by replacing the large backups with these small, "dummy" files, I have been able to clear enough space on my server to actually get a recent backup of each of my courses, which is nice. 

 

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

Ken,

With the lack of matching meta data, are you saying that if I accidentally attempt to restore one of these files I might crash, or would it just not be able to do the restore.  I don't really need the old backups, so not being able to use them would not be an issue. 

Am I right that simply deleting them from the filedir would be a bad idea.  A really, really, bad idea?

By the way, now that I have succeeded in clearing enough space on the server so that it has allowed me to make a recent backup of each of my two courses, I am considering wiping everything from the server and doing a fresh install. 

If I were to do this, and simply allow it to name the moodle data folder the default "moodledata" and leave everything in autobackup settings at default, do you think I could avoid the problem of zombie backup files in the future.  I am still not clear on why moodle cannot find and delete backup files that have been deleted from within moodle. 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

If I re-call correctly, Moodle doesn't immediately delete anything.  When one removes a file from the new file system and it's not linked anywhere else in any course, Moodle moves it to the trashdir.  The contents of trashdir gets cleaned every 4 days - IF one runs cron.  No cron.  No emptying of trashdir.  So your space usage might have been in the trashdir all along ... or it could also have been in the moodledata/temp/backup/ directory where the backups are built and then copied to whatever you've chosen to save/retain backups.

Since I've not read of anyone doing what you are doing (although it is perhaps a fix that will work) cannot say for sure what the consequences would be.  But, there is a readme.txt file in filedir.   Please read it.

What you've done is intelligent and I understand the reasoning behind it, but the goal now should be to get it operate as it should ... without the work-arounds, etc. in place.

If there are files in moodledata/temp/backup they can be removed manually without affecting anything.  Think the same might be true of anything found in trashdir ... although Moodle might complain about not finding anything there it was told to delete it should not be a critical error that would prevent Moodle from functiong.

Have to remember, that now, with the new file system, one is not only working with physical files (which are an adventure to find) but also a database.  It's very important when backing up now to have both an SQL dump of the database AND the datadirectory.

At this point, I don;t think one has to re-install, but one should keep an eye on backups ... any kind of backup now that you know where to look and for what to look.

It would be wise to test one of your course backups in the same production environment ... restore to another hidden course.  Or restore to a local version of Moodle running on your laptop/workstation.

And, one more thought, with your provider ... can you acquire some more space (more wiggle room)?  Yes, it won't be for free, but think of it  kinda like insurance.

My 2 cents, of course.

'spirit of sharing', Ken

 

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由Séverin TERRIER發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Testers的相片 Translators的相片

Just one thing :

  • files in trashdir are deleted 24h after being in trashdir (if cron running).
  • 4 days is for files in "draft" (that should NOT be deleted)

Information from Petr Škoda in MDL-25592 's comments.

Hope this helps
Séverin

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

It is not the renaming of the folder that did this.  It is you changing the location of your backups under automated backup settings.  If you put them in a directory on the server, they will stay in one folder.  If you select to have them in the backup and restore area, they will be created in each course folder (or a subfolder within there).

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由curt bixel發表於

Emma,

I never did actually change the location of my backups under automated backup settings.  I left them at their defaults when I first installed moodle, and have never changed them.  Any idea why moodle would fail to find and delete the old backups.  I have deleted all but one backup of each course through the "restore" page on moodle, but it appears the old backups are still taking up space on the server. 

My first course has a backup size of about 250 MB, my second course has a backup size of about 100 MB.  A few months ago, I was only using a few GB of my server space.  It has steadily grown so that it is now 32 MB, and with my server full, no more backups can be made.

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@ Curt ... how did you discover that you were running out of space?  Are you sure it's space to which you have access?  or is it space on the DB server ... which is probably not on your virtual server, but a virtual server itself dedicated to DB only.

With only 2 courses it's possible to have courses in the Gig range with lots of videos uploaded.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

Ken,

I discovered I was running out of space on my server when my backups started failing.  I looked into it, and it appear that they were failing because there is no more space on the server for them. 

I downloaded my full website so that I could analyze the files with "Disk Inventory X" and it showed the "filedir" at about 32 GB. 

Also, InMotion hosting has a neat feature called "Disk Space Usage" which showed the following:

You can also see the size of my public_html/ folder is a bit over 500 MB.  I have kept the two classes I am hosting small by purposefully not uploading any video, as that has been an issue in years past.  With my classes about 500MB, this should leave space for 60 or so full backups, more if they are automated backups.  This appears to be what has happened.

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

By the way, another thing I am trying to track down has to do with cron.  The printout cron sends to my email talks about me having 3 courses.  I only have two.  Is it possible that I have a third "zombie" course bouncing around that has lots of backups that could be deleted properly through moodle if I could find and open this course? 

The cron printout indicates that one of my courses is "hidden."  I don't seem to be able to find it.  The moodle docs I have found so far are all about courses that are "hidden" from students.  I would imagine the admin would be able to see these easily.  I cannot.  I also cannot find a place to change any setting that might make it appear.

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@Curt

Could be you have a role conflict of some sort.  Site Admins should be able to see all courses ... hidden or not.

If you have access to/ability to edit the config.php file and can see your user ID number, add this to the config.php file:

$CFG-siteadmins=2,#;

2 is the initial admin account that was created when the site was first installed.  The other numbers are other users.   Above, where # is your user ID number.

No need to reboot anything ... just login and see if one can access Site Admin Menu, Courses, Add-Edit Courses.

The only other thing it might be is related to roles and permissions, etc.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

I think I am already able to log in and see the course list.  Here is an image:

Here is what is in the Miscellaneous category:

 

This should mean that I have only two courses, but cron is telling me that I have two courses + 1 hidden course. 

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Front Page is considered a course, but I've only seen front page backups when autobackup is setup and functioning or manually backing up the front page. 

Site is called 'Moodle 25' short name 'MDL25'.  In the running of cron:

Running required automated backups...
... started 16:07:53. Current memory use 89.4MB.
Backing up Moodle25...
complete - next execution: Tue, 03 Sep 2013 02:55:00 -0500

In the cron job it should show all courses and if a course has not been changed (if backups left at their defaults) it should inform:

Backup of 'iPads 101' is scheduled on Tue, 03 Sep 2013 02:55:00 -0500
Skipping iPads 101 (Not modified in the past 5 days)

At the end of that routine:

Sending email to admin
Automated backups complete.

Do you get a message like this?

Date: Sun, 1 Sep 2013 16:07:58 -0500                                                                                   
From: Admin User <ktask@site>                                                                                  
To: Admin User <ktask@site>                                                                                    
Subject: MDL25: Automated backup status                                                      
Summary                                                                                                                
==================================================
  Courses: 22
  OK: 1
  Skipped: 21
  Error: 0
  Unfinished: 0
  Warning: 0
                                                                                                           
  Backup completed successfully

What are your autobackup settings?

Could you possibly send a screen shot/edited text of what you see when cron is run?

'spirit of sharing', Ken

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@Curt

Using whatever tool you have to query MySQL DB: Finds name,filesize,contenthash backup file metadata in DB

select filename,filesize,contenthash from `mdl_files` where (`mimetype` like "application/vnd.moodle.backup" and `filename` not like ".");

Will get something like this:

+----------------------------------------------------------------------+----------+------------------------------------------+
| filename                                                             | filesize | contenthash                              |
+----------------------------------------------------------------------+----------+------------------------------------------+
| backup-moodle2-course-2-backuptest-20121211-0035.mbz                 |    20287 | 129717ceaec0752f916d8a73c1a96f218fba59ce |
| backup-moodle2-course-38-ipads_1-20130123-1002.mbz                   | 15453225 | e2c2a07ff231e6f9e0064c44ae4a66a9a27c31c5 |

At the very bottom, one would see the front page backup:

| backup-moodle2-course-1-mdl25-20130901-1507.mbz                      |  5698873 | 8bb50981e095ef890cfc9e0cbc0b4140ba187138 |
+----------------------------------------------------------------------+----------+------------------------------------------+

Using whatever tool you have to browse files one could verify the file size using the contenthash to locate the directories/contenthash file name.

'spirit of sharing', Ken

 

評比平均分數：Useful (1)
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

The front page also backs up as a course - however it shouldn't be hidden but would explain your third course.

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由curt bixel發表於

Well, that explains the third course.  微笑  Mystery solved. 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由curt bixel發表於

By the way, thanks for all of the help you have all given me so far.  I seem to have yet another question. 

I have managed to get a full backup of both of my courses.  One I have been able to restore on another system, so I know I have a good backup.  The second hangs during restore.  The screen just goes white. 

Research has turned the following up as a possible solution:

....have been able to backup and restore large courses 800 meg+ with these settings.

Warning - once complete change back to  your current settings as this does not work well in a production environment. it leaves everything running for a long time and eventually runs out of resources and has to be rebooted.

change this in the php.cfg file

max_execution_time =19200
max_input_time = 1500
max_input_vars = 8000
memory_limit = 1024M
post_max_size = 1200M
upload_max_filesize = 1000M
max_file_uploads = 30

FastCGI Settings

Activity Timeout 3600
idle Timeout 3600
Queue Length 2000
Request Timeout 3600

Would temporarily making these changes be likely to help.  (Fortunately, the moodle installations I am running host only my two classes, so any possible damage is limited to me. )

And are their any other settings anywhere that might also possibly help moodle chug through the backup and restore of a large class? 

Note:  At the same time, I am working on the problem from the other direction.  I am finding ways to place larger files somewhere outside my courses and then simply link to them.  (Dropbox is my first choice here)  My goal is to keep all future classes clear of video and other files that seem large enough to create these sorts of problems in the future.  微笑

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

@Curt

Does the restore even begin?  When does the white screen of death appear ... 60 seconds?  Have you checked error logs for apache?   They can give clues ... like 'failed to allocate ...'.  Have you turned on debugging and attempt to restore again?

Some of those values are rather long/large considering it's a site with only 2 courses.

Through all of this, you've not mentioned system specifics (like shared system, provider, dedicated VPS - package limitations) ... even the OS ... which would help. 

It's not only PHP involved but MySQL.  Assuming a full backup with all users so there could be some data that contains a rather large chunks.   Consider setting max_allowed_packet=500M for your MySQL.

'spirit of sharing', Ken

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

I am going to try to dig into this more this weekend.  I am on the VPS1000 plan at InMotion hosting. 

  • RAM:   512MB / 1GB Burstable
  • Storage: 40GB
  • Bandwidth: 50GB/mo.
  • IP Addresses: 2
  • Environment: CentOS, cPanel, WHM
  • Backups: Every 24-36 hours

It looks like CentOS is based on RedHat Linux.

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

By the way,  I seem to have another question.  The reason I am running my own server is because the tech people in my district are a bit overwhelmed and understaffed to give enough attention to our district's moodle installation.  One of my courses suddenly stopped functioning and they could not find any way to access it, back it up, restore it, etc.  Pretty much a total loss.  Went to InMotion, set up a moodle site, worked a few hundred hours putting my course back together, and now I am up and running. 

After getting an idea that the blank screen might be a result of a time-out, I think perhaps you can help me to recover my old course on the school server.

Here is what happened immediately before my course quit working...

I have noticed that when you are trying to tab an activity to the right, if you click 3 times in rapid succession, it will move 3 jumps to the right, and you can save a bit of time. 

Well, I was attempting to make 4 duplicates of an activity.  I had the bright idea that if I clicked "duplicate" 4 times, it would make 4 different copies and save me some more time.  So I clicked 4 times....

That was pretty much it.... 

From that instant on, just the blank screen. 

Ken, from your responses so far, I sort of suspect you already know what happened, and you may even know a single setting that could be changed that would allow me to access this course again. 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

'already know what happened, and you may even know a single setting that could be changed that would allow me to access this course again'

Boy I wish I had 'magic bullets' and could 'mind meld', but, alas, just a mere mortal! :|

Is your school server Linux or other?   If Linux, think you're technical folks would object to opening access to it from a certain IP address via ssh?  If so, IM me and I'll provide the IP address I'll be attempting to access via.  Would also need to know the root password.

If Windows/Other ... sorry ... don't do Windows anymore.

However, if that is nixed/out of the question, then you say you do have a backup file of this course, do you not?  Sometimes, through much tinkering, it is possible to restore a frapped course if one re-constructs the backup file (involves editing XML files and re-zipping).  Should warn the backup (if you did a full backup) does contain student information.  However, I taught for 20+ years and have worked with schools for another 20+ years and have passed my 'pervert'/privacy test (if that means anything!).

I can grant you access to a server where you could upload that backup and I'll 'go the extra mile' and see what I can find working with that backup.

Really don't want to even attempt writing up a massive document on how to do that.  But, will be happy to share what I've learned, if anything, along the way.

'spirit of sharing', Ken

 

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

Unfortunately, I don't think the tech people in my district would provide ssh access to anyone.  Right now, the district's moodle installation is working passably, and I think the tech people are trying to shed responsibility for the moodle server, so they are just biding their time.  I, by the way, have tried to make it very clear to them as well as the powers that be in the district that there is a possibility that they are underestimating the investment the teachers in the district have made to their moodle courses, and unless a VERY solid porting system is set up, there might be more than a little pushback in abandoning moodle.  This is another reason I have chosen to go to my own server.  

They have been quite cooperative as far as going in and changing settings I suggest.  I was able to get backups working for large classes in the district by figuring out where the timeout setting was that was interfering with the creation of large zip files.  They changed that and we were back in business.  微笑  

 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Sometimes, the fixes are little things ... if one knows where to look and how to tweak.  That's good.   So, good, glad the school Moodle is now working.

I've been on both sides of tech. and server admin.  Can tell you that I've seen what you've described ... actually been involved in more than one of those 'pushes'.   Somewhat tough road to travel unless one is very good at internal politics and motivation, etc..  On their defense, however, you might ask them what their 'typical' day is like.  Or, take a day off and ask to shadow them.  You might gain some insights and help them ... which in turn means something coming the other way. ;)  With Moodle, it might take a village!

Moodle 2 does more ... requires more ... at for a while, some 'care and feeding'.  There are some great folks in these forums who are willing to help.

'spirit of sharing', Ken

 

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

The school moodle is "sort of working."  I am not really satisfied with the backup plan in place.  When my class malfunctioned, they were not able to restore it using an automated backup because they have gone to just backing up the entire server every day.  To recover my class would have meant to erase one day of everyone else's work.  With 500 teaching in the district, this really was not an option.  That is pretty much why I went to InMotion.

By the way, I have been at InMotion for a year and am overall quite satisfied with the experience.  I was able to get an introductory rate for the first year, but now it looks like it will be $500 to renew for the year. 

With the $500 per year staring me in the face, I am considering setting up my own server from home..  At InMotion, I have 1GB of RAM and 40GB hard drive space, it would seem that I could purchase a computer with vastly more RAM and hard drive space for not too much money. 

I imagine other expenses would be a good backup system as well as increased speed of internet service and a static IP.  Would not really need an un-interruptable power supply as only my two classes would be on the server and if service goes down for a few hours that is not all that big of a concern. 

Any idea how fast I would need my internet service to be?  And any suggestions on ISP?

評比平均分數： -
In reply to Ken Task

Re: How to I bulk delete Automated backups

由curt bixel發表於

On another note, I am with you in the tech people in our school system.  They are overworked and continually have more and more responsibilities added to their plate.  At the same time, their resources, both money and staffing, are under constant pressure.  I think they are doing a great job with the resources they have.   微笑

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Emma Richardson發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Plugin developers的相片

Honestly, if you only have two classes, why not just back them up yourself and use the school system.  If you only issue with it was back up, I don't understand why you don't just create your own backups, download them weekly, for example, and not go through the effort of setting up your own server.

If you do go forward with a server, the amount of bandwidth would be contingent on how many users you might have logged in at any one time.  While you only have two classes, if they have hundreds of students, you will obviously need more capability that if they only have 20 per class.  Also, I would suggest a linux setup if you can - works really well with Moodle.  ISP's would depend on what is available in your area.

評比平均分數： -
In reply to Emma Richardson

Re: How to I bulk delete Automated backups

由curt bixel發表於

Emma,

When I moved to my own server, I installed and am running Moodle 2.4 and plan to upgrade soon to 2.5.  The school is still running moodle 2.1, so I am concerned with backward compatibility. 

評比平均分數： -
In reply to curt bixel

Re: How to I bulk delete Automated backups

由Ken Task發表於
Particularly helpful Moodlers的相片

Obviously, there is no backward compatibility.

Am sure you know, however, might want to pass this along to the server admin/op at school:

This is the last formal release of the 2.1 branch. Support for this branch has been discontinued. We highly recommend you upgrade!

Date 14th January 2013

From: http://download.moodle.org/

But, maybe the reason for their delay is the upward requirements for running Moodle.  In any case, any app that is allowed to fall too far behind in updates is probably prone to problems and might be low picking fruit for any that decide to 'harvest'.

http://bit.ly/13DrUVp

Many listed beginning Jan. 2013 have granted access level: none. access: remote.  complexity: low.

'spirit of sharing', Ken

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Bill Antonia發表於

I had this problem as well. Spent a fair amount of time deleting the files through the UI then thought there must be a better way!

Eventually edited the /moodle/backup/util/helper/backup_cron_helper.class.php

adding the line:

backup_cron_automated_helper::remove_excess_backups($course);

somewhere soon after:

foreach ($rs as $course) {
                $backupcourse = $DB->get_record('backup_courses', array('courseid' => $course->id));
                if (!$backupcourse) {
                    $backupcourse = new stdClass;
                    $backupcourse->courseid = $course->id;
                    $backupcourse->laststatus = self::BACKUP_STATUS_NOTYETRUN;
                    $DB->insert_record('backup_courses', $backupcourse);
                    $backupcourse = $DB->get_record('backup_courses', array('courseid' => $course->id));
                }

which appears from line 124 in Moodle 2.6

Checked the the global automated backup settings, then ran the /moodle/admin/cli/automated_backups.php script from the command line.

Going back to the Moodle Restore UI, the backups have gone, except the last as in my settings! Whoopee!!!!

Commented out the additional line and now to wait 4 days for the "draft" files to be deleted to clean up the drive.

Job done.

Although the settings were to keep just one backup the call backup_cron_automated_helper::remove_excess_backups($course); was being skipped as there was no backup to do for all of my courses.

Good idea to test this on a non-production site first though.

Hope this is of some use.

評比平均分數： -
In reply to Bill Antonia

Re: How to I bulk delete Automated backups

由Bill Antonia發表於

Just an update, manage to grab back 382GB, yes I did have a very large number of backup files for each course and there are very many courses! It appears the deleted files were moved to the moodledata/trashdir then deleted overnight via cron.

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Laurent Wafflard發表於

Due to an issue of the external backup space (backup_auto_destination), we also had an explosion of space consumed on moodledata.
The script attached (in french, sorry)  :
    list the automatic backups stored on moodledata
    can copy them elsewhere ("demo" = dry-run, nothing is actually done but it show what would be)
    can delete them (with "demo")

It should be stored at the root of Moodle files (the same level as config.php). Only a Moodle Administrator can call it  (http://moodle_url/purge_autobackup.php)

DATE_BACKUP at beggining of script limit action to a specific date or year.

Hope this help.

評比平均分數：Useful (3)
In reply to Laurent Wafflard

Re: How to I bulk delete Automated backups

由Petrus 4發表於

Thanks Laurent Wafflard  that was php script worked tres bien!

I recommend it be added to moodle as one of the admin scripts! 微笑


 

評比平均分數： -
In reply to Petrus 4

Re: How to I bulk delete Automated backups

由Séverin TERRIER發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Testers的相片 Translators的相片

You should perhaps create a tracker issue for that (if none already exists), or talk about that in related issues in the tracker...

Séverin

評比平均分數： -
In reply to Laurent Wafflard

Re: How to I bulk delete Automated backups

由Jean Pierre Ducassou發表於

Awesome job Laurent!

I wanted to run it on console and be able to pass different parameters so I writed a script based on yours.

I hope you don't mind.

It will be on github in case anyone wants to fix or enhance something.

https://github.com/jpducassou/moodle-tool_autobackups

Thanks again to you and all the community.


評比平均分數：Useful (1)
In reply to Jean Pierre Ducassou

Re: How to I bulk delete Automated backups

由Laurent Wafflard發表於

Of course no problem for the new version 眨眼 Very good idea to make a cli script and even better one to share it !

Thanks for these improvements, we'll try it soon !

評比平均分數： -
In reply to Jean Pierre Ducassou

Re: How to I bulk delete Automated backups

由Julia Küfner發表於

Thank you Laurent and Jean Pierre!

Your tool saved me! Successfully deleted 51 GB!

(In my case I had to move to a different server and forgot so set the rights of the backup folder (a non-moodledatafolder on the same server), moodle couldn't access it and started adding backupfiles in filedir without deleting them...) 

here a short info to the script that I "learned" while using it:

-q (query; show the automated backupfiles )
-c (copy; copy files to location)
-d (directory)
-x (expunge; delete automated backupdfiles)

all files are deleted, which means they are send to trashdir (the timestamp of the last emptying of trashdir is set to the past), the next time cron is running tashdir is emptied.

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Gilles-Philippe Leblanc發表於

We have a similar problem but we found that the backups from deleted courses was not purged.

I just created a plugin to remove this kind of backups:
https://github.com/leblangi/moodle-tool_purgeautobackups

Also, you may vote on this task:
https://tracker.moodle.org/browse/MDL-48808

I hope this will help.

評比平均分數： -
In reply to Richard Ackland

Re: How to I bulk delete Automated backups

由Séverin TERRIER發表於
Documentation writers的相片 Particularly helpful Moodlers的相片 Testers的相片 Translators的相片

Hi,

About this, there is now an ad-hoc report in the documentation to find information about backup files (total size) and prepare a script to delete files.

But a imagine that if you delete files, you should also delete information about them in the DB? And perhaps something else?

Séverin

評比平均分數：Useful (2)