Backup using rsync

Backup using rsync

- Bente Olsen の投稿
返信数: 6

I have updated an existing (FTP) backup of my moodledata directory using rsync with the options from https://docs.moodle.org/31/en/Tools_for_backing_up_server_files#Rsync_in_linux:

rsync -auvtz --delete -e ssh mysshusername@example.com:/my/server/directory /my/backup/directory/

I noticed the -auvtz option, isn't it redundant to include 't', as I read the man rsync it is a part of the '-a' option?

Anyway, I expected it to perform an incremental backup, but rsync started to delete the existing content of the directory tree and then copied all files once again. 

  1. My question is now how to make an incremental backup? - &
  2. BTW, do I have to do this as root or is that not necessary?
評点平均: -
Bente Olsen への返信

Re: Backup using rsync

- Ken Task の投稿
画像 Particularly helpful Moodlers

incremental backups are backups that contains only those files that have been altered since the last full backup or that's kinda how that term is defined.   So the --delete option would do that .... and think you'd want that.

Maybe think of this as a mirror of the active/production server files to another location - that other location could be a different directory on same server or to a directory on a remote server.

If a new file exist in /my/server/directory then it is sync'd in /my/backup/directory.

If a file exist in /my/backup/directory that *no longer* exist in /my/server/directory then it is removed in /my/backup/directory.

There would be references to filedir files and trashdir files - those two directories being considered the most important in moodledata for backups.

There is an option for --dry-run which just runs through what rsync would have done and gives you a chance to see what would have occurred.   Sounds like you might have run the rsync the first time without --delete then ran it again with the --delete option.   That's  a guess.

As per: https://linux.die.net/man/1/rsync

Think the hardest thing to remember about rsync (and I have to look this up every time) is what you want to do ... as it relates to SRC (source) and DEST (destination).

Local:  rsync [OPTION...] SRC... [DEST]
Access via remote shell:
  Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
  Push: rsync [OPTION...] SRC... [USER@]HOST:DEST


Others might disagree, but if rsyncing to another server would think rsyncing as root would allow the freedom to archive just about anywhere on the DEST box.

'spirit of sharing', Ken


評点平均: -
Ken Task への返信

Ang: Re: Backup using rsync

- Bente Olsen の投稿

As I said, I expected the syntax from docs.moodle.org would give me an incremental backup. I understand that as an operation where the existing files will not be altered or deleted unless they no longer mirror the source files. But rsync deleted them and copied everything once again. It of course take a much more time than if it would only update the existing files and directories and copy new and changed content.

評点平均: -
Bente Olsen への返信

Re: Ang: Re: Backup using rsync

- Ken Task の投稿
画像 Particularly helpful Moodlers

Like I said ... I have trouble with SRC and DEST ... just trying to help out here, that's all! 笑顔

so did you type the rsync command or is the syncing in a script (bash shell/other shell)?

What does history |grep rync show you?

'spirit of sharing', Ken


評点平均: -
Bente Olsen への返信

Re: Ang: Re: Backup using rsync

- Ken Task の投稿
画像 Particularly helpful Moodlers

Pardon the length of this ...

I take it you were referring to this reference?

https://docs.moodle.org/31/en/Tools_for_backing_up_server_files

Basic command:

rsync OPTIONS SOURCE DESTINATION
where SOURCE would/should always be the moodledata directory
where DESTINATION would/should be either another location on same server that has
room/space or a remote server via ssh as a user that has access rights to write to and delete some directory and it's contents.

Referencing the page above, it shows:

rsync -auvtz --delete -e ssh mysshusername@example.com:/my/server/directory /my/backup/directory/

'to copy only the changed files to another host.

Seems to me the SRC and DEST parts are backwards there


rsync -auvtz --delete /my/server/directory -e ssh mysshusername@example.com:/my/backup/directory
SRC is /my/server/directory
DEST is via -e ssh mysshusername@example.com:/my/backup/directory

Here's one that I use to mirror the filedir to another location on the data partition of same server:

rsync -avzh --delete /home/Moodle/filedir /home/backup/filedir

When run it shows/does only those that are new to the mirrored directory (home/backup/filedir):

sending incremental file list
filedir/0d/e0/
filedir/0d/e0/0de0c3934f7abc56b96e87dd159c5fc13aac8ad9

sent 5.36M bytes  received 64.39K bytes  309.89K bytes/sec
total size is 564.21G  speedup is 104038.25

Here's another that syncs from one server to another server on an alternate ssh port (i.e., not port 22):

/usr/bin/rsync -av --delete . -e "ssh -p 7997" root@moodle.tld:/home/sosbackup/

Command is run from the /home/backup directory of the sos server ... the '.' at the tail end of options ... it is the SOURCE directory.
The DEST directory is on moodle.tld in /home/sosbackup/

When the command is run, here is output (other files ... shorten the list here):

sending incremental file list
./
drupalcode-2016111973558.tar
other files ....
deleting noodles/moodle306-code-20161008224625.tar
deleting other files
moodles/coloodle311-code-20160702.tar
moodles/ other files ...
moodles/moodle29/
deleting moodles/moodle29/backup-moodle2-course-8-meinconflict-20150727-1814.mbz
deleting moodles/moodle29/other files
sqldailydumps/campsigwp.sql
sqldailydumps/coloodle.sql
sqldailydumps/other sql files

sent 8457765445 bytes  received 248629 bytes  23994366.17 bytes/sec
total size is 15799043662  speedup is 1.87

Suggestion ... rather than working with a directory you *don't* want to lose or mess up in any form or fashion, suggest syncing a test directory with junk files in it to a test directory on another server.

Just remember one could be using rsync to push or pull.
In the case of backups ... archiving to another server ... the concept is PUSH.

'spirit of sharing', Ken


評点平均: -
Ken Task への返信

Ang: Re: Ang: Re: Backup using rsync

- Bente Olsen の投稿

Thanks Ken,

Your suggestion about history | grep rsync led me to think of what the root of the error  could be:

rsync source destination

is not the same as

rsync source/ destination

while the first syncs source to destination the latter syncs the content of source to destination 笑顔

A dry run (with an extra v option) illustrates the difference:

 ~ $ rsync -auvtzv --delete --dry-run -e ssh  me@domain.tld:/var/www/moodledata/filedir/0f/ moodledata/filedir/0f/
[...] password: 
receiving incremental file list
delta-transmission enabled
9b/0f9b429129f7fac4fb32cca41a7ce8b7587b2864 is uptodate
a8/0fa816c009cc4021f324af7c568551bcb07244f7 is uptodate
c2/0fc2b93be9cdd0f0040b8d33722251be5bdeda47 is uptodate
./
ca/
ca/0fca0abae0b77452c1c98654b9dadc2b194bd66b
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 59 bytes  received 483 bytes  47.13 bytes/sec
total size is 255,968,868  speedup is 472,267.28 (DRY RUN)

Here only the map ca do not fit the local ca map

and without source end slash

no maps are equal to the local because rsync vil copy the map 0f into the local map 0f:

 ~ $ rsync -auvtzv --delete --dry-run -e ssh  me@domain.tld:/var/www/moodledata/filedir/0f moodledata/filedir/0f/
[...]  password: 
receiving incremental file list
delta-transmission enabled
0f/
0f/09/
0f/9b/
0f/9b/0f9b429129f7fac4fb32cca41a7ce8b7587b2864
0f/a8/
0f/a8/0fa816c009cc4021f324af7c568551bcb07244f7
0f/c2/
0f/c2/0fc2b93be9cdd0f0040b8d33722251be5bdeda47
0f/ca/
0f/ca/0fca0abae0b77452c1c98654b9dadc2b194bd66b
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 60 bytes  received 497 bytes  53.05 bytes/sec
total size is 255,968,868  speedup is 459,549.13 (DRY RUN)

If I had been aware of that, my incremental would probably have been incremental!

評点平均:Useful (1)
Bente Olsen への返信

Re: Ang: Re: Ang: Re: Backup using rsync

- Ken Task の投稿
画像 Particularly helpful Moodlers

You are 100% correct!   One has to pay attention in the man for rsync ... they do mention how trailing slashes behave/modify the results.   Sorry ... didn't mention that! :\

'spirit of sharing', Ken



評点平均: -