best way to downlad lots of files

best way to downlad lots of files

by Art Lader -
Number of replies: 8

Hi, All,

I am getting ready to move lader.org to a new host and want to back up everything before I do that. I have tried downloading the entire public_html folder to my hard drive via ftp, but am having no luck.

Surely, there is there a better way to do this, right? A simple telnet command or something?

Maybe I should use a program like this one:
http://www.httrack.com/index.php

Thanks,
Art

Average of ratings: -
In reply to Art Lader

Re: best way to downlad lots of files

by Vu Hung -
Yeah!
httrack allows you to download the whole site so it is suitable with your purpose.
In reply to Art Lader

Re: best way to downlad lots of files

by Samuli Karevaara -
What's the problem with FTP, to me it would seem the most convinient way to download lots of files. Web site downloaders like httrack don't download it "as it is" but "as the surfer sees it", so if you have pictures or files there that are not linked anywhere, you loose them. You also loose all of the server side code (if any, obviously) and so on.
In reply to Samuli Karevaara

Re: best way to downlad lots of files

by Art Lader -

Hi,

I am using Filezilla and it keeps disconnecting after twent or thirty minutes. I am trying to download 1.1 Gigs and I guess it is too much.

-- Art

In reply to Art Lader

Re: best way to download lots of files

by Jan Dierckx -
Art, Someone once referred me to a command WGET. I use it because it has a mirroring option to only download the files that have been changed recently. I've never downloaded over 1 gigs in 1 go, but this command at least has the advantage that it skips files which have already been downloaded, so if things go wrong, you can at least repeat the process. There is a windows version of WGET. WGET is a unix console command so it takes a little to understand all the options. I've entered the following commands in a notepad document:
 
wget -o log.txt -np -m ftp://username:password@www.olvrode.be/private/moodledata 
and saved it with extension .cmd Now whenever I click on this file all the recently changed files are downloaded to my pc I believe it could work for you to if you just replace the location, username and password stuff. Another thought: if you have shell access you could zip directories before downloading them.
In reply to Art Lader

Re: best way to downlad lots of files

by Enrique Castro -
Picture of Core developers
Hi Art,
    I do use Filezilla also!. It's a great program.

There is a  queue  mode, you can set to download only modified files, or files newer than existing ones (just as the wget command, also a good option). You  can "resume" a broken queue.

Perhaps disconnecting is not due to filezilla. Go to Options and make sure you have enabled teh "keep alive" feature. You may try to increase timeout time to a couple of minutes, and set automatic reconnect, in addition.

- Enrique -
In reply to Art Lader

Re: best way to downlad lots of files

by Samuli Karevaara -
Art, I haven't used FileZilla. Enrique said that it has a feature to resume a broken download queue, but couldn't find a mention of this from FileZilla documentation.

I don't think the keep alive -feature helps here, but doesn't hurt to try. It's designed to not to disconnect when idling, downloading lot's of stuff sends messages to the ftp server "all the time".

Of the ftp programs I like FlashFXP the most. It's not free, but there is a 30-day fully working trial. It also has a download queue feature, and it reconnects and resumes the downloads if the connection is disconnected. You can set it to reconnect X times, waiting N minutes between the tries. You can also queue stuff from multiple different servers.

The wget solution might not work for you if you don't have the bulk of the files already.

Another thing that I do when downloading tons and tons of files is just to download them one folder at a time, it wears the "butt muscles" but doesn't break the connection so often.
In reply to Art Lader

Re: best way to downlad lots of files

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Do you have shell access on the new host? Just use FTP from the command line!
 
ftp oldhost.com
cd somedirectory
get somefile.tar.gz

I would archive the whole directory on the old host first, for example:

tar cvzf somefile.tar.gz somedirectory

Then unpack it on the newhost:

tar xvzf somefile.tar.gz
In reply to Martin Dougiamas

Re: best way to downlad lots of files

by Hannes Gassert -
If you have this kind of access to the two boxes you could as well use rsync. Most probably faster and certainly simpler.