[Web4lib] Off-site backups and scripted SCP/SFTP

[Jonathan Gorman]

On Tue, 25 Jul 2006, Joerg Messer wrote:

>
> I recommend going with TAR over SSH instead of SCP.  TAR will actually give 
> you an exact copy where CP/SCP doesn't play nice with symbolic links.  Just 
> pipe TAR through an SSH pipe to another TAR on the remote machine.
>
> tar cf - /data |  ssh user at backup.system.org "cd /backup; tar xvpf -"
>
> Works like a charm.


Assuming that a) they're on a linux/unix system and b)Assuming that 
you're not going from a system that has gnu tar to a system 
with sun tar ;).  Also, why not compress it while you're doing the 
transfer, ie

  tar jcf - /data |  ssh user at backup.system.org "cd /backup; tar jxvpf -"

That way you speed up transfer time.  (Again, won't work on sun tar)

Of course, this has little to do with the OP problem, which is a concern 
about a secure automated way to use keys that have passwords.  (Not sure 
if that's possible myself).

Ideally if I was transfering a large number of files, I'd

a) md5sum files
b) create a tarball
c) transfer tarball (whether pipe or via scp really is a matter of 
personal preference, I'd probably go via scp so I know if the transfer 
fails I'd have the tarball somewhere to try again)
d) expand tarball
e) check md5sums

With some error checking inbetween.

> Another popular approach is to use RSYNC.

Yes, rsync sounds like a good idea as well.

Jon Gorman