[Web4lib] Off-site backups and scripted SCP/SFTP

[Thomas Bennett]

I use paramiko ( http://www.lag.net/paramiko/ ) for ssh scripting but then I 
like python.  The following is a variation of an example sftp demo from 
paramiko I've edited to transfer backup.tar to the server and read it back.  
Error checking and command line arg checking and comments were left out to 
save room here, see the original file.  Comments are preceeded by # of 
course.

Thomas

# import libraries here
# you can setup logging
paramiko.util.log_to_file('demo_sftp.log')
port = 22
myhost='my.host.mydomain'
myusername = 'my_user'
mypassword ='my_password'
hostkeytype = 'ssh-rsa'
myhostkey = paramiko.RSAKey(data=base64.decodestring('AAIAv8CabCRjhl7cnU='))
# now, connect using paramiko Transport to negotiate SSH2 across a connection
try:
    t = paramiko.Transport((hostname, port))
    t.connect(username=myusername, password=mypassword, hostkey=myhostkey)
    sftp = paramiko.SFTPClient.from_transport(t)
# make a directory on the server
    try:
        sftp.mkdir("demo_sftp_folder")
    except IOError:
# read from local file       
    data = open('backup.tar', 'r').read()
# write tar file to server
    sftp.open('demo_sftp_folder/backup.tar', 'w').write(data)              
# copy the backup.tar back here
    data = sftp.open('demo_sftp_folder/backup.tar', 'r').read()
    open('backup.tar', 'w').write(data)
    t.close()

except Exception, e:
# no connection was made if first 'try' failed
    print '*** Caught exception: %s: %s' % (e.__class__, e)
    traceback.print_exc()
    try:
        t.close()
    except:
        pass
    sys.exit(1)



On Tuesday 25 July 2006 16:49, Keith D. Engwall wrote:
> I've had several suggestions to use SSH, which I do see the benefit of.  My
> question is still, though, how is authentication being handled for SSH?
>
> The choices, to my understanding are:
>
> 1) authenticate on the fly (no keys)
>
> 2) use keys with no password protection on the keys
>
> 3) use keys with password protection on the keys
>
> Based on my reading, #1 is not scriptable, #2 is a security risk, and #3 is
> only scriptable by jumping through some hoops using ssh-agent, which
> requires manually authenticating the key at some point after a reboot.
>
> Which of these three methods (or other method if I am missing any) are
> people using when they use SSH?
>
> Thanks,
>
> Keith
> -----Original Message-----
> From: web4lib-bounces at webjunction.org
> [mailto:web4lib-bounces at webjunction.org] Sent: Tuesday, July 25, 2006 1:32
> PM
> Cc: Web4Lib
> Subject: Re: [Web4lib] Off-site backups and scripted SCP/SFTP
>
>
> I recommend going with TAR over SSH instead of SCP.  TAR will actually give
> you an exact copy where CP/SCP doesn't play nice with symbolic links.  Just
> pipe TAR through an SSH pipe to another TAR on the remote machine.
>
> tar cf - /data |  ssh user at backup.system.org "cd /backup; tar xvpf -"
>
> Works like a charm.
>
> Another popular approach is to use RSYNC.
>
> Francis Kayiwa wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On Jul 25, 2006, at 7:36 AM, Keith D. Engwall wrote:
> >> I'm curious as to whether anyone is using commercial off-site backups
> >> and if so, what service they are using.
> >>
> >> We've got about 5GB (mostly from our LIS), and would prefer to use
> >> SFTP or SCP with keys (because of scripts).
> >>
> >> Also,
> >> I've found some information online about setting up SSH Keys for
> >> scripts, but it seems like either
> >>
> >> a) you have to not password protect the key
> >>
> >> or
> >>
> >> b) you have to manually enter the password after a reboot of the
> >> computer (using ssh-agent or similar solution)
> >>
> >> I'm trying to find a third, in between method... where the password
> >> is stored somewhere obscure.  Obviously this is still a security
> >> risk, but it's at least less of one than using cleartext ftp with the
> >> username and password in the script.
> >>
> >> Any ideas?
> >
> > Why not use SSH instead of SFTP/SCP?
> >
> > SSH will pass STDIN through the encrypted session to STDOUT on the
> > remote end. This takes care of it no? Also, forced commands permit
> > immediate command execution method through normal shell notation
> >
> > To specify a forced command to a system using OpenSSH, one prepends
> > 'command="some command" to the beginning of the line containing the
> > public key. Other SSH servers may use different syntax. Consult your
> > documentation on the correct format, or start using OpenSSH :-)
> >
> > I would probably use dd for something like this. Mostly to protect the
> > user from overwriting the public keys.
> >
> > command="/usr/bin/dd of=/path/to/file"
> >
> > This assuming you are using "put" (to borrow from FTP parlance)
> >
> > otherwise
> >
> > command="dd if=/file/to/send" for "get."
> >
> > All the above would be the modification at the "server" end.
> >
> > On the client end
> >
> > "cat file | ssh remotehost"
> >
> > or on Windows using putty
> >
> > "type file | plink remotehost"
> >
> > Depending on if the outside datacenter accepts SSH connections you may
> > need a "Man in the Middle" host but we would be getting way ahead of
> > ourselves. :-)
> >
> > regards,
> > ./fxk
> >
> > ===============
> > Francis Kayiwa
> > Library Systems Team
> > 4-180, MC 234
> > T: +1.312.996.2716
> > W: http://www.uic.edu/~kayiwa
> > Key: http://tigger.uic.edu/~kayiwa/kayiwa.gpg
> >
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.1 (Darwin)
> >
> > iD8DBQFExjfkN+YGKSXdLhkRAjLZAJ0ePy1g2fXzMONXPyH9yG94z8iUiwCeKutj
> > UYUxhnRlo7jCXHrWDZhAvpg=
> > =NoLv
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > Web4lib mailing list
> > Web4lib at webjunction.org
> > http://lists.webjunction.org/web4lib/
>
> --
> Joerg Messer
> Programmer/Analyst
> University of British Columbia Library
> 2206 East Mall, Vancouver, BC, Canada, V6T 1Z3 t. 1.604.822.5091 f.
> 1.604.822.3201 e. joerg.messer at ubc.ca
> _______________________________________________ Web4lib mailing list
> Web4lib at webjunction.org
> http://lists.webjunction.org/web4lib/
>
> _______________________________________________
> Web4lib mailing list
> Web4lib at webjunction.org
> http://lists.webjunction.org/web4lib/

-- 
====================================================================
Thomas McMillan Grant Bennett		Appalachian State University
Computer Consultant III			P O Box 32026
University Library				Boone, North Carolina 28608
(828) 262 6587

An important measure of effort in coding is the frequency with which you write 
something that doesn't actually match your mental representation of the 
problem, and have to backtrack on realizing that what you just typed won't 
actually tell the language to do what you're thinking. -Eric Raymond

Library Systems Help Desk: http://linux.library.appstate.edu/help
====================================================================