Support &
Documentation
There are several possible ways to transfer files and data to and from HPC2N's systems.
FTP (File Transfer Protocol) is a simple data transfer mechanism. FTP is the original program for data transfer, but it was not designed for secure communications. FTP exists on the systems, but HPC2N does not permit connections using FTP because of the security problems. There are several modern FTP clients which support either SFTP or SCP which are similar, secure protocols for file transfer. Use one of those methods instead of FTP.
SCP (Secure CoPy) is a simple way of transferring files between two machines that use the SSH (Secure SHell) protocol. You may use SCP to connect to any system where you have SSH (log-in) access. There are some graphical file transfer programs which offers SCP as protocol and it is also a command line program on most Linux, Unix, and Mac OS X systems. SCP can copy single files, but will also recursively copy directory contents if given a directory name.
Command-line usage:
(to a remote system from local)
$ scp sourcefilename user@hostname:somedir/destfilename
example:
$ scp irf.png user@machine.umu.se:C/irf.png Password: irf.png 100% 2100 2.1KB/s 00:00
(from a remote system to local)
$ scp user@hostname:somedir/sourcefilename destfilename
example:
$ scp user@machine.umu.se:irf.png irfpic.png Password: irf.png 100% 2100 2.1KB/s 00:00
(recursive directory copy to a remote system from local)
$ scp -r sourcedirectory/ user@hostname:somedir/
Linux / Solaris / AIX / HP-UX / Unix:
Microsoft Windows:
Mac OS X:
SFTP (SSH File Transfer Protocol or sometimes called Secure File Transfer Protocol) is a rnetwork protocol that provides file transfer over a reliable data stream. You may use SFTP to connect to most of HPC2N's systems. SFTP is a command -line program on most Unix, Linux, and Mac OS X systems. It is also available as a protocol choice in some graphical file transfer programs. SFTP has more features than SCP and allows for other operations on remote files, such as remote directory listing, and it is also possible to resume interrupted transfers. Note, however, that command-line SFTP cannot recursively copy directory contents. If you need to do so, you must either use SCP or a graphical SFTP client.
Command-line usage:
$ sftp -B buffersize user@hostname
examples:
(to a remote system from local)
enterprise-d [~]$ sftp user@kebnekaise.hpc2n.umu.se Connecting to kebnekaise.hpc2n.umu.se... user@kebnekaise.hpc2n.umu.se's password: sftp> put file.c C/file.c Uploading file.c to /home/u/user/C/file.c file.c 100% 1 0.0KB/s 00:00 sftp> put -P irf.png pic/ Uploading irf.png to /home/u/user/pic/irf.png irf.png 100% 2100 2.1KB/s 00:00 sftp>
(from a remote system to local)
sftp> get file2.c C/file2.c Fetching /home/u/user/file2.c to C/file2.c /home/u/user/file.txt 100% 1 0.1KB/s 00:00 sftp> get -P file3.c C/ Fetching /home/u/user/file3.c to C/file3.c /home/u/user/file.txt 100% 1 0.4KB/s 00:00 sftp> exit enterprise-d [~]$
The following two flags can be useful
Regarding buffer size; in order to find a optimal buffer size, use the following formula:
optimal buffer size = 2 x bandwidth x delay
or
optimal buffer size = bandwidth x RTT
RTT = round trip time, which you get from ping, since delay is about ping average time/2
Example:
ping average = 2.653 ms
the data link's capacity = 1 GB
So optimal buffer size should be 2.653 ms x (1 GB/8 bit) = 331625 bytes.
See http://fasterdata.es.net/TCP-tuning/ for more information.
Linux / Solaris / AIX / HP-UX / Unix:
Microsoft Windows:
Mac OS X:
LFTP is a command-line file-transfer program for Linux and Unix systems. FTP, HTTP, FISH, SFTP, HTTPS and FTPS protocols. LFTP has additional features not provided by SFTP such as bandwidth throttling, transfer queues, and parallel transfers. It may be used interactively or scripted. Every operation in LFTP is reliable, that is any non-fatal error is handled and the operation is retried automatically. So if downloading breaks, it will be restarted from the point automatically.
In order to connect over SFTP to our resources, the username and hostname shall be prefixed by sftp://
LFTP has shell-like command syntax allowing you to launch several commands in parallel in background (&). It is also possible to group commands within () and execute them in background. All background jobs are executed in the same single process. You can bring a foreground job to background with ^Z (c-z) and back with command `wait' (or `fg' which is alias to `wait'). To list running jobs, use command `jobs'. With parallel transfers LFTP can be much faster than SCP or SFTP, so its use is encouraged when possible.
LFTP is simply a client, so it is not needed on the remote machine involved in a transfer (the remote system need only support SFTP).
Examples:
lftp> cat file | gzip > file.gz lftp> get file & lftp> (cd /path && get file) &
The first command retrieves the file from the ftp server and passes its contents to gzip which in turn stores compressed data to file.gz. Other commands show how to start commands or command groups in the background.
LFTP has a built in mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server.
More interactive examples:
$ lftp sftp://user@hostname
(transfer all ".dat" files from remote system to local)
lftp :~> mget *.dat
(transfer "filename.dat" file from local system to remote)
lftp :~> put filename.dat
(transfer a directory and all contents from remote system to local, using 5 connections in parallel)
lftp :~> mirror --parallel=5 remotedir localdir/
(transfer a directory and all contents from local system to remote, using 8 connections in parallel)
lftp :~> mirror -R --parallel=8 localdir remotedir/
Batch usage:
(specify all actions on command line)
$ lftp sftp://user@hostname -e "mget *.dat"
(specify all actions in the script file "transfer.lftp")
$ lftp sftp://user@hostname -f transfer.lftp