Skip to content

Data Transfer

This page describes how to transfer data to and from the HPC filesystem (GPFS) and Research Archival filesystems using SFTP, Globus, or Rsync.

Choosing a Transfer Option#

There are three options for moving data to and from our storage systems:

  1. Secure File Transfer Protocol (SFTP) (on-campus or VPN only) - We recommend this method when your data transfer is relatively small and will take only a few minutes, or when you are connected on-campus via a reliable high-speed network. This method is quick and convenient, but less reliable for long-running transfers.
  2. Globus - We recommend this method when your data transfer will take a long time (at least several hours) and/or when the connection may be periodically interrupted during transfer. This method requires a bit of setup ahead of time, but it is robust and appropriate for any data transfer task. Globus is also the fastest method out of these three options.
  3. Rsync/SCP - We recommend this method when you are connected on-campus with a reliable high-speed network connection and your data transfer is part of a scripted process. Globus is still a more reliable option in this case, but Rsync/SCP doesn't require as much setup on the client-side.

Tip

If you are using SFTP, Rsync, or SCP, be sure to connect to export.rcc.fsu.edu and not a HPC login node. The export cluster has a 40Gb/s uplink to our parallel storage system (GPFS), and a 20Gb/s uplink to our Archival Storage system.

This is more than double the capacity of the HPC login node uplinks, and it will ensure that your transfers run faster.

Using MobaXTerm (Windows)#

MobaXTerm is our recommended SSH client for Windows users. It also has out-of-the-box SFTP support.

  1. Click the "Session" icon in the upper left-hand corner of the MobaXTerm window: Click the "Session" icon in the upper left-hand corner of the MobaXTerm window
  2. In the dialog that appears, click the "SFTP" icon: In the dialog that appears, click the "SFTP" icon
  3. In the Remote Host input, enter: export.rcc.fsu.edu. In the Username input, enter your RCC user account. In the Port input, enter 22. Click the OK button. Fill out the "Remote Host" and "Port" fields
  4. Enter your password when prompted. After that, you will be returned to the main SFTP interface: Fill out your password when prompted, and then you should see something similar to this window

Using SFTP with Cyberduck (Mac/Windows)#

Cyberduck is our recommended SFTP client option for Mac. It is also available for Windows.

  1. Click the "Open Connection" button in the upper left-hand corner of the CyberDuck Window: Click the "Open Connection" button in the upper left-hand corner of the CyberDuck Window
  2. In the "Open Connection" dialog, select SFTP (SSH File Transfer Protocol): In the "Open Connection" dialog, select SFTP
  3. Enter export.rcc.fsu.edu in the Server textfield and your RCC User account credentials: Fill out the Server and account credentials (RCC credentials)

Using SFTP from Nautilus File Manager (Linux)#

Most popular linux distributions come with the Nautilus file manager, which is part of the Gnome Desktop Environment. If your distribution uses KDE, Cinnamon, or another desktop environment, you will need to look up the procedure using your file manager du jour.

  1. Open the Files app, and locate the Other Locations option on the left side of the file browser. Open the _Files_ app, and locate the _Other Locations_ option on the left side of the file browser

  2. On the bottom of the window, enter sftp://export.rcc.fsu.edu in the Connect to Server dialog. On the bottom of the window, enter sftp://export.rcc.fsu.edu in the _Connect to Server_ dialog

  3. Enter your RCC user credentials in the dialog that appears. Enter your RCC user credentials in the dialog that appears

  4. Once logged in, the file system will appear as a location on the left-hand side of the Nautilus file manager. You can now drag/drop, copy and paste, and interact with the files on the server the same way that you do with files on your computer. Press the eject button to disconnect. Mounted SFTP filesystem

Using terminal commands#

There are many command-line utilities to transfer data, and automate transferring data to and from RCC storage systems. Below are some examples of commands supported on the RCC systems.

Warning

You need to be on-campus or connected to the FSU VPN to use the following commands.

The sftp command#

Use SFTP to interact with files on the storage systems

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
$ sftp USERNAME@export.rcc.fsu.edu
USERNAME@export.rcc.fsu.edu's password:
Connected to export.rcc.fsu.edu.
sftp>

# List files on the server
sftp> ls

# Copy local file (/tmp/test.txt) to the server
sftp> put /tmp/test.txt
/tmp/test.txt 100%   88KB  88.5KB/s   00:00

# Copy remote file on server to computer (downloads to current working directory)
sftp> get test.txt

# Access archival storage directory
sftp> cd /mnt/archival/GROUP_PATH

# Disconnect
sftp> exit

Read more information about using SFTP at this DigitalOcean tutorial page.

The scp command#

Use SCP to perform scripted, automated file transfers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Usage
$ scp [source-file(s)] [destination]

# Ex: use scp a file to your GPFS home directory
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:~

# Ex: use the `-r` (recursive) option to copy an entire directory to your GPFS home directory
$ scp -r ~/Documents/my_research_data USERNAME@export.rcc.fsu.edu:~

# Ex: copy data to a shared research volume on GPFS
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:/gpfs/research/GROUP_PATH/SUB_PATH

# Ex: copy data to a shared archival volume
$ scp /tmp/test.txt USERNAME@export.rcc.fsu.edu:/mnt/archival/GROUP_PATH/SUB_PATH

# Ex: copy data from the storage system to your local disk (. = current working path)
$ scp USERNAME@export.rcc.fsu.edu:~/test.txt .

Read more information about using SCP from this Linuxize tutorial.

The rsync command#

Use RSYNC as an alternative to scp when you want to keep files synchronized between two systems:

1
2
# Use Rsync to copy a directory to your GPFS home directory
$ rsync -v -a -e ssh /tmp/test.txt USERNAME@export.rcc.fsu.edu:~

Read more about rsync at this DigitalOcean tutorial page.

The rclone command#

Use Rclone to move data between cloud storage providers (AWS, OneDrive, etc.) and your storage space on our storage systems.

Warning

FSU has blocked inbound/outbound FTP connections throughout all of campus, so FTP remotes will not work in rclone.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ ssh USERNAME@export.rcc.fsu.edu

# Use the 'config' subcommand of rclone to enter an interactive setup utility.
[USERNAME@o22-export-31 ~]$ rclone config
Current remotes:

Name                 Type
====                 ====
onedrive             onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> my-cloud-storage
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Alias for a existing remote
   \ "alias"
 2 / Amazon Drive
   \ "amazon cloud drive"
 3 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
   \ "s3"
 4 / Backblaze B2
   \ "b2"
 5 / Box
   \ "box"
# etc...

For further information about rclone, refer to the official documentation page.

Data transfer between GPFS and Archival systems#

To transfer data between GPFS and Archival storage systems, use the export.rcc.fsu.edu server. Both filesystems are mounted on that server:

System Mount path on export.rcc.fsu.edu
HPC filesystems (GPFS) /gpfs/home and /gpfs/research
Archival /mnt/arch

Note

As of November 1, 2024, we have transitioned all Archival volumes to a new storage backend. The new path for all Archival volumes is /mnt/arch.