File Transfer

Overview

  • Time: 20 min

    1. Learn how to copy files between your local machine and the HPC cluster.

    2. Understand how to use Secure Copy Protocol (SCP) for file transfers.

In this section, we will learn how to copy files between your local machine and the HPC cluster. This is essential for transferring data, scripts, and results.

Secure Copy Protocol (SCP)

SCP is a secure method for transferring files between your local machine and the HPC cluster. It uses SSH for data transfer, ensuring that your files are encrypted during the transfer process.

To copy a file from your local machine to the HPC cluster, use the following command:

1scp /path/to/local/file username@hpc-cluster:/path/to/remote/directory

To copy a file from the HPC cluster to your local machine, use:

1scp username@hpc-cluster:/path/to/remote/file /path/to/local/directory

Explanation

  • Replace /path/to/local/file with the path to the file on your local machine.

  • Replace username@hpc-cluster with your username and the address of the HPC cluster.

  • Replace /path/to/remote/directory with the path to the directory on the HPC cluster where you want to copy the file.

  • Similarly, adjust paths for copying files from the HPC cluster to your local machine.

For Linix and MacOS users, the scp command is available by default in the terminal.

1 # Change to Downloads directory
2 cd $HOME/downloads
3
4 # Create a test file
5 touch file.txt
6
7 # Securely copy the file to Gadi
8 scp file.txt <username>@gadi.nci.org.au:/scratch/<project>>/<username>/

This copies a file named file.txt from your local downloads directory to your scratch directory on Gadi.

For Windows users, you can use tools like PuTTY or WinSCP to perform SCP operations. All the latest versions of Windows also support the scp command in the Command Prompt or PowerShell.

1 # Change to Downloads directory
2 cd C:\Users\<username>\Downloads
3
4 # Create a test file
5 echo "This is a test file." > file.txt
6
7 # Securely copy the file to Gadi
8 scp file.txt <username>@gadi.nci.org.au:/scratch/<project>>/<username>/

To copy a directory and its contents, use the -r option with scp:

1 scp -r /path/to/local/directory username@hpc-cluster:/path/to/remote/directory

In Linux and MacOS, do this:

 1 # Change to Downloads directory
 2 cd $HOME/downloads
 3
 4 # Create a new directory
 5 mkdir my_directory
 6
 7 # Navigate into it
 8 cd my_directory
 9
10 # Create multiple empty files
11 touch file1.txt file2.txt
12
13 # Go back to Downloads directory
14 cd ..
15
16 # Securely copy the directory to Gadi
17 scp -r my_directory <username>@gadi.nci.org.au:/scratch/<project>>/<username>/

In Windows, you can do the same with:

 1 # Change to Downloads directory
 2 cd "$env:USERPROFILE\Downloads"
 3
 4 # Create a new directory
 5 mkdir my_directory
 6
 7 # Navigate into it
 8 cd my_directory
 9
10 # Create multiple empty files
11 echo "This is a test file1." > file1.txt
12 echo "This is a test file2." > file2.txt
13
14 # Go back to Downloads directory
15 cd ..
16
17 # Securely copy the directory to Gadi
18 scp -r my_directory <username>@gadi.nci.org.au:/scratch/<project>/<username>/

Important

Ensure that you have the necessary permissions to read/write files in the specified directories on both your local machine and the HPC cluster.

Key Points

  • Use scp to securely copy files between your local machine and the HPC cluster.

  • The -r option allows you to copy directories recursively.