.. _hpc-data-transfer:

Transferring data to/from HPC storage
=====================================

There are multiple ways to transfer data to and from the cluster. The main ways are using SCP and rsync, or for Windows users an SFTP client.

Followings are some methods using the command line or GUI tools:

Open OnDemand
-------------

If the cluster has :ref:`software-ood` you can log in and use the "Files" feature to manage your files on the cluster via a web interface. This allows GUI-based file transfer, browsing, and editing, which may be preferred by users with minimal Linux experience.

SCP (Linux/macOS)
-----------------

To securely copy data to a remote host:

.. code-block:: console

    To Copy one file
    $ scp <path to the file> username@remotehost:/path/to/remotedir/
    To copy entire folder
    $ scp -r <Directory_to_copy> username@remotehost:/path/to/remotedir/

.. tip::
   Tired of typing your password? See our guide on :ref:`ssh-key-setup` to learn how to configure passwordless login using cryptographic keys.



*Examples:*

.. code-block:: console
    :caption: Use scp to copy the IMPORTANT_DATA directory on CoSTAR to the home directory on a remote machine called "myhost".

    [abc123@costar-login01 ~]$ scp -r IMPORTANT_DATA abc123@myhost:~
    abc123@myhost's password:
    DATA_FILE_4.txt                               100%    0     0.0KB/s   00:00    
    DATA_FILE_3.txt                               100%    0     0.0KB/s   00:00    
    DATA_FILE_1.txt                               100%    0     0.0KB/s   00:00

.. code-block:: console
    :caption: Use scp to copy the IMPORTANT_INPUT_FILES directory on "myhost" to the home directory on CoSTAR.

    [abc123@myhost ~]$ scp -r IMPORTANT_INPUT_FILES abc123@costar-login01:~
    abc123@myhost's password:
    DATA_FILE_4.txt                               100%    0     0.0KB/s   00:00    
    DATA_FILE_3.txt                               100%    0     0.0KB/s   00:00    
    DATA_FILE_1.txt                               100%


**Pro-Tip (ProxyJump):**

You can combine these steps into a single command from your local machine using the ``-J`` (ProxyJump) flag:

.. code-block:: bash

   ssh -J <username>@access.eps.surrey.ac.uk <username>@costar-login01

rsync (Linux/macOS)
-------------------

To synchronize a directory from a local machine to a remote machine (or vice versa):

.. code-block:: console

    $ rsync -avz <Directory> user@remotehost:/path/to/remotedir/

Why use rsync instead of scp? rsync is faster for repeated transfers because it only sends file changes (deltas), can resume interrupted copies, and preserves timestamps/permissions by default with ``-a``.

*Examples:*

.. code-block:: console
    :caption: Synchronize data in IMPORTANT_INPUT_FILES on "myhost" to a directory on CoSTAR.

    [abc123@myhost ~]$ rsync -avz IMPORTANT_INPUT_FILES abc123@costar-login01:~
    abc123@myhost's password:
    sending incremental file list
    IMPORTANT_INPUT_FILES/
    IMPORTANT_INPUT_FILES/INPUT_FILE_1.in
    IMPORTANT_INPUT_FILES/INPUT_FILE_2.in
    IMPORTANT_INPUT_FILES/INPUT_FILE_3.in

    sent 306 bytes  received 99 bytes  810.00 bytes/sec
    total size is 0  speedup is 0.00

.. code-block:: console
    :caption: Synchronize data in IMPORTANT_DATA on "costar-login01" to a directory on host "myhost".

    [abc123@costar-login01 ~]$ rsync -avz IMPORTANT_DATA abc123@myhost:/user/HS204/abc123/
    abc123@myhost's password:
    sending incremental file list
    IMPORTANT_DATA/
    IMPORTANT_DATA/DATA_FILE_1.txt
    IMPORTANT_DATA/DATA_FILE_2.txt
    IMPORTANT_DATA/DATA_FILE_3.txt

    sent 290 bytes  received 92 bytes  69.45 bytes/sec
    total size is 0  speedup is 0.00

.. caution::

    PLEASE ENSURE you use the trailing "/" exactly as shown above to avoid overwriting any folders/data,
    as the rsync command is sensitive to the way in which trailing slashes are used.


.. danger::

    Rsync is very useful for copying, moving and backing up/synchronizing data, but it can be very easy to
    make a mistake in a command, and slashes in paths make a big difference.

    Read a guide to ensure you're doing exactly what you want (test out commands):

    https://www.thegeekstuff.com/2010/09/rsync-command-examples

Windows Data Transfer Methods
-----------------------------

MobaXterm allows you to transfer files to and from a cluster using a built-in terminal, so you can use all previously mentioned rsync and scp commands. It also provides an SFTP (Secure File Transfer Protocol) panel, which enables simple drag-and-drop file transfers between your local machine and the cluster.

.. image::  images/moba_sftp.png
    :align: center

.. Note::

   Windows-based text editors (e.g. Notepad++) may insert an extra *carriage return* character (``^M``) at the end of each line.
   This can cause issues for many Linux-based applications.

   To fix this, run the built-in ``dos2unix`` utility on any ASCII file transferred from Windows to CoSTAR.

   Example:

   ::

      [abc123@costar-login01 ~]$ dos2unix example.txt
      dos2unix: converting file example.txt to Unix format ...