Working with files on the cluster

This page focuses on active file management while you are developing or running work on the cluster. The best choice depends on whether you are editing code, moving large datasets, or running interactive debug sessions.

See the Login to the Cluster page if you are unsure which access method to use.

Code files, scripts and slurm job submission scripts

For code under active development, a predictable workflow is:

  1. Develop and test locally using your preferred tools and version control.

    • Transfer updates to the cluster when you are ready to run (see Transferring data to/from HPC storage).

    • Push code to GitHub or another repository for version control and backup. Then pull the code on the cluster when you need to run.

  2. Make quick in-place edits only when needed.

    Good options for lightweight edits on the cluster:

    • Open OnDemand (OOD) file manager: The Open OnDemand web interface is the quickest way to browse, upload, and edit small files without opening a full terminal.

    • SSH terminal editing: Use the terminal inside OnDemand or connect via SSH and use vi or nano for small changes and job script edits. For smoother logins, see ho to set up SSH keys.

Data files, models, and large inputs

For datasets, model weights, checkpoints, and large input files (for example, videos or images), aim to transfer and stage data deliberately:

  • See the HPC data transfer page for guidance on bulk movement and syncing (scp, rsync, or SFTP tools).

  • For larger or long-lived datasets, follow the guidance in Dataset Management so you land data in the right storage tier.

Interactive jobs for debugging and testing

Use interactive allocations when you need to debug, profile, or test code on the cluster. This keeps heavy workloads off the login nodes and matches how your jobs will actually run.

One option is to perfom interactive work directly on the compute nodes using a terminal connected via SSH or OnDemand. This is ideal for quick tests and debugging.

If you want to use VS Code, it is supported only when it runs on a compute or interactive node, not on the shared login nodes. The login nodes are meant for lightweight tasks, and running vscode-server there can degrade performance for everyone.

Warning

Do not connect VS Code to the login nodes. If a vscode-server (or code-server) process is detected on a login node, the process and its associated SSH session will be terminated automatically. Connections that pass through to a compute or interactive node are not affected.

Safe options for using VS Code on the cluster include:

  • Open OnDemand VS Code: Runs fully inside a web browser and is friendly for newcomers.

  • Remote-SSH to a compute node: Requests an interactive job and attaches VS Code to the compute node rather than the login node.

For a full setup guide, please refer to our guidance on VS Code.