Dataset Management

Effectively managing large datasets is crucial for productive HPC workflows on CoSTAR. This guide covers best practices for transferring, organizing, and maintaining your data on the cluster.

Warning

Important: All data on the cluster is non-backed up. Please ensure you maintain a backup copy of any mission-critical data and files on an external storage system or your local machine.

Transferring Large Datasets

Moving large datasets (e.g. >10 TB) can be time-consuming and prone to failure if the connection is interrupted. To ensure successful data transfers, please follow the detailed instructions provided in the Transferring data to/from HPC storage section.

Recommended Transfer Method:

Rsync is the recommended method for transferring large datasets to and from the CoSTAR cluster because it:

  • Supports resumable transfers if the connection is interrupted.

  • Only transfers changed files on subsequent runs, saving bandwidth.

  • Preserves file permissions and timestamps.

  • Works efficiently over high-latency connections.

For detailed transfer instructions, command examples, and alternative methods (SCP, SFTP), see the Transferring data to/from HPC storage guide.

Dataset Organization Best Practices

Organizing your data effectively will help you manage large datasets and improve workflow efficiency:

  • Use descriptive folder structures - Organize data by project, date, or data type.

  • Document your data - Keep README files describing dataset contents, formats, and processing steps.

  • Remove intermediate files - Clean up temporary and intermediate files after processing to save storage space.

  • Track file versions - Use version control for code and configuration files.

  • Monitor storage usage - Regularly check your storage quotas. See Checking Your Storage Usage for detailed instructions.

Getting Help

If you encounter issues during data transfer or have questions about data management, please reach out:

  • Support Portal: Use the Ticket System to submit a request to the CoSTAR support team.

  • Microsoft Teams: Join the Teams Community to connect with other users and get updates about the cluster.

  • Check cluster status: Monitor the Teams channel for any announcements about cluster maintenance or data transfer issues.