.. _dataset-management:

******************
Dataset Management
******************

Effectively managing large datasets is crucial for productive HPC workflows on CoSTAR. This guide covers best practices for transferring, organizing, and maintaining your data on the cluster.

.. warning::

    **Important:** All data on the cluster is **non-backed up**. Please ensure you maintain a backup copy of any mission-critical data and files on an external storage system or your local machine.


Transferring Large Datasets
============================

Moving large datasets (e.g. >10 TB) can be time-consuming and prone to failure if the connection is interrupted. To ensure successful data transfers, please follow the detailed instructions provided in the :ref:`hpc-data-transfer` section.

**Recommended Transfer Method:**

Rsync is the recommended method for transferring large datasets to and from the CoSTAR cluster because it:

- Supports resumable transfers if the connection is interrupted.
- Only transfers changed files on subsequent runs, saving bandwidth.
- Preserves file permissions and timestamps.
- Works efficiently over high-latency connections.

For detailed transfer instructions, command examples, and alternative methods (SCP, SFTP), see the :ref:`hpc-data-transfer` guide.


Dataset Organization Best Practices
====================================

Organizing your data effectively will help you manage large datasets and improve workflow efficiency:

- **Use descriptive folder structures** - Organize data by project, date, or data type.
- **Document your data** - Keep README files describing dataset contents, formats, and processing steps.
- **Remove intermediate files** - Clean up temporary and intermediate files after processing to save storage space.
- **Track file versions** - Use version control for code and configuration files.
- **Monitor storage usage** - Regularly check your storage quotas. See :ref:`hpc-storage-check` for detailed instructions.


Getting Help
============

If you encounter issues during data transfer or have questions about data management, please reach out:

- **Support Portal:** Use the :ref:`support-tickets` to submit a request to the CoSTAR support team.
- **Microsoft Teams:** Join the `Teams Community <https://teams.microsoft.com/l/team/19%3A1zWAQ0evBFgRyMy5w1jZGKR4I6YYnyYvOLWyg_CvYao1%40thread.tacv2/conversations?groupId=a9caa11e-fb07-40fa-b70d-a7ac2e510e54&tenantId=6b902693-1074-40aa-9e21-d89446a2ebb5>`_ to connect with other users and get updates about the cluster.
- **Check cluster status:** Monitor the Teams channel for any announcements about cluster maintenance or data transfer issues.