Data Transfers

The Center has deployed Data Transfer Nodes (DTN), which are servers dedicated to transfer data between remote sites and the DSMS (/global/u/<userid>), or between remote sites and the /scratch/<userid> on the designated HPC system on which the user has an account. DTNs are tuned to transfer data efficiently.

There are three methods of transferring data between the CUNY HPC systems and the rest of the world:

• Globus Online: The preferred method for large files, with extra features for parallel data streams, auto-tuning and auto-fault recovery. Globus online is to transfer files between systems—between the CUNY HPC Center resources and XSEDE facilities, or even users’ desktops. A typical transfer rate ranges from 100 to 400 Mbps.

• SCP/SFTP: For smaller files (<1GB). Secure Copy (SCP) and Secure FTP (SFTP) can be used to securely transfer files between two hosts using the Secure Shell (SSH) protocol. A typical transfer rate ranges from 1 to 30 megabytes/second.

• iRODS: The data grid/data management tool provided by CUNY HPC Center for accessing the “SR1” resource. iRODS clients (iput, iget, irsync) provide a data transfer mechanism featuring bulk upload and parallel streams. Not all methods are offered for all file systems. Here is a summary on the available methods per file space:

File System	Available Transfer Methods	URL
User: /global/u/<userid>	Globus Online SFTP	cunyhpc#cea cea.csi.cuny.edu
Project: SR1/<PID>	iRODS	irods.csi.cuny.edu
Scratch: /scratch/<userid> on PENZIAS /scratch/<userid> on ANDY /scratch/<userid> on KARLE /scratch/<userid> on SALK /scratch/<userid> on BOB	Globus Online SFTP	cunyhpc#cea cea.csi.cuny.edu