File Transfers
There are several methods for transferring files to HPCC depend on tier and file system used.
Basic and advanced tier file transfer
Basic and advanced tiers are attached to DSMS file system as it described in figure 1. above. The users utilizing these resources may transfer files via Globus online or via Cea - the file transfer node as it is described below In addition the users of these tiers may use iRODS data management system as described below.
- • Globus Online: The preferred method for large files, with extra features for parallel data streams, auto-tuning and auto-fault recovery. Globus online is to transfer files between systems—between the CUNY HPC Center resources and XSEDE facilities, or even users’ desktops. A typical transfer rate ranges from 100 to 400 Mbps.
- Details on connecting via Globus can be found here.
- • cea.csi.cuny.edu: Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt
- • iRODS: The data grid/data management tool provided by CUNY HPC Center for accessing the SR1 resource. iRODS clients (iput, iget, irsync) provide a data transfer mechanism featuring bulk upload and parallel streams.
Transfer files to/from Arrow and Condo servers:
For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the cea not be applied as main data move node so the data has to be tunneled from/to Arrow via scp or sftp using chizen as intermediate post. Note that files must not be explicitly copied to Chizen and then copied to Arrow on second pass. Files copied to Chizen and left there will be deleted.
Users must use tunneled sftp secure shell to tunnel their files from/to /scratch directory on Arrow. To put or get files from local machine to directory in Arrow please open sftp dialog by typing:
sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>
Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.
(<user_id>@chizen.csi.cuny.edu) Password:
(<user_id>@arrow) Password:
Changing to: /scratch/<user_id>
sftp>
Within sftp the files can be obtained from /scratch/<user_id> with the command get and put in /scratch/<user_id> with the command put. For example, the following dialog shows how to get copy of file test.sh from /scratch/<user_id> to local directory and changing the name of the copied file to test5.sh.
sftp> get test.sh test5.sh
Fetching /pfssfs1/scratch/<user_id>/test.sh to test5.sh
test.sh 100% 172 8.7KB/s 00:00
sftp> bye
Users who want a quick copy from local machine to arrow:/scratch/<user_id> can also use the tunneled secure copy scp via chizen by typing:
scp -J <user_id>@chizen.csi.cuny.edu <file_to_transfer> <user_id>@arrow:/scratch/<user_id/.
Users who want a quick copy from arrow:scratch/<user_id> to local machine via chizen tunneled scp:
scp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:<file_name_on_scratch> <file_name_on_local_machine>
Note that white symbols work for both sftp and scp.