File Transfers: Difference between revisions
Jump to navigation
Jump to search
No edit summary Tags: Reverted Visual edit: Switched |
|||
| (8 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
There are | There are several methods for transferring files to HPCC depend on tier and file system used. | ||
=== <u>Basic and advanced tier file transfer</u> === | |||
Basic and advanced tiers are attached to DSMS file system as it described in figure 1. above. The users utilizing these resources may transfer files via '''Globus online''' or via '''Cea - the file transfer node as''' it is described below In addition the users of these tiers may use iRODS data management system as described below. | |||
:• '''Globus Online''': The preferred method for large files, with extra features for parallel data streams, auto-tuning and auto-fault recovery. Globus online is to transfer files between systems—between the CUNY HPC Center resources and XSEDE facilities, or even users’ desktops. A typical transfer rate ranges from 100 to 400 Mbps. | :• '''Globus Online''': The preferred method for large files, with extra features for parallel data streams, auto-tuning and auto-fault recovery. Globus online is to transfer files between systems—between the CUNY HPC Center resources and XSEDE facilities, or even users’ desktops. A typical transfer rate ranges from 100 to 400 Mbps. | ||
: | :[https://cunyhpc.csi.cuny.edu/zircon_d8/sites/default/files/Globus_Instructions.pdf Details on connecting via Globus can be found here.] | ||
:• '''cea.csi.cuny.edu''': Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. '''sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt''' | :• '''cea.csi.cuny.edu''': Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. '''sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt''' | ||
:[[File:SFTP example.png|left|thumb|421x421px|Use sftp to transfer files to/from Penzias file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams. | |||
: | === <u>Transfer files to/from Arrow cluster:</u> === | ||
For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the acquired storage system will not be applied as main storage for HPCC and consequently the files can only be tunneled to Arrow. Note that files must and cannot be copied on Chizen and then copied to Arrow. Files copied to Chizen will be deleted. The users of that resource should consult HPCC for further guidance. | |||
<span class="nv">scp</span><span class="w"> </span><span class="o">-</span><span class="nv">J</span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">chizen</span><span class="p">.</span><span class="nv">csi</span><span class="p">.</span><span class="nv">cuny</span><span class="p">.</span><span class="nv">edu</span><span class="w"> </span><span class="nv"><file_to_transfer></span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">arrow</span><span class="p">:/</span><span class="nv">scratch</span><span class="p">/</span><span class="o"><</span><span class="nv">user_id</span><span class="p">/.</span> | |||
Latest revision as of 19:05, 15 September 2023
There are several methods for transferring files to HPCC depend on tier and file system used.
Basic and advanced tier file transfer
Basic and advanced tiers are attached to DSMS file system as it described in figure 1. above. The users utilizing these resources may transfer files via Globus online or via Cea - the file transfer node as it is described below In addition the users of these tiers may use iRODS data management system as described below.
- • Globus Online: The preferred method for large files, with extra features for parallel data streams, auto-tuning and auto-fault recovery. Globus online is to transfer files between systems—between the CUNY HPC Center resources and XSEDE facilities, or even users’ desktops. A typical transfer rate ranges from 100 to 400 Mbps.
- Details on connecting via Globus can be found here.
- • cea.csi.cuny.edu: Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt
- • iRODS: The data grid/data management tool provided by CUNY HPC Center for accessing the SR1 resource. iRODS clients (iput, iget, irsync) provide a data transfer mechanism featuring bulk upload and parallel streams.
Transfer files to/from Arrow cluster:
For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the acquired storage system will not be applied as main storage for HPCC and consequently the files can only be tunneled to Arrow. Note that files must and cannot be copied on Chizen and then copied to Arrow. Files copied to Chizen will be deleted. The users of that resource should consult HPCC for further guidance.
scp -J <user_id>@chizen.csi.cuny.edu <file_to_transfer> <user_id>@arrow:/scratch/<user_id/.