File Transfers - Revision history

Alex: /* Transfer files to/from Arrow and Condo servers: */

2024-12-02T20:40:13Z

Transfer files to/from Arrow and Condo servers:

← Older revision		Revision as of 20:40, 2 December 2024
Line 16:		Line 16:
	'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''		'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''

	Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''

Alex: /* Transfer files to/from Arrow and Condo servers: */

2024-12-02T20:39:57Z

Transfer files to/from Arrow and Condo servers:

← Older revision		Revision as of 20:39, 2 December 2024
Line 7:		Line 7:
	:[https://cunyhpc.csi.cuny.edu/zircon_d8/sites/default/files/Globus_Instructions.pdf Details on connecting via Globus can be found here.]		:[https://cunyhpc.csi.cuny.edu/zircon_d8/sites/default/files/Globus_Instructions.pdf Details on connecting via Globus can be found here.]
	:• '''cea.csi.cuny.edu''': Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. '''sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt'''		:• '''cea.csi.cuny.edu''': Only Secure FTP (SFTP) can be used to transfer files to cea.csi.cuny.edu. Each of the servers is mounted on cea under its own name. Thus the users could place their files on particular server simply by preceding /scratch with /<name of the server> e.g. '''sftp> put TEST.txt /penzias/scratch/john.doe/TEST2.txt'''
	:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from ~~Penzias~~ file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.		:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.

	=== <u>Transfer files to/from Arrow and Condo servers:</u> ===		=== <u>Transfer files to/from Arrow and Condo servers:</u> ===
Line 16:		Line 16:
	'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''		'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''

	Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.
	~~'''(<user_id>@chizen.csi.cuny.edu) Password:'''~~

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''

Alex: /* Transfer files to/from Arrow and Condo servers: */

2024-12-02T20:39:02Z

Transfer files to/from Arrow and Condo servers:

← Older revision		Revision as of 20:39, 2 December 2024
Line 17:		Line 17:

	Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.
			'''(<user_id>@chizen.csi.cuny.edu) Password:'''

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''

Alex: /* Transfer files to/from Arrow and Condo servers: */

2024-12-02T20:38:43Z

Transfer files to/from Arrow and Condo servers:

← Older revision		Revision as of 20:38, 2 December 2024
Line 16:		Line 16:
	'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''		'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''

	Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.
	~~'''(<user_id>@chizen.csi.cuny.edu) Password:'''~~

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''

Alex: /* Transfer files to/from Arrow and Condo servers: */

2024-12-02T20:38:21Z

Transfer files to/from Arrow and Condo servers:

← Older revision		Revision as of 20:38, 2 December 2024
Line 12:		Line 12:
	For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the '''cea''' not be applied as main data move node so the data has to be tunneled from/to '''Arrow''' via '''scp or sftp''' using chizen as intermediate post. Note that files '''must not be explicitly copied''' to Chizen and then copied to Arrow on second pass. <u>Files copied to Chizen and left there will be deleted.</u>		For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the '''cea''' not be applied as main data move node so the data has to be tunneled from/to '''Arrow''' via '''scp or sftp''' using chizen as intermediate post. Note that files '''must not be explicitly copied''' to Chizen and then copied to Arrow on second pass. <u>Files copied to Chizen and left there will be deleted.</u>

	Users must use '''tunneled sftp secure shell''' to tunnel their files from/to /scratch directory on Arrow. To put or get files from local machine to directory in Arrow please open '''sftp dialog''' by typing:		Users must use '''tunneled sftp secure shell''' to tunnel their files from/to '''/scratch''' directory on Arrow. To put or get files from local machine to directory in Arrow please open '''sftp dialog''' by typing:

	'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''		'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''

Alex: /* Transfer files to/from Arrow cluster and condo servers: */

2024-12-02T20:36:50Z

Transfer files to/from Arrow cluster and condo servers:

← Older revision		Revision as of 20:36, 2 December 2024
Line 9:		Line 9:
	:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from Penzias file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.		:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from Penzias file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.

	=== <u>Transfer files to/from Arrow ~~cluster~~ and ~~condo~~ servers:</u> ===		=== <u>Transfer files to/from Arrow and Condo servers:</u> ===
	For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the '''cea''' not be applied as main data move node so the data has to be tunneled from/to '''Arrow''' via '''scp or sftp''' using chizen as intermediate post. Note that files '''must not be explicitly copied''' to Chizen and then copied to Arrow on second pass. <u>Files copied to Chizen and left there will be deleted.</u>		For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the '''cea''' not be applied as main data move node so the data has to be tunneled from/to '''Arrow''' via '''scp or sftp''' using chizen as intermediate post. Note that files '''must not be explicitly copied''' to Chizen and then copied to Arrow on second pass. <u>Files copied to Chizen and left there will be deleted.</u>

Alex: /* Transfer files to/from Arrow cluster and condo servers: */

2024-12-02T20:36:05Z

Transfer files to/from Arrow cluster and condo servers:

← Older revision		Revision as of 20:36, 2 December 2024
Line 17:		Line 17:

	Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.
			'''(<user_id>@chizen.csi.cuny.edu) Password:'''

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''

Alex: /* Transfer files to/from Arrow cluster: */

2024-12-02T20:35:37Z

Transfer files to/from Arrow cluster:

← Older revision		Revision as of 20:35, 2 December 2024
Line 9:		Line 9:
	:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from Penzias file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.		:[[File:SFTP example.png\|left\|thumb\|421x421px\|Use sftp to transfer files to/from Penzias file systems]]• '''iRODS''': The data grid/data management tool provided by CUNY HPC Center for accessing the '''SR1''' resource. iRODS clients ('''<font face="courier">iput, iget, irsync</font>''') provide a data transfer mechanism featuring bulk upload and parallel streams.

	=== <u>Transfer files to/from Arrow cluster:</u> ===		=== <u>Transfer files to/from Arrow cluster and condo servers:</u> ===
	For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the ~~acquired storage system will~~ not be applied as main ~~storage for HPCC and consequently~~ the ~~files can only~~ be tunneled from/to Arrow. Note that files '''must not be explicitly copied''' on Chizen and then copied to Arrow. Files copied to Chizen and left there will be deleted.		For the first project year the storage purchased with NSF grant will be accessible only by the core members of the NSF grant. For that time the '''cea''' not be applied as main data move node so the data has to be tunneled from/to '''Arrow''' via '''scp or sftp''' using chizen as intermediate post. Note that files '''must not be explicitly copied''' to Chizen and then copied to Arrow on second pass. <u>Files copied to Chizen and left there will be deleted.</u>

	Users ~~can~~ use '''tunneled sftp secure shell''' to tunnel their files from/to /scratch directory on Arrow. To put or get files from local machine to directory in Arrow please open sftp dialog by typing:		Users must use '''tunneled sftp secure shell''' to tunnel their files from/to /scratch directory on Arrow. To put or get files from local machine to directory in Arrow please open '''sftp dialog''' by typing:

	'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''		'''sftp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:/scratch/<user_id>'''

	Then provide twice the user's password. ~~e.g~~.		Then provide twice the user's password and look for sftp prompt. The first password will be used to open connection between local server/desktop/laptop and chizen and the second to establish tunneled connection to arrow server.

	'''(<user_id>@chizen.csi.cuny.edu) Password:'''		'''(<user_id>@chizen.csi.cuny.edu) Password:'''
Line 26:		Line 26:
	'''sftp>'''		'''sftp>'''

	~~The first password will be used to open connection between local server/desktop/laptop and chizen and~~ the ~~second to establish tunneled connection to arrow server. The~~ files can be ~~obtrained~~ from /scratch/<user_id> with command '''get''' and put ~~there~~ with command '''put'''. For example, the following dialog shows how to get copy of file '''test.sh''' from '''/scratch/<user_id>''' to local directory and changing the name of the copied file to '''test5.sh.'''		Within sftp the files can be obtained from '''/scratch/<user_id>''' with the command '''get''' and put in '''/scratch/<user_id>''' with the command '''put'''. For example, the following dialog shows how to get copy of file '''test.sh''' from '''/scratch/<user_id>''' to local directory and changing the name of the copied file to '''test5.sh.'''

	'''sftp> get test.sh test5.sh'''		'''sftp> get test.sh test5.sh'''

Alex: /* Transfer files to/from Arrow cluster: */

2024-12-02T20:28:09Z

Transfer files to/from Arrow cluster:

← Older revision		Revision as of 20:28, 2 December 2024
Line 44:		Line 44:
	'''scp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:<file_name_on_scratch> <file_name_on_local_machine>'''		'''scp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:<file_name_on_scratch> <file_name_on_local_machine>'''

	Note that white symbols work for sftp.		Note that white symbols work for both <u>sftp and scp.</u>

Alex at 20:26, 2 December 2024

2024-12-02T20:26:28Z

← Older revision		Revision as of 20:26, 2 December 2024
Line 36:		Line 36:
	'''sftp>''' bye		'''sftp>''' bye

			Users who want a quick copy '''from local machine''' to '''arrow:/scratch/<user_id>''' can also use the tunneled secure copy '''scp''' via chizen by typing:

	~~The users~~ who want ~~only (one directional)~~ quick copy '''from local machine to arrow:/scratch/<user_id>''' can also use the tunneled secure copy '''scp''' via chizen by typing:

	'''<span class="nv">scp</span><span class="w"> </span><span class="o">-</span><span class="nv">J</span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">chizen</span><span class="p">.</span><span class="nv">csi</span><span class="p">.</span><span class="nv">cuny</span><span class="p">.</span><span class="nv">edu</span> <span class="nv"><file_to_transfer></span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">arrow</span><span class="p">:/</span><span class="nv">scratch</span><span class="p">/</span><span class="o"><</span><span class="nv">user_id</span><span class="p">/.</span>'''		'''<span class="nv">scp</span><span class="w"> </span><span class="o">-</span><span class="nv">J</span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">chizen</span><span class="p">.</span><span class="nv">csi</span><span class="p">.</span><span class="nv">cuny</span><span class="p">.</span><span class="nv">edu</span> <span class="nv"><file_to_transfer></span><span class="w"> </span><span class="nv"><user_id></span><span class="err">@</span><span class="nv">arrow</span><span class="p">:/</span><span class="nv">scratch</span><span class="p">/</span><span class="o"><</span><span class="nv">user_id</span><span class="p">/.</span>'''

	~~To get file~~ from scratch/<user_id> ~~type~~:		Users who want a quick copy '''from arrow:scratch/<user_id>''' to '''local machine''' via chizen tunneled scp:

	scp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:<file_name_on_scratch> <file_name_on_local_machine>		'''scp -J <user_id>@chizen.csi.cuny.edu <user_id>@arrow:<file_name_on_scratch> <file_name_on_local_machine>'''

	Note that white symbols work for sftp.		Note that white symbols work for sftp.