HPCC Wiki - Recent changes [en]

Training & Workshops

Alex — Mon, 23 Mar 2026 19:31:53 GMT

← Older revision		Revision as of 19:31, 23 March 2026
(One intermediate revision by the same user not shown)
Line 1:		Line 1:
	The CUNY HPCC provides training course and organizes seminars on various HPC topics. The training courses are provided at no cost and may be held at any CUNY campus site, at the CUNY HPCC at College of Staten Island, or at the Graduate Center. The training course at the Graduate Center and its '''online''' version ~~(during COVID-19)~~ is course on parallel programming and use of HPC architectures. ~~Therefore it~~ covers various topics from basic SLURM scripting to basic GPU programming to intermediate parallel programming with use of MPI and OpenACC. Please note that lectures cover each topic systematically so particular topic may be discussed in several lectures. Users who do want to attend the course should send an e-mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] and ask for registration. All participants will get student account on CUNY-HPCC servers unless they already have one. ~~Registered users will receive active Zoom link for the upcomming lecture 24 hours in advance (Tuesday by 2 PM).~~		The CUNY HPCC provides training course and organizes seminars on various HPC topics. The training courses are provided at no cost and may be held at any CUNY campus site, at the CUNY HPCC at College of Staten Island, or at the Graduate Center. The training course at the Graduate Center and its '''online''' version is course on parallel programming and use of HPC architectures. The on-site course takes place if enough students express interest. It covers various topics from basic SLURM scripting to basic GPU programming to intermediate parallel programming with use of MPI and OpenACC. Please note that lectures cover each topic systematically so particular topic may be discussed in several lectures. Users who do want to attend the course should send an e-mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] and ask for registration. All participants will get student account on CUNY-HPCC servers unless they already have one.

	In addition HPCC provides in person ~~(online only during COVID-19)~~ consultations with individuals or small groups of users every Wednesday. These consultations should help new users and those with no experience to start quickly with HPCC resources. At that time users may discuss their particular problems and get guidance in development of their own parallel scientific code(s). Please send a mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] or to [[mailto:alexander.tzanov@.csi.cuny.edu]] for available time slots not later than 3PM on ~~previous day (Tuesday)~~. HPCC will make all efforts to accommodate all users so any time slot may be shared by several users.		In addition HPCC provides in person and zoom consultations with individuals or small groups of users every Wednesday - 11AM to 3 PM. The interested users should register by sending e-mail to alex.tzanov@csi.cuny.edu by Mon on the same week. These consultations should help new users and those with no experience to start quickly with HPCC resources. At that time users may discuss their particular problems and get guidance in development of their own parallel scientific code(s). Please send a mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] or to [[mailto:alexander.tzanov@.csi.cuny.edu]] for available time slots not later than 3PM on Mon. HPCC will make all efforts to accommodate all users so any time slot may be shared by several users.

	For any additional information, please send an email to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]].		For any additional information, please send an email to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]]. Every Wednesday between 11 AM and 2 PM, HPCC conducts remote help session/consultation. Please sent request for invite to hpchelp@csi.cuny.edu. <u>'''''Note that the consultation is open only for CUNY faculty, students and staff.'''''</u> <br/>
	<br/><br/>

	<h2>Schedule </h2>		<h2>Schedule </h2>
Line 31:		Line 30:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~1/29/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Introduction to HPC and HPCC</p></td>		<td width="303" valign="top"><p align="center">Introduction to HPC and HPCC</p></td>
Line 37:		Line 36:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~2/5/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Introduction to parallel programming </p></td>		<td width="303" valign="top"><p align="center">Introduction to parallel programming </p></td>
Line 43:		Line 42:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~2/19/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Distributed parallel programming with MPI part 1</p></td>		<td width="303" valign="top"><p align="center">Distributed parallel programming with MPI part 1</p></td>
Line 49:		Line 48:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~2/26/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 2</p></td>		<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 2</p></td>
Line 55:		Line 54:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~3/5/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 3</p></td>		<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 3</p></td>
Line 61:		Line 60:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~3/19/26~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 4</p></td>		<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 4</p></td>
Line 67:		Line 66:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~3/26/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 5</p></td>		<td width="303" valign="top"><p align="center">Distributed Parallel programming with MPI part 5</p></td>
Line 73:		Line 72:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~4/2/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center"> Distributes Parallel programming with MPI – Hands on </p></td>		<td width="303" valign="top"><p align="center"> Distributes Parallel programming with MPI – Hands on </p></td>
Line 79:		Line 78:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~4/9/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">GPGPU programming part 1.</p></td>		<td width="303" valign="top"><p align="center">GPGPU programming part 1.</p></td>
Line 85:		Line 84:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~4/23/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">GPGPU programming part 2</p></td>		<td width="303" valign="top"><p align="center">GPGPU programming part 2</p></td>
Line 91:		Line 90:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~4/30/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">GPGPU - hands on </p></td>		<td width="303" valign="top"><p align="center">GPGPU - hands on </p></td>
Line 97:		Line 96:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~5/7/25~~</td>		<td width="72" valign="top"></td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Easy GPU programming with OpenACC part 1. </p></td>		<td width="303" valign="top"><p align="center">Easy GPU programming with OpenACC part 1. </p></td>
Line 103:		Line 102:
	</tr>		</tr>
	<tr>		<tr>
	<td width="72" valign="top">~~5/14/25~~		<td width="72" valign="top"></td>
	</td>
	<td width="48" valign="top"><p align="center">WED</p></td>		<td width="48" valign="top"><p align="center">WED</p></td>
	<td width="303" valign="top"><p align="center">Easy GPU programming with OpenACC part 2 </p></td>		<td width="303" valign="top"><p align="center">Easy GPU programming with OpenACC part 2 </p></td>

Main Page

Alex — Sat, 14 Mar 2026 15:27:23 GMT

Recovery of operational costs

Show changes

Data Storage and Management System

Alex — Fri, 13 Mar 2026 00:46:13 GMT

/scratch

← Older revision		Revision as of 00:46, 13 March 2026
(One intermediate revision by the same user not shown)
Line 1:		Line 1:
	This file system is resides on Hybrid Parallel File System (HPFS).		This file system resides on '''Hybrid Parallel File System (HPFS).'''

	__TOC__		__TOC__
	=="Home" directories are on <font face="courier">/global/u</font>==		=="Home" directories are on <font face="courier">/global/u</font>==
	'''<font face="courier">/global/u</font>''' is ~~a standard Unix~~ file system ~~that~~ holds the home directories of individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<font face="courier"><font color="red"><userid></font color></font>''' and a 50 GB allocation of disk space for home directories on '''<font face="courier">/global/u/<font color="red"><userid></font color></font>'''. These '''home''' directories are on ~~the '''DSMS''', not~~ on ~~the HPC systems, but~~ can be ~~accessed~~ from ~~any Center system~~. All home directories are backed up on weekly basis.		'''<font face="courier">/global/u</font>''' is partition in parallel high performance Linux file system based on HPE parallel file system (HPFS). It holds the home directories of '''all''' individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<font face="courier"><font color="red"><userid></font color></font>''' and a 100 GB allocation of disk space for home directories on '''<font face="courier">/global/u/<font color="red"><userid></font color></font>'''. These '''home''' directories are on global file system mounted only on login nodes. There is no local storage on nodes . That means that 1. data can be accesses only from PFFS and login node(s) 2. no local storage on nodes is available. Codes that write intermediate results on disk are typically slow and should be run only on condo nodes that have local disk storage. All home directories are backed up on weekly basis.

	==<font face="courier">/scratch</font>==		==<font face="courier">/scratch</font>==
	~~Disk storage~~ on ~~the HPC systems is used only for '''~~scratch~~''' files.~~ '''scratch''' files are temporary and are '''<font color="red">not backed up</font color>'''. '''<font face="courier">/scratch</font>''' is used by jobs ~~queued for or in execution~~. Output from jobs may temporarily be ~~located~~ in '''<font face="courier">/scratch</font>'''.		/Scratch is fast file system from which all jobs start. There is no quota on scratch, but '''scratch''' files are temporary and are '''<font color="red">not backed up</font color>'''. Single user cannot however take whole space with his/her data. That means users can run data sets that exceed their home space but they cannot use /scratch for storage. It is important to understand that '''<font face="courier">/scratch</font>''' <u>must be used only for submitting jobs.</u> Output from jobs may ONLY temporarily be stored on '''<font face="courier">/scratch (up to 10 days). Consequently in</font>''' order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''<font face="courier">/scratch</font>''' from '''<font face="courier">/global/u</font>''' using UNIX commands and/or from '''<font face="courier">SR1</font>''' using '''iRODS''' commands. The later is mounting data directly from project space. Note that SR is slower file system so large files is better to be staged from /home directories. Files in '''<font face="courier">/scratch</font>''' are '''automatically relocated to local archive''' when their inactive residence on scratch exceeds 90 days. Local archive has limited capacity and serves as data buffer, so '''strict policies of cleaning up temporary archive are in place'''. Upon relocation the user will get a warning ( via e-mail) and must either move files to his/her home directory or to SR1. Note that files left in temp archive will be purged after 30 days. Users must not '''store valuable data or compiled codes on scratch''' since these are static type of files.

	In order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''<font face="courier">/scratch</font>''' from '''<font face="courier">/global/u</font>''' using UNIX commands and/or from '''<font face="courier">SR1</font>''' using '''iRODS''' commands.

	Files in '''<font face="courier">/scratch</font>''' on a system are '''automatically purged''' when (1) usage reaches 70% of available space, or (2) file residence on scratch exceeds two weeks, whichever occurs first.


Line 16:		Line 13:
	==“Project” directories==		==“Project” directories==

	'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands. In iRODS terminology, a “collection” is the equivalent of “directory”.		'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands. In iRODS terminology, a “collection” is the equivalent of “directory”.

	A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files. A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist. The PI is the individual responsible to the University or a granting agency for the “Project”. The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”.		A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files. A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist. The PI is the individual responsible to the University or a granting agency for the “Project”. The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”. Project data are stored on project space on main file system. Valuable parts of the projects must be curated by PI's and stored in HPCC long term archive.

	~~Additional information on the '''DSMS''' is available in Section 4 of the User Manual~~ <br />		<br />
	~~http://www.csi.cuny.edu/cunyhpc/pdf/User_Manual.pdf~~


Line 37:		Line 34:
	</font>		</font>

	If working with '''SR1''':<br />		If working with '''SR1 (storage repository)''':<br />
	~~<font face="courier">~~
	cd /scratch/<font color="red"><userid></font color>		cd /scratch/<font color="red"><userid></font color>
	mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>		mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>

Main Page

Alex — Mon, 09 Mar 2026 17:16:48 GMT

Empire AI and CUNY-HPCC

← Older revision		Revision as of 17:16, 9 March 2026
(14 intermediate revisions by the same user not shown)
Line 3:		Line 3:
	[[Image:hpcc-panorama3.png]]		[[Image:hpcc-panorama3.png]]

	The City University of New York (CUNY) High Performance Computing Center (HPCC) is located ~~on the~~		== Mission of CUNY-HPCC ==
	campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. HPCC		The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.
	~~goals are~~ to:

	:*~~Support the scientific~~ computing ~~needs of~~ CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.		== Empire AI and CUNY-HPCC ==
	:*~~Create~~ opportunities for the CUNY research community to develop new partnerships with the government and private sectors~~; and~~		The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.
	:*Leverage the ~~HPC Center~~ capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
			== CUNY-HPCC services ==
			CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

			:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
			:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
			:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
			:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
			:*Maintains tickets for all CUNY users with allocation on EAI.

	==Organization of systems and data storage (architecture)==		==Organization of systems and data storage (architecture)==