Main Page

The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. HPCC goals are to:

• Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
• Create opportunities for the CUNY research community to develop new partnerships with the government and private sectors; and
• Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

Organization of systems and data storage (architecture)

All user data and project data are kept on Data Storage and Management System (DSMS) which is mounted only on login node(s) of all servers. Consequently, no jobs can be started directly from DSMS storage. Instead, all jobs must be submitted from a separate (fast but small) /scratch file system mounted on all computational nodes and on all login nodes. As the name suggests, the /scratch file system is not home directory for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

• /scratch/<userid> – this is temporary workspace on the HPC systems

• /global/u/<userid> – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

• In some instances a user will also have use of disk space on the DSMS in /cunyZone/home/<projectid> (IRods).

The /global/u/<userid> directory has quota (see below for details) while the /scratch/<userid> do not have. However the /scratch space is cleaned up following the rules described below. There are no guarantees of any kind that files in /scratch will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called 'chizen. The Data Transfer Node called Cea allows file transfer from/to remote sites directly to/from /global/u/<userid> or to/from /scratch/<userid>

HPC systems

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

Overview of Computational architectures:

SMP servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

Cluster is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a node. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called Arrow. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster Herbert dedicated only to education.

Distributed shared memory computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the NUMA systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the NUMA node at Arrow named Appel. This node does not have GPU.

Infrastructure systems:

o Master Head Node (MHN/Arrow) is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o Chizen is a redundant gateway server which provides access to protected HPCC domain.

o Cea is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. Cea is accessible directly (not only via Chizen), but allows only limited set of shell commands.

Table 1 below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

Master Head Node	Sub System	Tier	Type	Type of Jobs	Nodes	CPU Cores	GPUs	Mem/node	Mem/core	Chip Type	GPU Type and Interface
Arrow	Penzias	Advanced	Hybrid Cluster	Sequential & Parallel jobs w/wo GPU	66	16	2	64 GB	4 GB	SB, EP 2.20 GHz	K20m GPU, PCIe v2
				Sequential & Parallel jobs	1	24	-	1500 GB	62 GB	HL, 2.30 GHz	-
						36	-	768 GB	21 GB		-
						24	-	768 GB	32 GB		-
	Appel		NUMA	Massive Parallel, sequential, OpenMP	1	384	-	11 TB	28 GB	IB, 3 GHz	-
	Cryo		SMP	Sequential and Parallel jobs, with GPU	1	40	8	1500 GB	37 GB	SL, 2.40 GHz	V100 (32GB) GPU, SXM
	Blue Moon		Hybrid Cluster	Sequential and Parallel jobs w/wo GPU	24	32	-	192 GB	6 GB	SL, 2.10 GHz	-
					2	32	2				V100(16GB) GPU, PCIe
	Karle		SMP	Visualization, MATLAB/Mathematica	1	36*	-	768 GB	21 GB	HL, 2.30 GHz	-
	Chizen		Gateway	No jobs allowed	-
	CFD	Condo	SMP	Parallel, Seq, OpenMP	1	48	2	768 GB		EM, 4.8 GHz	A40, PCIe, v4
					1	48	-	512 GB		ER, 4.3 GHz	-
	PHYS	Condo	SMP		1	48	2	640 GB		ER, 4 GHz	L40, PCIe, v4
					1	48	-	512 GB		ER, 4.3 GHz	-
	CHEM	Condo	SMP		1	48	2	256 GB		EM, 2.8 GHz	A30, PCIe, v4
					1	128	8	512 GB		ER, 2.0 GHz	A100/40, SXM
	ASRC	Condo	SMP		1	48	2	256 GB		ER, 2.8 GHz	A30, PCIe, v4

Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

Recovery of operational costs

CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only operational costs with no profit for HPCC. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on unit-hour. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:

Definitions of unit-hour

Type of resource	Unit-hour	For V100, A30, A40 or L40	For A100
CPU unit	1 cpu core/hour	--	--
GPU unit	(4 cpu cores + 1 GPU thread )/hour	4 cpu cores + 1 GPU	4 cpu cores and 1/7 A100

HPCC access plans

a.     Minimum access (MAP):

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

·     A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

·     B:  Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

·      C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.  

The MAP users get charged per CPU/GPU hour at low rate of $0.015 per cpu hour and $0.09 per GPU hour.

Cost recovery fees for MAP users

Job	Cpu cores	GPU	Cost/hour
1 core no GPU	1	0	$0.015/hour
16 cores no GPU	4	0	$0.24/hour
4 cores + 1 GPU	4	1	$0.15/hour
16 cores + 1 GPU	16	1	$0.33/hour
16 cores + 2 GPU	16	2	$0.42/hour
32 cores + 2 GPU	32	2	$0.66/hour
40 cores + 8 GPU	40	8	$1.32/hour

b.     Computing on demand (CODP)

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are $0.018 per cpu hour and $0.11 per GPU hour.  In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month.  The examples in following table explain the fees structure:

Cost recovery fees for CODP plan

Job	Cpu cores	GPU	Cost/hour
1 core no GPU	1	0	$0.018/hour
16 cores no GPU	16	0	$0.288/hour
4 cores + 1 GPU	4	1	$0.293/hour
16 cores + 1 GPU	16	1	$0.334/hour
32 cores + 1 GPU	32	1	$0.666/hour
32 cores + 2 GPU	32	2	$0.756/hour

c. Leasing node(s) (LNP)

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined.  In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.  

Lease node(s) fees for MAP users

Job (MAP users)	Cpu cores	GPU	Cost/30 days
1 core no GPU	1	0	NA
16 cores no GPU	16	0	$172.80
32 cores no GPU	32	0	$264.96
16 cores + 2 GPU	16	1	$302.40
32 cores + 2 GPU	32	2	$475.20
40 cores + 8 GPU	40	8	$760.0
64 cores + 8 GPU	64	8	$950.40

Fees for lease a node(s) for non -MAP users

Job (non-MAP users)	Cpu cores	GPU	Cost/month
1 core no GPU	1	0	NA
16 cores no GPU	16	0	$249.82
32 cores no GPU	32	0	$497.64
16 cores + 1 GPU	16	2	$443.23
32 cores + 2 GPU	32	2	$886.64
40 cores + 8 GPU	40	8	$1399.68

d.     Condo Ownership (COP)

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow”  (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.

Condo owners costs per year

Type of condo node	Cpu cores	GPU	Cost/year
Large hybrid SXM	128	8	$4518.92
Small hybrid	48	2	$1540.54
Medium compute	96	0	$2464.86
Large compute	128	0	$3286.49

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.

Type of nodes and lease fees for condo nodes

Type of node	Renters cost/month	Long term (90+ days) rent cost/month	CPU/node	CPU type	GPU/node	GPU type	GPU interface
Laghe Hybrid	$602.52	$564.86	128	EPYC, 2.2 GHz	8	A100/80	SXM
Small Hybrid	$205.41	$192.57	48	EPYC, 2.8 GHz	2	A40, A30, L40	PCIe v4
Medium Non GPU	$328.65	$308.11	96	EPYC, 4.11GHz	48	None	NA
Lagre Non GPU	$438.20	$410.81	128	EPYC, 2.0 GHz	128	None	NA

Free time

In order to establish a project all new users from colleges that participate in MAP (B and C only) plan are entitled to free 11520 CPU hours and 1440 GPU hours. Any additional hours are charged with MAP plan rates. Note that free time is per user account not per project so any user can have free time only once. External collaborators to CUNY are not normally eligible for free time. Please contact CUNY-HPCC director for further details.

Support for research grants

All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later that require computational resources must include budget for cost recovery fees at CUNY-HPCC. For a project the PI can choose between:

• lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
• use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
• participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should contact the Director of CUNY-HPCC Dr. Alexander Tzanov (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.    

Partitions and jobs

The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in partitions. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).

Partition	Max cores/job	Max jobs/user	Total cores/group	Time limits	Tier		GPU types
partnsf	128	50	256	240 Hours	Advanced		K20m, V100/16, A100/40
partchem	128	50	256	No limit	Condo		A100/80, A30
partcfd	96	50	96	No limit	Condo		A40
partsym	96	50	96	No limit	Condo		A30
partasrc	48	16	16	No limit	Condo		A30
partmatlabD	128	50	256	240 Hours	Advanced		V100/16,A100/40
partmatlabN	384	50	384	240 Hours	Advanced		None
partphys	96	50	96	No limit	Condo		L40

• partnsf is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
• partchem is CONDO partition.
• partphys is CONDO partition
• partsym is CONDO partition
• partasrc is CONDO partition
• partmatlabD partition allows to run MATLAB's Distributes Parallel Server across main cluster.
• partmatlabN partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
• partdev is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

Hours of Operation

In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

User Support

Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY community in parallel programming techniques, HPC computing architecture, and the essentials of using our systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

 hpchelp@csi.cuny.edu 

Warnings and modes of operation

1. hpchelp@csi.cuny.edu is for questions and accounts help communication only and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address. Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. Do not send questions to individual CUNY HPC Center staff members directly. These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared to other HPC Centers of similar size our staff is extremely lean. Please make full use of the tools that we have provided (especially the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using our systems will be predictably good and productive.

User Manual

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.