HPCC Wiki - User contributions [en]

Main Page

2026-04-19T16:36:27Z

Alex: /* Condo */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center employs a diverse range of architectures to accommodate intricate and demanding workflows. All computational resources of various types are consolidated into a single hybrid cluster known as Arrow. This cluster comprises symmetric multiprocessor (SMP) nodes equipped with and without GPUs, distributed shared memory (NUMA) node(s), high-memory nodes, and advanced SMP nodes featuring multiple GPUs. The number of GPUs per node varies between two and eight, along with the utilized GPU interface and GPU family. Consequently, the fundamental GPU nodes are equipped with two Tesla K20m GPUs connected via the PCIe interface, while the most advanced nodes support eight Ampere A100 GPUs connected via the SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC, a not-for-profit core research facility affiliated with CUNY, is dedicated to supporting a wide range of research endeavors that necessitate advanced computational resources. Notably, CUNY-HPCC’s operations are not directly or indirectly funded by CUNY or the College of Staten Island (CSI). Consequently, CUNY-HPCC employs a cost recovery model that exclusively recoups operational expenses, without generating any profit for the HPCC. The recovered costs are meticulously calculated using comprehensive documentation of actual operational expenditures and are designed to achieve a break-even point for all CUNY users. This methodology is approved by CUNY-RF and is employed in other CUNY research facilities. The cost recovery charging schema is based on unit-hour usage, encompassing both CPU and GPU units. Definitions for these units are provided in the accompanying table.
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year.

==== Lease on condo node(s) ====
Users may lease condo nodes for a project, ensuring them 100% access and eliminating any time or job limitations on the leased resource. The minimum lease duration '''is 30 days (one month).''' Longer leases (more than 90 days) receive a 10% discount. The lease of a condo node is contingent upon the owner of the required node agreeing to lease for a specific time period and duration. Users interested in leasing a condo node of a particular type must contact the HPCC director for options. Lease fees vary depending on the type of node and are currently (note that prices are reviewed once every six months) between $230 and $1100 per month.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum Access is designed to provide extensive support for research activities across various colleges, foster collaboration between institutions, facilitate the establishment of new research projects, and serve as a testing ground for innovative studies. MAP accounts operate under a stringent fair share policy, which determines the actual waiting time for job allocation in a queue based on the resources utilized by that account in previous cycles. Furthermore, all jobs are subject to strict time constraints. Consequently, extended jobs necessitate the implementation of checkpoints.

The MAP offers three tiers of access:

· A: The Basic tier incurs a yearly fee of $5,000. It is tailored to support users from colleges with limited research activities. The fee covers infrastructure expenses associated with one to two users from these colleges.

· B: The Medium tier incurs a yearly fee of $15,000. It covers infrastructure expenses for up to twelve users from these colleges. Additionally, every account under the Medium tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation.

· C: The Advanced tier incurs a yearly fee of $25,000. It covers infrastructure expenses for all users from these colleges. Furthermore, every new account from this tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation. Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-04-19T16:35:16Z

Alex: /* Lease on condo node(s) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center employs a diverse range of architectures to accommodate intricate and demanding workflows. All computational resources of various types are consolidated into a single hybrid cluster known as Arrow. This cluster comprises symmetric multiprocessor (SMP) nodes equipped with and without GPUs, distributed shared memory (NUMA) node(s), high-memory nodes, and advanced SMP nodes featuring multiple GPUs. The number of GPUs per node varies between two and eight, along with the utilized GPU interface and GPU family. Consequently, the fundamental GPU nodes are equipped with two Tesla K20m GPUs connected via the PCIe interface, while the most advanced nodes support eight Ampere A100 GPUs connected via the SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC, a not-for-profit core research facility affiliated with CUNY, is dedicated to supporting a wide range of research endeavors that necessitate advanced computational resources. Notably, CUNY-HPCC’s operations are not directly or indirectly funded by CUNY or the College of Staten Island (CSI). Consequently, CUNY-HPCC employs a cost recovery model that exclusively recoups operational expenses, without generating any profit for the HPCC. The recovered costs are meticulously calculated using comprehensive documentation of actual operational expenditures and are designed to achieve a break-even point for all CUNY users. This methodology is approved by CUNY-RF and is employed in other CUNY research facilities. The cost recovery charging schema is based on unit-hour usage, encompassing both CPU and GPU units. Definitions for these units are provided in the accompanying table.
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

==== Lease on condo node(s) ====
Users may lease condo nodes for a project, ensuring them 100% access and eliminating any time or job limitations on the leased resource. The minimum lease duration '''is 30 days (one month).''' Longer leases (more than 90 days) receive a 10% discount. The lease of a condo node is contingent upon the owner of the required node agreeing to lease for a specific time period and duration. Users interested in leasing a condo node of a particular type must contact the HPCC director for options. Lease fees vary depending on the type of node and are currently (note that prices are reviewed once every six months) between $230 and $1100 per month.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum Access is designed to provide extensive support for research activities across various colleges, foster collaboration between institutions, facilitate the establishment of new research projects, and serve as a testing ground for innovative studies. MAP accounts operate under a stringent fair share policy, which determines the actual waiting time for job allocation in a queue based on the resources utilized by that account in previous cycles. Furthermore, all jobs are subject to strict time constraints. Consequently, extended jobs necessitate the implementation of checkpoints.

The MAP offers three tiers of access:

· A: The Basic tier incurs a yearly fee of $5,000. It is tailored to support users from colleges with limited research activities. The fee covers infrastructure expenses associated with one to two users from these colleges.

· B: The Medium tier incurs a yearly fee of $15,000. It covers infrastructure expenses for up to twelve users from these colleges. Additionally, every account under the Medium tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation.

· C: The Advanced tier incurs a yearly fee of $25,000. It covers infrastructure expenses for all users from these colleges. Furthermore, every new account from this tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation. Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-04-19T16:34:46Z

Alex: /* Condo */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center employs a diverse range of architectures to accommodate intricate and demanding workflows. All computational resources of various types are consolidated into a single hybrid cluster known as Arrow. This cluster comprises symmetric multiprocessor (SMP) nodes equipped with and without GPUs, distributed shared memory (NUMA) node(s), high-memory nodes, and advanced SMP nodes featuring multiple GPUs. The number of GPUs per node varies between two and eight, along with the utilized GPU interface and GPU family. Consequently, the fundamental GPU nodes are equipped with two Tesla K20m GPUs connected via the PCIe interface, while the most advanced nodes support eight Ampere A100 GPUs connected via the SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC, a not-for-profit core research facility affiliated with CUNY, is dedicated to supporting a wide range of research endeavors that necessitate advanced computational resources. Notably, CUNY-HPCC’s operations are not directly or indirectly funded by CUNY or the College of Staten Island (CSI). Consequently, CUNY-HPCC employs a cost recovery model that exclusively recoups operational expenses, without generating any profit for the HPCC. The recovered costs are meticulously calculated using comprehensive documentation of actual operational expenditures and are designed to achieve a break-even point for all CUNY users. This methodology is approved by CUNY-RF and is employed in other CUNY research facilities. The cost recovery charging schema is based on unit-hour usage, encompassing both CPU and GPU units. Definitions for these units are provided in the accompanying table.
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Lease on condo node(s) ===
Users may lease condo nodes for a project, ensuring them 100% access and eliminating any time or job limitations on the leased resource. The minimum lease duration '''is 30 days (one month).''' Longer leases (more than 90 days) receive a 10% discount. The lease of a condo node is contingent upon the owner of the required node agreeing to lease for a specific time period and duration. Users interested in leasing a condo node of a particular type must contact the HPCC director for options. Lease fees vary depending on the type of node and are currently (note that prices are reviewed once every six months) between $230 and $1100 per month.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum Access is designed to provide extensive support for research activities across various colleges, foster collaboration between institutions, facilitate the establishment of new research projects, and serve as a testing ground for innovative studies. MAP accounts operate under a stringent fair share policy, which determines the actual waiting time for job allocation in a queue based on the resources utilized by that account in previous cycles. Furthermore, all jobs are subject to strict time constraints. Consequently, extended jobs necessitate the implementation of checkpoints.

The MAP offers three tiers of access:

· A: The Basic tier incurs a yearly fee of $5,000. It is tailored to support users from colleges with limited research activities. The fee covers infrastructure expenses associated with one to two users from these colleges.

· B: The Medium tier incurs a yearly fee of $15,000. It covers infrastructure expenses for up to twelve users from these colleges. Additionally, every account under the Medium tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation.

· C: The Advanced tier incurs a yearly fee of $25,000. It covers infrastructure expenses for all users from these colleges. Furthermore, every new account from this tier receives complimentary 11,520 CPU hours and 1,440 GPU hours upon account creation. Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-04-15T18:52:13Z

Alex: /* Recovery of operational costs */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center employs a diverse range of architectures to accommodate intricate and demanding workflows. All computational resources of various types are consolidated into a single hybrid cluster known as Arrow. This cluster comprises symmetric multiprocessor (SMP) nodes equipped with and without GPUs, distributed shared memory (NUMA) node(s), high-memory nodes, and advanced SMP nodes featuring multiple GPUs. The number of GPUs per node varies between two and eight, along with the utilized GPU interface and GPU family. Consequently, the fundamental GPU nodes are equipped with two Tesla K20m GPUs connected via the PCIe interface, while the most advanced nodes support eight Ampere A100 GPUs connected via the SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC, a not-for-profit core research facility affiliated with CUNY, is dedicated to supporting a wide range of research endeavors that necessitate advanced computational resources. Notably, CUNY-HPCC’s operations are not directly or indirectly funded by CUNY or the College of Staten Island (CSI). Consequently, CUNY-HPCC employs a cost recovery model that exclusively recoups operational expenses, without generating any profit for the HPCC. The recovered costs are meticulously calculated using comprehensive documentation of actual operational expenditures and are designed to achieve a break-even point for all CUNY users. This methodology is approved by CUNY-RF and is employed in other CUNY research facilities. The cost recovery charging schema is based on unit-hour usage, encompassing both CPU and GPU units. Definitions for these units are provided in the accompanying table.
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-04-15T18:50:20Z

Alex: /* Computing architectures at HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center employs a diverse range of architectures to accommodate intricate and demanding workflows. All computational resources of various types are consolidated into a single hybrid cluster known as Arrow. This cluster comprises symmetric multiprocessor (SMP) nodes equipped with and without GPUs, distributed shared memory (NUMA) node(s), high-memory nodes, and advanced SMP nodes featuring multiple GPUs. The number of GPUs per node varies between two and eight, along with the utilized GPU interface and GPU family. Consequently, the fundamental GPU nodes are equipped with two Tesla K20m GPUs connected via the PCIe interface, while the most advanced nodes support eight Ampere A100 GPUs connected via the SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-04-15T18:48:40Z

Alex: /* Mission of CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) serves as a pivotal research and educational hub for the university. Situated on the campus of the College of Staten Island, located at 2800 Victory Boulevard, Staten Island, New York 10314, the center’s primary objective is to enhance educational opportunities and foster scientific research and discovery within the university. This is achieved through the management of state-of-the-art computing infrastructure and the provision of comprehensive research support services. Notably, CUNY-HPCC offers domain-specific expertise in various aspects of computationally intensive research. Furthermore, CUNY’s membership in the Empire AI (EAI) consortium positions CUNY-HPCC as a stepping stone for CUNY researchers seeking access to EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI Consortium comprises the '''CUNY Graduate Center, Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester.''' CUNY-HPCC provides support and maintains tickets for all CUNY users with allocation on EAI. Additionally, CUNY-HPCC serves as a stepping stone for CUNY researchers as it operates (on a smaller scale) architectures (including nodes with Hopper) similar to EAI, including extended “Alpha” servers and new “Beta” computers. The latter will consist of 288 B200 GPUs and recently added RTX 6000 Pro nodes. The anticipated cost for EAI is $0.50 per unit (SU), which will provide CUNY PIs with a rate that is significantly lower than a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison, the CUNY-HPCC recovery costs for public servers are $0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). For further details, please refer to the section on HPCC access plans.

== CUNY-HPCC services ==

CUNY-HPCC offers a professionally maintained, modern computational environment and architectures, along with advanced storage and fast interconnects. CUNY-HPCC serves the following purposes:

<nowiki>*</nowiki> Supports research computing at CUNY, benefiting faculty, their collaborators at other universities, and their public and private sector partners. It also supports CUNY students and research staff.

<nowiki>*</nowiki> Provides state-of-the-art computing resources and comprehensive research support services, including expertise and full support for users with allocation on EMPIRE-AI.

<nowiki>*</nowiki> Creates opportunities for the CUNY research community to establish new partnerships with the government and private sectors.

<nowiki>*</nowiki> Utilizes HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

<nowiki>*</nowiki> Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Administrative Information

2026-04-15T18:45:09Z

Alex:

__TOC__
==How to get an account==

=== Definitions and procedures ===
The CUNY-HPCC operates on a cost recovery scheme, which mandates that all accounts be linked to research projects or class accounts. Research accounts are those associated with projects sponsored by the Principal Investigator (PI) who leads the project. These accounts are associated with the project’s title, funding, and duration. Class accounts are for the duration of the class and are associated with the instructor or lecturer. No other accounts are permitted.

'''A Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project. The PI ensures compliance with regulations and oversees the project’s financial aspects. The PI is a faculty member or a qualified researcher at CUNY who has the authority to apply for research funding and manage the project.'''

The procedure to open an account is as follows:

Step 1: Creation of a sponsor account (PI account) - form A or B. At this step, the PI must create an account for themselves and provide information about the project title, funding, and duration. The request for resources is not mandatory.

Step 2: Upon creating the account, the PI will receive a unique code that must be shared with members of a group (students and postdocs) who require an account on HPCC.

Step 3: Members of the research group (lab) and academic collaborators can apply for an account at CUNY-HPCC by using form C, D, E, or F. It is mandatory to use the code mentioned in Step 2 (from the CUNY PI) in these forms.

Step 4: The PI should assign students to their project.

===Accounts overview===
All users of HPCC resources must register with HPCC to obtain an account as outlined in the table below. Users are encouraged to create an account on the HPCC web portal at:

hpchelp.csi.cuny.edu

Each user account is issued to an individual user and should not be shared. HPCC will communicate exclusively via CUNY emails to users from groups A to E. For users from groups F and G, communication will be facilitated through their verified work accounts, CCed to the CUNY collaborator (for group F only). Additionally, if resources are available and at the discretion of the CUNY-HPCC director, external researchers can obtain an external research account (type G) at CUNY-HPCC by renting HPC resources and paying the full cost recovery fee in advance. Please contact the HPCC director for further details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail.
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY E-mail, For '''PhD students the first E-mail is their GC E-mail address. Second mail is the college E-mail.'''
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address. '''These accounts are for class only''' '''Undergraduate students doing research work must have faculty sponsor and be registered in a project led by legitimate PI.'''
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator E-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation, every research user account is provided with a home directory of 50 GB (with a maximum of 10,000 files on /global/u) mounted as /global/u/<userid>. If necessary, a user may request an increase in the size of their home directory. The HPC Center will endeavor to accommodate reasonable requests. If you anticipate having more than 10,000 files, please compress several small files into a single larger zip file. Please ensure that only organized and relevant information is stored in your space to optimize the utilization of the existing storage.

Student class accounts (group d) are provided with a home directory of 10 GB. Please note that class accounts and data will be deleted 30 days after the semester concludes, unless otherwise agreed upon. Students are responsible for backing up their own data prior to the end of the semester.

Upon account establishment, only the user has read/write access to their files. The user can modify their UNIX permissions to grant others in their group read/write access to their files.

Kindly inform the HPC Center if user accounts require removal or addition to a specific group. Please refer to the policies outlined below for account management. It is important to note that accounts are not perpetual and will be removed if they are not accessed or active (as detailed below).

=== User accounts policies ===
The CUNY HPCC implements stringent security measures for user account management. The institution employs a “account period” system. Account periods vary depending on the account type: one year for accounts in A, C, E, and F categories, and one semester for accounts in B and D categories. All accounts undergo periodic reviews, and inactive accounts are promptly removed. All student accounts automatically expire and are deleted after each semester, unless an advisor requests an extension.

User accounts for groups A, C, E, and F must be renewed annually by September 30th. User accounts in groups B and D must be renewed within two weeks after each semester. Accounts that remain inactive for one account period or are not renewed are automatically disabled or locked and will be deleted 60 days after the lockout. The deletion of a specific account signifies the permanent removal of all data associated with that account.

'''Reset Password'''

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user wishes to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu. Supervisors who wish to modify the access of their researchers and/or students should contact the HPC to remove, add, or modify access. User accounts that are not accessed or renewed for more than one year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked, and users must contact HPCC to recover access.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Administrative Information

2026-04-15T17:35:52Z

Alex: /* Definitions and procedures */

__TOC__
==How to get an account==

=== Definitions and procedures ===
The CUNY-HPCC operates on a cost recovery scheme, which mandates that all accounts be linked to research projects or class accounts. Research accounts are those associated with projects sponsored by the Principal Investigator (PI) who leads the project. These accounts are associated with the project’s title, funding, and duration. Class accounts are for the duration of the class and are associated with the instructor or lecturer. No other accounts are permitted.

'''A Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project. The PI ensures compliance with regulations and oversees the project’s financial aspects. The PI is a faculty member or a qualified researcher at CUNY who has the authority to apply for research funding and manage the project.'''

The procedure to open an account is as follows:

Step 1: Creation of a sponsor account (PI account) - form A or B. At this step, the PI must create an account for themselves and provide information about the project title, funding, and duration. The request for resources is not mandatory.

Step 2: Upon creating the account, the PI will receive a unique code that must be shared with members of a group (students and postdocs) who require an account on HPCC.

Step 3: Members of the research group (lab) and academic collaborators can apply for an account at CUNY-HPCC by using form C, D, E, or F. It is mandatory to use the code mentioned in Step 2 (from the CUNY PI) in these forms.

Step 4: The PI should assign students to their project.

===Accounts overview===
All users of HPCC resources must register with HPCC to obtain an account as outlined in the table below. Users are encouraged to create an account on the HPCC web portal at:

hpchelp.csi.cuny.edu

Each user account is issued to an individual user and should not be shared. HPCC will communicate exclusively via CUNY emails to users from groups A to E. For users from groups F and G, communication will be facilitated through their verified work accounts, CCed to the CUNY collaborator (for group F only). Additionally, if resources are available and at the discretion of the CUNY-HPCC director, external researchers can obtain an external research account (type G) at CUNY-HPCC by renting HPC resources and paying the full cost recovery fee in advance. Please contact the HPCC director for further details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail.
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY E-mail, For '''PhD students the first E-mail is their GC E-mail address. Second mail is the college E-mail.'''
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address. '''These accounts are for class only''' '''Undergraduate students doing research work must have faculty sponsor and be registered in a project led by legitimate PI.'''
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator E-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''A,C,E and F''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''A,C,E and F''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Administrative Information

2026-04-15T17:31:34Z

Alex: /* Definitions and procedures */

__TOC__
==How to get an account==

=== Definitions and procedures ===
The CUNY-HPCC operates on a cost recovery scheme, which mandates that all accounts be linked to research projects or class accounts. Research accounts are those associated with projects sponsored by the Principal Investigator (PI) who leads the project. These accounts are associated with the project’s title, funding, and duration. Class accounts are for the duration of the class and are associated with the instructor or lecturer. No other accounts are permitted.

'''A Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project. The PI ensures compliance with regulations and oversees the project’s financial aspects. The PI is a faculty member or a qualified researcher at CUNY who has the authority to apply for research funding and manage the project.'''

The procedure to open an account is as follows:

Step 1: Creation of a sponsor account (PI account) - form A or B. At this step, the PI must create an account for themselves and provide information about the project title, funding, and duration. The request for resources is not mandatory.

Step 2: Upon creating the account, the PI will receive a unique code that must be shared with members of a group (students and postdocs) who require an account on HPCC.

Step 3: Members of the research group (lab) and academic collaborators can apply for an account at CUNY-HPCC by using form C, D, E, or F. It is mandatory to use the code mentioned in Step 2 (from the CUNY PI) in these forms.

Step 4: The PI should assign students to their project.

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. To do so the users are encouraged to create an account on HPCC web portal:

hpchelp.csi.cuny.edu.

A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from '''F,''' and '''G''' account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail.
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY E-mail, For '''PhD students the first E-mail is their GC E-mail address. Second mail is the college E-mail.'''
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address. '''These accounts are for class only''' '''Undergraduate students doing research work must have faculty sponsor and be registered in a project led by legitimate PI.'''
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator E-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''A,C,E and F''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''A,C,E and F''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Administrative Information

2026-04-15T17:21:58Z

Alex: /* Definitions and procedures */

__TOC__
==How to get an account==

=== Definitions and procedures ===
CUNY-HPCC operates on cost recovery scheme. That '''demands all accounts to be associated with research project(s) or to be class accounts.''' The accounts associated with project are called '''research accounts and are are sponsored by Principle Investigator (PI) that leads that project(s).''' The class accounts are for the duration of the class and are associated with the teacher/lecturer of the class. No individual accounts of any other type are possible. A '''Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project, ensuring compliance with regulations and overseeing the project's financial aspects. PI is a faculty member or a qualified researcher who has the authority to apply for research funding and manage the project.''' The procedure to open an account is as follows:

'''Step 1.''' Creation of sponsor account (PI account) - '''form A or B below.''' At that step the PI must create account for him/her and provide information about project title, funding and duration. The request of resources is not mandatory.

'''Step 2.''' Upon creating account the PI will get unique code which has to be shared with members of a group (students and post docs) that require account on HPCC.

'''Step 3.''' Members of research group (lab) and academic collaborators can apply for account at CUNY-HPCC by using form C,D, E or F It is mandatory to use a code mentioned in Step 2 (from CUNY PI) in these forms.

'''Step 4.''' PI should assign students to his project.

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. To do so the users are encouraged to create an account on HPCC web portal:

hpchelp.csi.cuny.edu.

A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from '''F,''' and '''G''' account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail.
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY E-mail, For '''PhD students the first E-mail is their GC E-mail address. Second mail is the college E-mail.'''
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address. '''These accounts are for class only''' '''Undergraduate students doing research work must have faculty sponsor and be registered in a project led by legitimate PI.'''
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator E-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''A,C,E and F''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''A,C,E and F''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Training & Workshops

2026-03-23T19:31:53Z

Alex:

The CUNY HPCC provides training course and organizes seminars on various HPC topics. The training courses are provided at no cost and may be held at any CUNY campus site, at the CUNY HPCC at College of Staten Island, or at the Graduate Center. The training course at the Graduate Center and its '''online''' version is course on parallel programming and use of HPC architectures. The on-site course takes place if enough students express interest. It covers various topics from basic SLURM scripting to basic GPU programming to intermediate parallel programming with use of MPI and OpenACC. Please note that lectures cover each topic systematically so particular topic may be discussed in several lectures. Users who do want to attend the course should send an e-mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] and ask for registration. All participants will get student account on CUNY-HPCC servers unless they already have one.

In addition HPCC provides in person and zoom consultations with individuals or small groups of users every Wednesday - 11AM to 3 PM. The interested users should register by sending e-mail to alex.tzanov@csi.cuny.edu by Mon on the same week. These consultations should help new users and those with no experience to start quickly with HPCC resources. At that time users may discuss their particular problems and get guidance in development of their own parallel scientific code(s). Please send a mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] or to [[mailto:alexander.tzanov@.csi.cuny.edu]] for available time slots not later than 3PM on Mon. HPCC will make all efforts to accommodate all users so any time slot may be shared by several users.

For any additional information, please send an email to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]]. Every Wednesday between 11 AM and 2 PM, HPCC conducts remote help session/consultation. Please sent request for invite to hpchelp@csi.cuny.edu. '''''Note that the consultation is open only for CUNY faculty, students and staff.''''' 

<h2>Schedule </h2>
 
CUNY High-performance Computing Center (HPCC) provides Help Desk/Consultation support and lectures at the Graduate Center on programming and using Unix-based HPC cluster systems. 
Dr. Alex Tzanov will conduct the lectures and consulattions. Please see schedule below. 

<table border="1" cellspacing="0" cellpadding="0" width="621">
<tr>
<td width="72" rowspan="2" valign="top">Date 
 
 </td>
<td width="48" rowspan="2" valign="top">Day</td>
<td width="303" valign="top">Lecture</td>
<td width="188" valign="top">Consultation</td>
</tr>
<tr>
<td width="303" valign="top">Room 4434 10 AM - 12 AM</td>
<td width="188" valign="top">Room 4411, GC</td>
</tr>
<tr>
<td width="72" valign="top"> </td>
<td width="48" valign="top"> </td>
<td width="303" valign="top"></td>
<td width="188" valign="top"></td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Introduction to HPC and HPCC</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Introduction to parallel programming </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed parallel programming with MPI part 1</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 2</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 3</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 4</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 5</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top"> Distributes Parallel programming with MPI – Hands on </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU programming part 1.</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU programming part 2</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU - hands on </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Easy GPU programming with OpenACC part 1. </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Easy GPU programming with OpenACC part 2 </td>
<td width="188" valign="top">ONLINE</td>
</tr>
</table>
<div align="center"></div>

Apart from that the HPCC center provides a short introductory course at Graduate center for new users. The course covers HPCC structure
and workflow, HPC servers information, basic SLURM scripting, basic Linux and Unix commands, how to compile and run the program on HPCC
servers and basics of data storage and management system. For more information please contact hpchelp@csi.cuny.edu.

Training & Workshops

2026-03-23T19:18:43Z

Alex:

The CUNY HPCC provides training course and organizes seminars on various HPC topics. The training courses are provided at no cost and may be held at any CUNY campus site, at the CUNY HPCC at College of Staten Island, or at the Graduate Center. The training course at the Graduate Center and its '''online''' version is course on parallel programming and use of HPC architectures. The on-site course takes place if enough students express interest. It covers various topics from basic SLURM scripting to basic GPU programming to intermediate parallel programming with use of MPI and OpenACC. Please note that lectures cover each topic systematically so particular topic may be discussed in several lectures. Users who do want to attend the course should send an e-mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] and ask for registration. All participants will get student account on CUNY-HPCC servers unless they already have one.

In addition HPCC provides in person and zoom consultations with individuals or small groups of users every Wednesday - 11AM to 3 PM. The interested users should register by sending e-mail to alex.tzanov@csi.cuny.edu by Mon on the same week. These consultations should help new users and those with no experience to start quickly with HPCC resources. At that time users may discuss their particular problems and get guidance in development of their own parallel scientific code(s). Please send a mail to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]] or to [[mailto:alexander.tzanov@.csi.cuny.edu]] for available time slots not later than 3PM on Mon. HPCC will make all efforts to accommodate all users so any time slot may be shared by several users.

For any additional information, please send an email to [[mailto:hpchelp@.csi.cuny.edu hpchelp@csi.cuny.edu]].
 

<h2>Schedule </h2>
 
CUNY High-performance Computing Center (HPCC) provides Help Desk/Consultation support and lectures at the Graduate Center on programming and using Unix-based HPC cluster systems. 
Dr. Alex Tzanov will conduct the lectures and consulattions. Please see schedule below. 

<table border="1" cellspacing="0" cellpadding="0" width="621">
<tr>
<td width="72" rowspan="2" valign="top">Date 
 
 </td>
<td width="48" rowspan="2" valign="top">Day</td>
<td width="303" valign="top">Lecture</td>
<td width="188" valign="top">Consultation</td>
</tr>
<tr>
<td width="303" valign="top">Room 4434 10 AM - 12 AM</td>
<td width="188" valign="top">Room 4411, GC</td>
</tr>
<tr>
<td width="72" valign="top"> </td>
<td width="48" valign="top"> </td>
<td width="303" valign="top"></td>
<td width="188" valign="top"></td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Introduction to HPC and HPCC</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Introduction to parallel programming </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed parallel programming with MPI part 1</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 2</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 3</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 4</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Distributed Parallel programming with MPI part 5</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top"> Distributes Parallel programming with MPI – Hands on </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU programming part 1.</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU programming part 2</td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">GPGPU - hands on </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Easy GPU programming with OpenACC part 1. </td>
<td width="188" valign="top">1 PM - 5 PM</td>
</tr>
<tr>
<td width="72" valign="top"></td>
<td width="48" valign="top">WED</td>
<td width="303" valign="top">Easy GPU programming with OpenACC part 2 </td>
<td width="188" valign="top">ONLINE</td>
</tr>
</table>
<div align="center"></div>

Apart from that the HPCC center provides a short introductory course at Graduate center for new users. The course covers HPCC structure
and workflow, HPC servers information, basic SLURM scripting, basic Linux and Unix commands, how to compile and run the program on HPCC
servers and basics of data storage and management system. For more information please contact hpchelp@csi.cuny.edu.

Main Page

2026-03-14T15:27:23Z

Alex: /* Recovery of operational costs */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans details and examples ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:26:24Z

Alex: /* Minimal Access Plan (MAP) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

==== Minimal Access Plan (MAP) ====
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:26:01Z

Alex: /* Minimal Access Plan (MAP) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

=== Minimal Access Plan (MAP) ===
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:25:33Z

Alex: /* Minimal Access Plan (MAP) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

=== Minimal Access Plan (MAP) ===

==== Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows: ====
- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

==== Compute on demand (CODP) ====
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

==== Lease a public node(s) ====
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:24:50Z

Alex: /* Compute on public resources */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== Compute on public resources ===
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

=== Minimal Access Plan (MAP) ===
Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:20:09Z

Alex:

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==Computing architectures at HPCC==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:16:57Z

Alex: /* Hours of Operation */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

= Hours of Operation =
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

= User Support =
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

= Warnings and modes of operation =

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

= User Manual =
The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:15:32Z

Alex: /* Partitions and jobs */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

= Partitions and jobs =
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:13:48Z

Alex: /* Support for research grants */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

= Support for research grants =
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:13:13Z

Alex: /* Recovery of operational costs */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

= Recovery of operational costs =

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:12:33Z

Alex: /* Organization of systems and data storage (architecture) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

== Organization of systems and data storage (architecture) ==
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:12:04Z

Alex: /* Organization of systems and data storage (architecture) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

= Organization of systems and data storage (architecture) =
All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T15:04:10Z

Alex: /* Lease a public node(s) */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
Users may lease a node(s) for project. That ensures them 100% access and no time or job limitations over leased resource. The minimum lease time is 30 days (one month). Longe leases (more than 90 days) have 10% discount. '''MAP users''' are charged between $172 and $950 per month (see below), depending on the type of node. '''Non-MAP''' users are charged between $249.58 and $1,399 per month, depending on the type of node. Please see below for details and examples.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see table below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users example
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T14:58:15Z

Alex: /* Compute public resources */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==

=== Compute on public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
MAP users are charged between $172 and $950 per month (see below), depending on the type of node. Non-MAP users are charged between $249.58 and $1,399 per month, depending on the type of node.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-14T14:57:45Z

Alex: /* Recovery of operational costs */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==

=== Compute public resources ===
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}
The cost recovery model for public (non-condominium) servers offers the following options:

# Minimal Access Plan (MAP)
# Compute on Demand (CODP)
# Lease a node(s)

Colleges may participate in any of the Minimal Access Plan tiers. The tiered pricing structure is as follows:

- Tier A: $5,000 per year

- Tier B: $10,000 per year

- Tier C: $25,000 per year

Within each tier, the cost is '''$0.015 per CPU hour and $0.09 per GPU hour.''' It is important to note that the Minimum Access Plan (MAP) tiers A, B, and C are '''not all-inclusive.''' Therefore, even if a college pays for a higher tier, it does not guarantee unlimited use of all college employees, faculty, and students throughout the year.

# The MAP plan is indirectly linked to the number of hours used. Consequently, the definition of “up to 12 users for B tier” should not be interpreted as “all users up to 12 per college receive unlimited access to HPCC resources.”
# The number of users per tier is determined by statistical analysis of resource usage and statistics for the average duration of a job across all CUNY institutions. This means that if the number of users from a college exceeds the number of hours encoded in the MAP fee, the additional hours will be charged at the preferred rate of $0.015 per CPU hour and $0.09 per GPU thread hour.
# Furthermore, the MAP fee may fully cover the expenses for individuals from a given college for a year, but it may also not. This depends on the actual usage and type of resources, as well as the number of additional individuals from the same college who appear during the year. It is crucial to understand that allocation on HPCC is not restricted; our focus is solely on the completion of tasks.
# The free 11,540 CPU hours and 1,440 GPU hours are allocated per PI account and project and are available only for MAP-B and MAP-C (see below). These free hours are intended to facilitate project establishment. Consequently, the PI can request that a member or members of a group utilize time and explore new research opportunities. For instance, upon creating his account the PI X will receive free hours. If he/she/whatever hires a graduate student(s), they may share these free hours if they work on the same project.
# It is important to note that each GPU requires at least four CPU cores to operate. Therefore, if a user requests one GPU thread, it equates to one GPU thread and four CPU threads, which is equivalent to $0.15 per unit hour for that unit (units are explained above). Note that not all GPU support virtualization, so unit may include the whole GPU depend on used GPU type.

=== Compute on demand (CODP) ===
Users from colleges that do not participate in MAP (A,B,C) are charged '''$0.018 per CPU hour and $0.11 per GPU hour.''' There is no free time associated with CODP.

=== Lease a public node(s) ===
MAP users are charged between $172 and $950 per month (see below), depending on the type of node. Non-MAP users are charged between $249.58 and $1,399 per month, depending on the type of node.

=== Condo resources ===
Condo users only pay for infrastructure support. The annual fee depends on the type of node and ranges from $1,540 to $4,520 per year. Please see below for details.

=== Storage ===
Storage costs are $60 per TB per year, backup costs are $45 per TB per year, and archive costs are $35 per TB per year. The first 50 GB of scratch storage are free. Prices are calculated at the end of each month.

# Additionally, note that '''file transfers from/to HPCC are free''', but it is important to consider the CUNY network speed, which is significantly slower than modern standards. This will impact the time required to download large data sets. For large data HPCC utilizes and recommend to use secure parallel download via Globus.
# All services associated with data and storage provided by HPCC are free for CUNY users.

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing public node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Fees for lease node(s) for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and monthly lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Data Storage and Management System

2026-03-13T00:46:13Z

Alex: /* /scratch */

This file system resides on '''Hybrid Parallel File System (HPFS).'''

__TOC__
=="Home" directories are on /global/u==
'''/global/u''' is partition in parallel high performance Linux file system based on HPE parallel file system (HPFS). It holds the home directories of '''all''' individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<userid>''' and a 100 GB allocation of disk space for home directories on '''/global/u/<userid>'''. These '''home''' directories are on global file system mounted only on login nodes. There is no local storage on nodes . That means that 1. data can be accesses only from PFFS and login node(s) 2. no local storage on nodes is available. Codes that write intermediate results on disk are typically slow and should be run only on condo nodes that have local disk storage. All home directories are backed up on weekly basis.

==/scratch==
/Scratch is fast file system from which all jobs start. There is no quota on scratch, but '''scratch''' files are temporary and are '''not backed up'''. Single user cannot however take whole space with his/her data. That means users can run data sets that exceed their home space but they cannot use /scratch for storage. It is important to understand that '''/scratch''' must be used only for submitting jobs. Output from jobs may ONLY temporarily be stored on '''/scratch (up to 10 days). Consequently in''' order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''/scratch''' from '''/global/u''' using UNIX commands and/or from '''SR1''' using '''iRODS''' commands. The later is mounting data directly from project space. Note that SR is slower file system so large files is better to be staged from /home directories. Files in '''/scratch''' are '''automatically relocated to local archive''' when their inactive residence on scratch exceeds 90 days. Local archive has limited capacity and serves as data buffer, so '''strict policies of cleaning up temporary archive are in place'''. Upon relocation the user will get a warning ( via e-mail) and must either move files to his/her home directory or to SR1. Note that files left in temp archive will be purged after 30 days. Users must not '''store valuable data or compiled codes on scratch''' since these are static type of files.

==“Project” directories==

'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands. In iRODS terminology, a “collection” is the equivalent of “directory”.

A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files. A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist. The PI is the individual responsible to the University or a granting agency for the “Project”. The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”. Project data are stored on project space on main file system. Valuable parts of the projects must be curated by PI's and stored in HPCC long term archive.

 

==Typical Workflow==
Typical workflows for Penzias Appel and Karle in are described below:

1. Copying files from a user’s home directory or from '''SR1''' to '''SCRATCH'''. 
If working with '''HOME''': 

cd /scratch/<userid>
mkdir mySLURM_Job && cd mySLURM_Job
cp /global/u/<userid>/myProject/a.out ./
cp /global/u/<userid>/myProject/mydatafile ./


If working with '''SR1 (storage repository)''': 

cd /scratch/<userid>
mkdir mySLURM_Job && cd mySLURM_Job
iget myProject/a.out
iget myProject/mydatafile


2. Prepare SLURM job script. Typical SLURM sript is similar to the following: 

#!/bin/bash
#SBATCH --partition production
#SBATCH -J test
#SBATCH --nodes 1
#SBATCH --ntasks 8
#SBATCH --mem 4000
echo "Starting…"

cd $SLURM_SUBMIT_DIR
mpirun -np 4 ./a.out ./mydatafile > myoutputs
echo "Done…"

Your SLURM may be different depending on your needs. Read section Submitting Jobs for a reference. 

3. Run the job 

sbatch ./mySLURM_script

4. Once job is finished, clean up '''SCRATCH''' and store outputs in your user home directory or in '''SR1'''. 

If working with '''HOME''': 

mv ./myoutputs /global/u/<userid>/myProject/.
cd ../
rm -rf mySLURM_Job 
== iRODS (The iRODS Section is in REVIEW and may not be CURRENT) ==

'''iRODS''' is the integrated Rule-Oriented Data-management System, a
community-driven, open source, data grid software solution. '''iRODS''' is
designed to abstract data services from data storage hardware and
provide users with hardware-agnostic way to manipulate data.

'''iRODS''' is a primary tool that is used by the CUNY HPCC users to
seamlessly access 1PB storage resource (further referenced as '''SR1'''
here) from any of the HPCC's computational systems.

Access to '''SR1''' is provided via so-called '''i-commands''':

<pre>iinit
ils
imv</pre>

Comprehesive list of i-commands with detailed description can be
obtained at [https://wiki.irods.org/index.php/icommands iRODS wiki].

To obtain quick help on any of the commands while being logged into
any of the HPCC's machines type '''i-command -h'''. For example:
<pre>
ils -h
</pre>

Following is the list of some of the most relevant '''i-commands''':

:'''iinit''' -- Initialize session and store your password in a scrambled form for automatic use by other icommands.
:'''iput''' -- Store a file
:'''iget''' -- Get a file
:'''imkdir''' -- Like mkdir, make an iRODS collection (similar to a directory or Windows folder)
:'''ichmod''' -- Like chmod, allow (or later restrict) access to your data objects by other users.
:'''icp''' -- Like cp or rcp, copy an iRODS data object
:'''irm''' -- Like rm, remove an iRODS data object
:'''ils''' -- Like ls, list iRODS data objects (files) and collections (directories)
:'''ipwd''' -- Like pwd, print the iRODS current working directory
:'''icd''' -- Like cd, change the iRODS current working directory
:'''ichksum''' -- Checksum one or more data-object or collection from iRODS space.
:'''imv''' -- Moves/renames an irods data-object or collection.
:'''irmtrash''' -- Remove one or more data-object or collection from a RODS trash bin.
:'''imeta''' -- Add, remove, list, or query user-defined Attribute-Value-Unit triplets metadata
:'''iquest''' -- Query (pose a question to) the ICAT, via a SQL-like interface

Before using any of the i-commands users need to identify themselves to the iRODS server running command
<pre>
# iinit
</pre>
and providing HPCC's password.

Typical workflow that involves operations on files stored in SR1
include storing/getting data to and from SR1, tagging data with
metadata, searching for data, sharing (setting permissions).

==== Storing data to SR1 ====

1. Create '''iRODS''' directory (aka 'collection'): 

# imkdir myProject

2. Store all files ''''myfile*'''' into this directory (collection): 

# iput -r <userid> myfile* myProject/.

3. Verify that files are stored: 

# ils
/cunyZone/home/<userid>:
C- /cunyZone/home/<userid>/myProject
# ils myProject
/cunyZone/home/<userid>/myProject:
myfile1
myfile2
myfile3


Symbol ''''C-'''' in the beginning of output of ''''ils'''' shows that listed item is a collection.

4. Combining ''''ils', 'imkdir', 'iput', 'icp', 'ipwd', 'imv'''' user can create iRODS directories and store files in them similarly to what is normally done with UNIX commands ''''ls', 'mkdir', 'cp', 'pwd', 'mv'''' etc...

==== Getting data from SR1 ====

1. To copy file from SR1 to current working directory run
# iget myProject/myfile1

2. Now listing current working directory should reveal '''myfile1''':
# ls
myfile1

3. Instead of individual files the whole directory (with
sub-directories) can be copied with ''''-r'''' flag (stands for
'recursive')
# iget -r myProject

NOTE: wildcards are not supported, therefore the command below will not work
# iget myProject/myfile*

=== Tagging data with metadata ===

iRODS provides users with extremely powerful mechanism of managing
data with metadata. While working with large datasets it's
sometimes easy to forget what is stored in this or the other file.
Metadata tags help organizing data in a very easy and reliable
manner.

Let's tag files from previous example with some metadata:
<pre>
# imeta add -d myProject/myfile1 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 comment "This is file number 1"
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 zvalue 10 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 comment "This is file number 2"
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 colorLabel BLUE
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 comment "This is file number 3"
AVU added to 1 data-objects
</pre>
Here we've tagged myfile1 with 3 metadata labels:

- zvalue 10 meters

- colorlabel RED

- comment "This is file number 1"

Similar tags were added to 'myfile2' and 'myfile3'

Metadata come in form of AVU -- Attribute|Value|Unit. As seen from
the above examples Unit is not necessary.

Let's list all metadata assigned to file 'myfie1':
<pre>
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: zvalue
value: 15
units: meters
----
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
</pre>

To remove an AVU assigned to a file run:
<pre>
# imeta rm -d myProject/myfile1 zvalue 15 meters
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
#
#
# imeta add -d myProject/myfile1 zvalue 15 meters
</pre>

Metadata may be assigned to directories as well:
<pre>
# imeta add -C myProject simulationsPool 1
# imeta ls -C myProject
AVUs defined for collection myProject:
attribute: simulationsPool
value: 1
units:
</pre>

Note the '-C' key that is used instead of '-d'.

=== Searching for data ===

Power of metadata becomes obvious when data needs to be found in
large collections. Here is an illustration of how easy this task is
done with iRODS via imeta queries:

# imeta qu -d zvalue = 15
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile3

We see both files that were tagged with label 'zvalue 10 meters'.
Here is different query:

# imeta qu -d colorLabel = RED
collection: /cunyZone/home/<userid></myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile2

Another powerful mechanism to query data is provided with 'iquest'.
Following is a number of examples that show 'iquest' capabilities:

iquest "SELECT DATA_NAME, DATA_SIZE WHERE DATA_RESC_NAME like 'cuny%'"
iquest "For %-12.12s size is %s" "SELECT DATA_NAME , DATA_SIZE WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest "SELECT COLL_NAME WHERE COLL_NAME like '/cunyZone/home/%' AND USER_NAME like '<userid>'"
iquest "User %-6.6s has %-5.5s access to file %s" "SELECT USER_NAME, DATA_ACCESS_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest " %-5.5s access has been given to user %-6.6s for the file %s" "SELECT DATA_ACCESS_NAME, USER_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>>'"
iquest no-distinct "select META_DATA_ATTR_NAME"
iquest "select COLL_NAME, DATA_NAME WHERE DATA_NAME like 'myfile%'"
iquest "User %-9.9s uses %14.14s bytes in %8.8s files in '%s'" "SELECT USER_NAME, sum(DATA_SIZE),count(DATA_NAME),RESC_NAME"
iquest "select sum(DATA_SIZE) where COLL_NAME = '/cunyZone/home/<userid>'"
iquest "select sum(DATA_SIZE) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select sum(DATA_SIZE), RESC_NAME where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select order_desc(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select count(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select RESC_NAME where RESC_CLASS_NAME IN ('bundle','archive')"
iquest "select DATA_NAME,DATA_SIZE where DATA_SIZE BETWEEN '100000' '100200'"

=== Sharing data ===

Access to the data can be controlled via 'ichmod' command. It's
behavior is similar to UNIX 'chmod' command. For example if there is a
need to provide user '''<userid>''' with read access to file
'''myProject/myfile1''' execute the following command:
ichmod read <userid1> myProject/myfile1

To see who has access to a file/directory use:
# ils -A myProject/myfile1
/cunyZone/home/<userid>/myProject/myfile1
ACL - <userid1>
#cunyZone:read object <userid>#cunyZone:own

In the above example user '''<userid1>''' has read access to the file and
user '''<userid>''' is an owner of the file.

Possible levels of access to a data object are null/read/write/own.
==Backups (IN REVIEW)==

'''Backups.''' '''/global/u''' user directories. Project files are backed up automatically to a remote tape silo system over a fiber optic network. Backups are performed daily.

If the user deletes a file from '''/global/u''', it will remain on the tape silo system for 30 days, after which it will be deleted and cannot be recovered. If a user, within the 30 day window finds it necessary to recover a file, the user must expeditiously submit a request to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu].

Less frequently accessed files are automatically transferred to the HPC Center robotic tape system, freeing up space in the disk storage pool and making it available for more actively used files. The selection criteria for the migration are age and size of a file. If a file is not accessed for 90 days, it may be moved to a tape in the tape library – in fact to two tapes, for backup. This is fully transparent to the user. When a file is needed, the system will copy the file back to the appropriate disk directory. No user action is required.

==Data retention and account expiration policy (IN REVIEW)==

Project directories on SR1 are retained as long as the project is active. The HPC Center will coordinate with the Principal Investigator of the project before deleting a project directory. If the PI is no longer with CUNY, the HPC Center will coordinate with the PI’s departmental chair or Research Dean, whichever is appropriate.

For user accounts, current user directories under /global/u are retained as long as the account is active. If a user account is inactive for one year, the HPC Center will attempt to contact the user and request that the data be removed from the system. If there is no response from the user within three months of the initial notice, or if the user cannot be reached, the user directory will be purged.

==DSMS Technical Summary (IN REVIEW)==
{| class="wikitable"
|+
!File Space
!Purpose
!Accessibility
!Quota
!Backups
!Purges
|-
|Scratch:
/scratch/<userid>
on *PENZIAS, ANDY, SALK, BOB*
|High Performance Parallel scratch filesystems. Work area for jobs, datasets, restart files, files to be pre-/post processed. Temporary space for data that will be removed within a short amount of time.
|Not globally accessible.
Separate /scratch/<userid> exists on each system. Visible on login and compute nodes of each system and on the data transfer nodes.
|None
|None
|Files older than 2 weeks are automatically deleted
OR
when scratch filesystem reaches 70% utilization
|-
|Home:
/global/u/<userid>
|User home filespace. Essential data should be stored here, such as user's source code, documents, and data structures.
|Globally accessible on the login and on the data transfer nodes through native GPFS or NFS mounts
|Nominally 50 GB
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days.
| rowspan="2" |Not purged
|-
|Project:
/SR1/<PID>
|Project space allocations
|Accessible on the login and on the data transfer nodes. Accessible outside CUNY HPC Center through iRODS.
|Allocated according to project needs
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days and retrievable on request, but the iRODS metadata may be lost.
|}
• '''SR1''' is tuned for high bandwidth, redundancy, and resilience. It is not optimal for handling large quantities of small files. If you need to archive more than a thousand of files on '''SR1''', please create a single archive using '''tar'''.

• A separate '''/scratch/<userid>''' exists on each system. On PENZIAS, SALK, KARLE, and ANDY, this is a Lustre parallel file system, on HERBERT it is NFS. These '''/scratch''' directories are visible on the login and compute nodes of the system only and on the data transfer nodes, but are not shared across HPC systems.

• '''/scratch/<userid>''' is used as a high performance parallel scratch filesystem, for example, temporary files (e.g. restart files) should be stored here.

• There are no quotas on '''/scratch/<userid>''', however any files older than 2 weeks are automatically deleted. Also, a cleanup script is scheduled to run every two weeks or whenever the /scratch disk space utilization exceeds 70%. Dot-files are generally left intact from these cleanup jobs.

• /scratch space is available to all users. If the scratch space is exhausted, jobs will not be able to run. Purge any files in '''/scratch/<userid>''', which are no longer needed, even before the automatic deletion kicks in.

• '''/scratch/<userid>''' directory may be empty when you login, you will need to copy any files required for submitting your jobs (submission scripts, data sets) from '''/global/u''' or from '''SR1'''. Once your jobs complete copy any files you need to keep back to '''/global/u''' or '''SR1''' and remove all files from /scratch.

• Do not use '''/tmp''' for storing temporary files. The file system where '''/tmp''' resides in memory is very small and slow. Files will be regularly deleted by automatic procedures.

• '''/scratch/<userid>''' is not backed up and there is no provision for retaining data stored in these directories.
 

==Data Handling Practices ==
===HPFS, i.e., /global/u ===

• The HPFS is not an archive for non-HPC users. It is an archive for users who are processing data at the HPC Center. “Parking” files on the '''HPFS''' as a back-up to local data stores is prohibited.

• Do not store more than 1,000 files in a single directory. Store collections of small files into an archive (for example, tar). Note that for every file, a stub of about 4MB is kept on disk even if the rest of the file is migrated to tape, meaning that even migrated files take up some disk space. It also means that files smaller than the stub size are never migrated to tape because that would not make sense. Storing a large number of small files in a single directory degrades the file system performance.
 

===/scratch===

• Please regularly remove unwanted files and directories and avoid keeping duplicate copies in multiple locations. File transfer among the HPC Center systems is very fast. It is forbidden to use "touch jobs" to prevent the cleaning policy from automatically deleting your files from the '''/scratch''' directories. Use '''tar -xmvf''', not '''tar -xvf''' to unpack files. '''tar -xmvf''' updates the times stamp on the unpacked files. The '''tar -xvf''' command preserves the time stamp from the original file and not the time when the archive was unpacked. Consequently, the automatic deletion mechanism may remove files unpacked by '''tar –xvf''', which are only a few days old.

Data Storage and Management System

2026-03-13T00:25:45Z

Alex: /* "Home" directories are on /global/u */

This file system is resides on Hybrid Parallel File System (HPFS).

__TOC__
=="Home" directories are on /global/u==
'''/global/u''' is a standard parallel high performance Linux file system based on HPE parallel file system (PFFS) It holds the home directories of individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<userid>''' and a 100 GB allocation of disk space for home directories on '''/global/u/<userid>'''. These '''home''' directories are on global file system not on local disks at nodes. That means that 1. data can be accesses only from PFFS and 2. no local storage on nodes is available. The exception are few nodes in CONDO tier that are designed to have local "buffer" space. All home directories are backed up on weekly basis.

==/scratch==
/Scratch is fast file system from which all jobs start. There is no quota on scratch, but '''scratch''' files are temporary and are '''not backed up'''. '''/scratch''' is used by jobs queued for or in execution. Output from jobs may ONLY temporarily be located in '''/scratch (up to 30 days).'''

In order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''/scratch''' from '''/global/u''' using UNIX commands and/or from '''SR1''' using '''iRODS''' commands. The later is moaning data directly from project space.

Files in '''/scratch''' on a system are '''automatically relocated to local archive''' when file(s)' residence on scratch exceeds 90 days. Active file are not relocated. Thus user must not store valuable data or compiled codes on scratch since the latter are static type of files. Upon move the user get warning from HPCC and must either move files to his/her home directory. Files in temporary archive left for 45 days are removed.

==“Project” directories==

'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands. In iRODS terminology, a “collection” is the equivalent of “directory”.

A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files. A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist. The PI is the individual responsible to the University or a granting agency for the “Project”. The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”. Project data are stored on project space on main file system. Valuable parts of the projects must be curated by PI's and stored in HPCC archive.

 

==Typical Workflow==
Typical workflows for Penzias Appel and Karle in are described below:

1. Copying files from a user’s home directory or from '''SR1''' to '''SCRATCH'''. 
If working with '''HOME''': 

cd /scratch/<userid>
mkdir mySLURM_Job && cd mySLURM_Job
cp /global/u/<userid>/myProject/a.out ./
cp /global/u/<userid>/myProject/mydatafile ./


If working with '''SR1 (storage repository)''': 

cd /scratch/<userid>
mkdir mySLURM_Job && cd mySLURM_Job
iget myProject/a.out
iget myProject/mydatafile


2. Prepare SLURM job script. Typical SLURM sript is similar to the following: 

#!/bin/bash
#SBATCH --partition production
#SBATCH -J test
#SBATCH --nodes 1
#SBATCH --ntasks 8
#SBATCH --mem 4000
echo "Starting…"

cd $SLURM_SUBMIT_DIR
mpirun -np 4 ./a.out ./mydatafile > myoutputs
echo "Done…"

Your SLURM may be different depending on your needs. Read section Submitting Jobs for a reference. 

3. Run the job 

sbatch ./mySLURM_script

4. Once job is finished, clean up '''SCRATCH''' and store outputs in your user home directory or in '''SR1'''. 

If working with '''HOME''': 

mv ./myoutputs /global/u/<userid>/myProject/.
cd ../
rm -rf mySLURM_Job 
== iRODS (The iRODS Section is in REVIEW and may not be CURRENT) ==

'''iRODS''' is the integrated Rule-Oriented Data-management System, a
community-driven, open source, data grid software solution. '''iRODS''' is
designed to abstract data services from data storage hardware and
provide users with hardware-agnostic way to manipulate data.

'''iRODS''' is a primary tool that is used by the CUNY HPCC users to
seamlessly access 1PB storage resource (further referenced as '''SR1'''
here) from any of the HPCC's computational systems.

Access to '''SR1''' is provided via so-called '''i-commands''':

<pre>iinit
ils
imv</pre>

Comprehesive list of i-commands with detailed description can be
obtained at [https://wiki.irods.org/index.php/icommands iRODS wiki].

To obtain quick help on any of the commands while being logged into
any of the HPCC's machines type '''i-command -h'''. For example:
<pre>
ils -h
</pre>

Following is the list of some of the most relevant '''i-commands''':

:'''iinit''' -- Initialize session and store your password in a scrambled form for automatic use by other icommands.
:'''iput''' -- Store a file
:'''iget''' -- Get a file
:'''imkdir''' -- Like mkdir, make an iRODS collection (similar to a directory or Windows folder)
:'''ichmod''' -- Like chmod, allow (or later restrict) access to your data objects by other users.
:'''icp''' -- Like cp or rcp, copy an iRODS data object
:'''irm''' -- Like rm, remove an iRODS data object
:'''ils''' -- Like ls, list iRODS data objects (files) and collections (directories)
:'''ipwd''' -- Like pwd, print the iRODS current working directory
:'''icd''' -- Like cd, change the iRODS current working directory
:'''ichksum''' -- Checksum one or more data-object or collection from iRODS space.
:'''imv''' -- Moves/renames an irods data-object or collection.
:'''irmtrash''' -- Remove one or more data-object or collection from a RODS trash bin.
:'''imeta''' -- Add, remove, list, or query user-defined Attribute-Value-Unit triplets metadata
:'''iquest''' -- Query (pose a question to) the ICAT, via a SQL-like interface

Before using any of the i-commands users need to identify themselves to the iRODS server running command
<pre>
# iinit
</pre>
and providing HPCC's password.

Typical workflow that involves operations on files stored in SR1
include storing/getting data to and from SR1, tagging data with
metadata, searching for data, sharing (setting permissions).

==== Storing data to SR1 ====

1. Create '''iRODS''' directory (aka 'collection'): 

# imkdir myProject

2. Store all files ''''myfile*'''' into this directory (collection): 

# iput -r <userid> myfile* myProject/.

3. Verify that files are stored: 

# ils
/cunyZone/home/<userid>:
C- /cunyZone/home/<userid>/myProject
# ils myProject
/cunyZone/home/<userid>/myProject:
myfile1
myfile2
myfile3


Symbol ''''C-'''' in the beginning of output of ''''ils'''' shows that listed item is a collection.

4. Combining ''''ils', 'imkdir', 'iput', 'icp', 'ipwd', 'imv'''' user can create iRODS directories and store files in them similarly to what is normally done with UNIX commands ''''ls', 'mkdir', 'cp', 'pwd', 'mv'''' etc...

==== Getting data from SR1 ====

1. To copy file from SR1 to current working directory run
# iget myProject/myfile1

2. Now listing current working directory should reveal '''myfile1''':
# ls
myfile1

3. Instead of individual files the whole directory (with
sub-directories) can be copied with ''''-r'''' flag (stands for
'recursive')
# iget -r myProject

NOTE: wildcards are not supported, therefore the command below will not work
# iget myProject/myfile*

=== Tagging data with metadata ===

iRODS provides users with extremely powerful mechanism of managing
data with metadata. While working with large datasets it's
sometimes easy to forget what is stored in this or the other file.
Metadata tags help organizing data in a very easy and reliable
manner.

Let's tag files from previous example with some metadata:
<pre>
# imeta add -d myProject/myfile1 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 comment "This is file number 1"
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 zvalue 10 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 comment "This is file number 2"
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 colorLabel BLUE
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 comment "This is file number 3"
AVU added to 1 data-objects
</pre>
Here we've tagged myfile1 with 3 metadata labels:

- zvalue 10 meters

- colorlabel RED

- comment "This is file number 1"

Similar tags were added to 'myfile2' and 'myfile3'

Metadata come in form of AVU -- Attribute|Value|Unit. As seen from
the above examples Unit is not necessary.

Let's list all metadata assigned to file 'myfie1':
<pre>
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: zvalue
value: 15
units: meters
----
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
</pre>

To remove an AVU assigned to a file run:
<pre>
# imeta rm -d myProject/myfile1 zvalue 15 meters
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
#
#
# imeta add -d myProject/myfile1 zvalue 15 meters
</pre>

Metadata may be assigned to directories as well:
<pre>
# imeta add -C myProject simulationsPool 1
# imeta ls -C myProject
AVUs defined for collection myProject:
attribute: simulationsPool
value: 1
units:
</pre>

Note the '-C' key that is used instead of '-d'.

=== Searching for data ===

Power of metadata becomes obvious when data needs to be found in
large collections. Here is an illustration of how easy this task is
done with iRODS via imeta queries:

# imeta qu -d zvalue = 15
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile3

We see both files that were tagged with label 'zvalue 10 meters'.
Here is different query:

# imeta qu -d colorLabel = RED
collection: /cunyZone/home/<userid></myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile2

Another powerful mechanism to query data is provided with 'iquest'.
Following is a number of examples that show 'iquest' capabilities:

iquest "SELECT DATA_NAME, DATA_SIZE WHERE DATA_RESC_NAME like 'cuny%'"
iquest "For %-12.12s size is %s" "SELECT DATA_NAME , DATA_SIZE WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest "SELECT COLL_NAME WHERE COLL_NAME like '/cunyZone/home/%' AND USER_NAME like '<userid>'"
iquest "User %-6.6s has %-5.5s access to file %s" "SELECT USER_NAME, DATA_ACCESS_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest " %-5.5s access has been given to user %-6.6s for the file %s" "SELECT DATA_ACCESS_NAME, USER_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>>'"
iquest no-distinct "select META_DATA_ATTR_NAME"
iquest "select COLL_NAME, DATA_NAME WHERE DATA_NAME like 'myfile%'"
iquest "User %-9.9s uses %14.14s bytes in %8.8s files in '%s'" "SELECT USER_NAME, sum(DATA_SIZE),count(DATA_NAME),RESC_NAME"
iquest "select sum(DATA_SIZE) where COLL_NAME = '/cunyZone/home/<userid>'"
iquest "select sum(DATA_SIZE) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select sum(DATA_SIZE), RESC_NAME where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select order_desc(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select count(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select RESC_NAME where RESC_CLASS_NAME IN ('bundle','archive')"
iquest "select DATA_NAME,DATA_SIZE where DATA_SIZE BETWEEN '100000' '100200'"

=== Sharing data ===

Access to the data can be controlled via 'ichmod' command. It's
behavior is similar to UNIX 'chmod' command. For example if there is a
need to provide user '''<userid>''' with read access to file
'''myProject/myfile1''' execute the following command:
ichmod read <userid1> myProject/myfile1

To see who has access to a file/directory use:
# ils -A myProject/myfile1
/cunyZone/home/<userid>/myProject/myfile1
ACL - <userid1>
#cunyZone:read object <userid>#cunyZone:own

In the above example user '''<userid1>''' has read access to the file and
user '''<userid>''' is an owner of the file.

Possible levels of access to a data object are null/read/write/own.
==Backups (IN REVIEW)==

'''Backups.''' '''/global/u''' user directories. Project files are backed up automatically to a remote tape silo system over a fiber optic network. Backups are performed daily.

If the user deletes a file from '''/global/u''', it will remain on the tape silo system for 30 days, after which it will be deleted and cannot be recovered. If a user, within the 30 day window finds it necessary to recover a file, the user must expeditiously submit a request to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu].

Less frequently accessed files are automatically transferred to the HPC Center robotic tape system, freeing up space in the disk storage pool and making it available for more actively used files. The selection criteria for the migration are age and size of a file. If a file is not accessed for 90 days, it may be moved to a tape in the tape library – in fact to two tapes, for backup. This is fully transparent to the user. When a file is needed, the system will copy the file back to the appropriate disk directory. No user action is required.

==Data retention and account expiration policy (IN REVIEW)==

Project directories on SR1 are retained as long as the project is active. The HPC Center will coordinate with the Principal Investigator of the project before deleting a project directory. If the PI is no longer with CUNY, the HPC Center will coordinate with the PI’s departmental chair or Research Dean, whichever is appropriate.

For user accounts, current user directories under /global/u are retained as long as the account is active. If a user account is inactive for one year, the HPC Center will attempt to contact the user and request that the data be removed from the system. If there is no response from the user within three months of the initial notice, or if the user cannot be reached, the user directory will be purged.

==DSMS Technical Summary (IN REVIEW)==
{| class="wikitable"
|+
!File Space
!Purpose
!Accessibility
!Quota
!Backups
!Purges
|-
|Scratch:
/scratch/<userid>
on *PENZIAS, ANDY, SALK, BOB*
|High Performance Parallel scratch filesystems. Work area for jobs, datasets, restart files, files to be pre-/post processed. Temporary space for data that will be removed within a short amount of time.
|Not globally accessible.
Separate /scratch/<userid> exists on each system. Visible on login and compute nodes of each system and on the data transfer nodes.
|None
|None
|Files older than 2 weeks are automatically deleted
OR
when scratch filesystem reaches 70% utilization
|-
|Home:
/global/u/<userid>
|User home filespace. Essential data should be stored here, such as user's source code, documents, and data structures.
|Globally accessible on the login and on the data transfer nodes through native GPFS or NFS mounts
|Nominally 50 GB
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days.
| rowspan="2" |Not purged
|-
|Project:
/SR1/<PID>
|Project space allocations
|Accessible on the login and on the data transfer nodes. Accessible outside CUNY HPC Center through iRODS.
|Allocated according to project needs
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days and retrievable on request, but the iRODS metadata may be lost.
|}
• '''SR1''' is tuned for high bandwidth, redundancy, and resilience. It is not optimal for handling large quantities of small files. If you need to archive more than a thousand of files on '''SR1''', please create a single archive using '''tar'''.

• A separate '''/scratch/<userid>''' exists on each system. On PENZIAS, SALK, KARLE, and ANDY, this is a Lustre parallel file system, on HERBERT it is NFS. These '''/scratch''' directories are visible on the login and compute nodes of the system only and on the data transfer nodes, but are not shared across HPC systems.

• '''/scratch/<userid>''' is used as a high performance parallel scratch filesystem, for example, temporary files (e.g. restart files) should be stored here.

• There are no quotas on '''/scratch/<userid>''', however any files older than 2 weeks are automatically deleted. Also, a cleanup script is scheduled to run every two weeks or whenever the /scratch disk space utilization exceeds 70%. Dot-files are generally left intact from these cleanup jobs.

• /scratch space is available to all users. If the scratch space is exhausted, jobs will not be able to run. Purge any files in '''/scratch/<userid>''', which are no longer needed, even before the automatic deletion kicks in.

• '''/scratch/<userid>''' directory may be empty when you login, you will need to copy any files required for submitting your jobs (submission scripts, data sets) from '''/global/u''' or from '''SR1'''. Once your jobs complete copy any files you need to keep back to '''/global/u''' or '''SR1''' and remove all files from /scratch.

• Do not use '''/tmp''' for storing temporary files. The file system where '''/tmp''' resides in memory is very small and slow. Files will be regularly deleted by automatic procedures.

• '''/scratch/<userid>''' is not backed up and there is no provision for retaining data stored in these directories.
 

==Data Handling Practices ==
===HPFS, i.e., /global/u ===

• The HPFS is not an archive for non-HPC users. It is an archive for users who are processing data at the HPC Center. “Parking” files on the '''HPFS''' as a back-up to local data stores is prohibited.

• Do not store more than 1,000 files in a single directory. Store collections of small files into an archive (for example, tar). Note that for every file, a stub of about 4MB is kept on disk even if the rest of the file is migrated to tape, meaning that even migrated files take up some disk space. It also means that files smaller than the stub size are never migrated to tape because that would not make sense. Storing a large number of small files in a single directory degrades the file system performance.
 

===/scratch===

• Please regularly remove unwanted files and directories and avoid keeping duplicate copies in multiple locations. File transfer among the HPC Center systems is very fast. It is forbidden to use "touch jobs" to prevent the cleaning policy from automatically deleting your files from the '''/scratch''' directories. Use '''tar -xmvf''', not '''tar -xvf''' to unpack files. '''tar -xmvf''' updates the times stamp on the unpacked files. The '''tar -xvf''' command preserves the time stamp from the original file and not the time when the archive was unpacked. Consequently, the automatic deletion mechanism may remove files unpacked by '''tar –xvf''', which are only a few days old.

Main Page

2026-03-09T17:16:48Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI. In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended "Alpha" server as well new "Beta" computer.''' The latter will consist of 288 B200 GPU and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit (SU),''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section HPCC access plans for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T17:12:37Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).''' See section '''HPCC access plans''' for further details.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T17:12:07Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). See section HPCC access plans for further details.'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T17:11:28Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs for public servers are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units). See section'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T17:09:05Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T17:06:53Z

Alex: /* Empire AI and CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper GPU) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50/unit,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. One SU corresponds to one hour of H100 compute, and one hour of B200 compute corresponds to two SU. In comparison the CUNY-HPCC recovery costs are '''$0.015 per CPU hour (1 unit) and $0.09 per GPU hour (6 units).'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:56:15Z

Alex: /* CUNY-HPCC services */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper GPU) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. In comparison the CUNY-HPCC recovery costs are '''$0.015 per CPU hour and $0.09 per GPU hour.'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.
:*Maintains tickets for all CUNY users with allocation on EAI.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:55:20Z

Alex:

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== Empire AI and CUNY-HPCC ==
The Empire AI consortium includes the '''CUNY Graduate Center''', Columbia University, Cornell University, Icahn School of Medicine, New York University, Rochester Institute of Technology, Rensselaer Polytechnic Institute, the State University of New York, University at Buffalo, and University of Rochester. '''CUNY-HPCC provides supports and maintains tickets for all CUNY users with allocation on EAI.''' In addition CUNY-HPCC is a stepping stone for CUNY researchers since it operates (on smaller scale) architectures (Including nodes with Hopper GPU) similar to EAI, including extended Alpha server as well new Beta. The latter will consist of '''288 B200 GPU''' and recently added RTX 6000 pro nodes. The expected cost for '''EAI is $0.50,''' which will provide CUNY PIs with a rate that is well below a typical AWS rate. In comparison the CUNY-HPCC recovery costs are '''$0.015 per CPU hour and $0.09 per GPU hour.'''

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:25:24Z

Alex: /* CUNY-HPCC services */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Supports research computing at CUNY - for faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPCC capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:23:58Z

Alex: /* CUNY-HPCC services */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects. CUNY-HPCC:

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:23:32Z

Alex: /* Mission of CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward EAI advanced facilities.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects.

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:21:38Z

Alex: /* Mission of CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is a member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward advanced architectures at EAI.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects.

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:21:15Z

Alex: /* Mission of CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance education and boost scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward advanced architectures at EAI.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects.

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:20:24Z

Alex: /* Mission of CUNY-HPCC */

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research and education facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance advances scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward advanced architectures at EAI.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects.

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-09T16:19:04Z

Alex:

__TOC__

[[Image:hpcc-panorama3.png]]

== Mission of CUNY-HPCC ==
The City University of New York (CUNY) High Performance Computing Center (HPCC) is a core research facility for the University. It is located at campus of the college of College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The core mission of CUNY-HPCC is to advance advances scientific research and discovery at the University by managing state-of-the-art computing infrastructure and by providing comprehensive research support services including domain-specific expertise in various aspects of computationally intensive research. CUNY is member of Empire AI (EAI) consortium, so CUNY-HPCC is a stepping stone for CUNY researchers toward advanced architectures at EAI.

== CUNY-HPCC services ==
CUNY-HPCC provides professionally maintained, modern computational environment and architectures, advanced storage and fast interconnects.

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Provides state-of-the-art computing resources and comprehensive research support services including expertise and full support for users with allocation on EMPIRE-AI.
:*Creates opportunities for the CUNY research community to develop new partnerships with the government and private sectors.
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Administrative Information

2026-03-05T19:25:23Z

Alex: /* Accounts overview */

__TOC__
==How to get an account==

=== Definitions and procedures ===
CUNY-HPCC operates on cost recovery scheme which requires all accounts to be associated with research project(s) or to be class accounts. The research accounts are sponsored by Principle Investigator (PI). A '''Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project, ensuring compliance with regulations and overseeing the project's financial aspects. PI is a faculty member or a qualified researcher who has the authority to apply for research funding and manage the project.''' The procedure to open an account is as follows:

'''Step 1.''' Creation of sponsor account (PI account) - '''form A or B below.''' At that step the PI must create account for him/her and provide information about project title, funding and duration. The request of resources is not mandatory.

'''Step 2.''' Upon creating account the PI will get unique code which has to be shared with members of a group (students and post docs) that require account on HPCC.

'''Step 3.''' Members of research group (lab) and academic collaborators can apply for account at CUNY-HPCC by using form C,D, E or F It is mandatory to use a code mentioned in Step 2 (from CUNY PI) in these forms.

'''Step 4.''' PI should assign students to his project.

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from '''F,''' and '''G''' account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail.
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY E-mail, For '''PhD students the first E-mail is their GC E-mail address. Second mail is the college E-mail.'''
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail. First E-mail is the college E-mail address.
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator E-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''A,C,E and F''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''A,C,E and F''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Main Page

2026-03-05T19:19:56Z

Alex: /* Free time */

__TOC__

[[Image:hpcc-panorama3.png]]

The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the
campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. HPCC
goals are to:

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Create opportunities for the CUNY research community to develop new partnerships with the government and private sectors; and
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from CUNY colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Main Page

2026-03-05T19:19:33Z

Alex: /* Free time */

__TOC__

[[Image:hpcc-panorama3.png]]

The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the
campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. HPCC
goals are to:

:*Support the scientific computing needs of CUNY faculty, their collaborators at other universities, and their public and private sector partners, and CUNY students and research staff.
:*Create opportunities for the CUNY research community to develop new partnerships with the government and private sectors; and
:*Leverage the HPC Center capabilities to acquire additional research resources for its faculty and graduate students in existing and major new programs.

==Organization of systems and data storage (architecture)==

All user data and project data are kept on Parallel File System Storage (PFSS) which is mounted only on login node(s) of all servers. It holds both user directories and specific partition called '''/scratch (see below).''' The main features of '''/scratch''' partition are''': 1.''' It is mounted on all computational nodes and on all login nodes '''2.''' It is fast '''3.''' '''/scratch''' partition is temporary space, but is '''not a home directory''' for accounts nor can be used for long term data preservation. Users must use "staging" procedure described below to ensure preservation of their data, codes and parameters files. The figure below is a schematic of the environment.

Upon registering with HPCC every user will get 2 directories:

:• '''/scratch/<userid>''' – this is temporary workspace on the HPC systems. Currently scratch resides on the same file system as global/u.
:• '''/global/u/<userid>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

:• In some instances a user will also have use of disk space on the DSMS in '''/cunyZone/home/<projectid>''' (IRods).
[[File:HPCC_structure.png|center|frameless|900x900px]]
The '''/global/u/<userid>''' directory has quota (see below for details) while the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up following the rules described below. There are no guarantees of any kind that files in '''/scratch''' will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''/global/u/<userid>''' or to/from '''/scratch/<userid>'''

==HPC systems==

The HPC Center operates variety of architectures in order to support complex and demanding workflows. All computational resources of different types are united into single hybrid cluster called Arrow. The latter deploys symmetric multiprocessor (also referred as SMP) nodes with and without GPU, distributed shared memory (NUMA) node, fat (large memory) nodes and advanced SMP nodes with multiple GPU. The number of GPU per node varies between 2 and 8 as well as employed GPU interface and GPU family. Thus the basic GPU nodes hold two Tesla K20m (plugged through PCIe interface) while the most advanced ones support eight Ampere A100 GPU connected via SXM interface.

''Overview of Computational architectures'':

'''SMP''' servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU.

'''Cluster''' is defined as a single system comprising set of servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. The most common cluster type is the one that consists of several identical SMP servers connected via fast interconnect. Each SMP member of the cluster is called a '''node'''. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU.

Hybrid clusters combine nodes of different architectures. For instance the main CUNY-HPCC machine is a hybrid cluster called '''Arrow'''. Sixty two (62) of its nodes are identical GPU enabled SMP servers each with 2 x GPU K20m, 3 are SMP but with extended memory (fat nodes), one node is distributed shared memory node (NUMA, see below) and 2 are fat SMP servers especially designed to support 8 NVIDIA GPU per node. The latter are connected via SXM interface. In addition HPCC operates the cluster '''Herbert''' dedicated only to education.

'''Distributed shared memory''' computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the '''NUMA''' systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the '''NUMA''' node at Arrow named '''Appel'''. This node does not have GPU.

'' Infrastructure systems'':

o Master Head Node ('''MHN/Arrow)''' is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus. Note that name of main server and its login nodes are the same Arrow. Thus users can access the Arrow login nodes using name Arrow or MHN.

o '''Chizen''' is a redundant gateway server which provides access to protected HPCC domain.

o '''Cea''' is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. '''Cea''' is accessible directly (not only via '''Chizen'''), but allows only limited set of shell commands.

'''Table 1''' below provides a quick summary of the attributes of each of the sub clusters of the main HPC Center called Arow.

{| class="wikitable"
|+
!Master Head Node
!Sub System
!Tier
!Type
!Type of Jobs
!Nodes
!CPU Cores
!GPUs
!Mem/node
!Mem/core
!Chip Type
!GPU Type and Interface
|-
| rowspan="17" |'''<big>Arrow</big>'''
| rowspan="4" |Penzias
| rowspan="10" |Advanced
| rowspan="4" |Hybrid Cluster
|Sequential & Parallel jobs w/wo GPU
|66
|16
|2
|64 GB
|4 GB
|SB, EP 2.20 GHz
|K20m GPU, PCIe v2
|-
| rowspan="3" |Sequential & Parallel jobs
| rowspan="3" |1
|24
| -
|1500 GB
|62 GB
| rowspan="3" |HL, 2.30 GHz
| -
|-
|36
| -
|768 GB
|21 GB
| -
|-
|24
| -
|768 GB
|32 GB
| -
|-
|Appel
|NUMA
|Massive Parallel, sequential, OpenMP
|1
|384
| -
|11 TB
|28 GB
|IB, 3 GHz
| -
|-
|Cryo
|SMP
|Sequential and Parallel jobs, with GPU
|1
|40
|8
|1500 GB
|37 GB
|SL, 2.40 GHz
|V100 (32GB) GPU, SXM
|-
| rowspan="2" |Blue Moon
| rowspan="2" |Hybrid Cluster
| rowspan="2" |Sequential and Parallel jobs w/wo GPU
|24
|32
| -
| rowspan="2" |192 GB
| rowspan="2" |6 GB
| rowspan="2" |SL, 2.10 GHz
| -
|-
|2
|32
|2
|V100(16GB) GPU, PCIe
|-
|Karle
|SMP
|Visualization, MATLAB/Mathematica
|1
|36*
| -
|768 GB
|21 GB
|HL, 2.30 GHz
| -
|-
|Chizen
|Gateway
|No jobs allowed
| colspan="7" | -
|-
| rowspan="2" |CFD
| rowspan="2" |Condo
| rowspan="2" |SMP
| rowspan="7" |Parallel, Seq, OpenMP
|1
|48
|2
|768 GB
|
|EM, 4.8 GHz
|A40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |PHYS
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|640 GB
|
|ER, 4 GHz
|L40, PCIe, v4
|-
|1
|48
| -
|512 GB
|
|ER, 4.3 GHz
| -
|-
| rowspan="2" |CHEM
| rowspan="2" |Condo
| rowspan="2" |SMP
|1
|48
|2
|256 GB
|
|EM, 2.8 GHz
|A30, PCIe, v4
|-
|1
|128
|8
|512 GB
|
|ER, 2.0 GHz
|A100/40, SXM
|-
|ASRC
|Condo
|SMP
|1
|48
|2
|256 GB
|
|ER, 2.8 GHz
|A30, PCIe, v4
|}
Note: SB = Intel(R) Sandy Bridge, HL = Intel (R) Haswell, IB = Intel (R) Ivy Bridge, SL = Intel (R) Xeon(R) Gold, ER = AMD(R) EPYC ROMA, EM = AMD(R) EPYC MILAN, EG = AMD (R) EPYC GENOA

== Recovery of operational costs ==
CUNY-HPCC is not for profit core research facility at CUNY. Our mission is to support all types of research that require advanced computational resources. CUNY-HPCC operations are not for profit and are NOT directly or indirectly sponsored by CUNY or College of Staten Island (CSI). Consequently CUNY-HPCC applies cost recovery model recapturing only '''operational costs with no profit for HPCC'''. The recovered costs are calculated using actual documented operational expenses and are break even for all CUNY users. The used methodology is approved by CUNY-RF methodology used in other CUNY research facilities. The costs are reviewed and consequently updated twice a year. The cost recovery charging schema is based on '''unit-hour'''. The unit can be either CPU unit or GPU unit. The definitions of these is given in a table below:
{| class="wikitable mw-collapsible"
|+Definitions of unit-hour
!Type of resource
!Unit-hour
!For V100, A30, A40 or L40
!For A100
|-
|CPU unit
|1 cpu core/hour
| --
| --
|-
|GPU unit
|(4 cpu cores + 1 GPU thread )/hour
|4 cpu cores + 1 GPU
|4 cpu cores and 1/7 A100
|}

=== HPCC access plans ===
a. '''Minimum access (MAP):'''

Minimum access is designed to provide wide support for research activities in any college, to promote collaboration between colleges, to help establish a new research project, and/or to be testbed for new studies. MAP accounts operate under strict fair share policy so actual waiting time for a job in a que depends on resources used by that account in previous cycles. In addition all jobs have strict time limitations. Therefore long jobs must use check-points.

The MAP has 3 tiers:

· A: Basic tier fee is $5000 per year. It is designed to provide support for users from colleges with low level of research activities. The fee covers infrastructure expenses associated with 1-2 users from these colleges.

· B: Medium tier fee is $15,000 per year. The fee covers infrastructure expenses of up to 12 users from these colleges. In addition, every account under medium tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

· C: Advanced tier is $25,000 per year. The fee covers infrastructure expenses for all users from these colleges. In addition every new account from this tier gets free 11520 CPU hours and free 1440 GPU hours upon opening.

The MAP users get charged per CPU/GPU hour at low rate of '''$0.015 per cpu hour and $0.09 per GPU hour.'''

{| class="wikitable mw-collapsible"
|+Cost recovery fees for MAP users
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.015/hour
|-
|16 cores no GPU
|4
|0
|$0.24/hour
|-
| 4 cores + 1 GPU
|4
|1
|$0.15/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.33/hour
|-
|16 cores + 2 GPU
|16
|2
|$0.42/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.66/hour
|-
|40 cores + 8 GPU
|40
|8
|$1.32/hour
|}

b. '''Computing on demand (CODP)'''

Computing on demand plan (CODP) is open for all users from all CUNY colleges that do not participate in MAP plan, but want to use the HPCC resources. CODP accounts operate under strict fair share policy, so actual waiting time for a job in a que depends on resources previously used. In addition, all jobs have time limitations, so long jobs must use check-points. The users in CODP are charged for the time (CPU and GPU) per hour. The current rates are '''$0.018 per cpu hour and $0.11 per GPU hour.''' In difference to MAP, the new CODP accounts does not come with free time. The invoices are generated and send to users (PI only) at the end of each month. The examples in following table explain the fees structure:

{| class="wikitable mw-collapsible"
|+Cost recovery fees for CODP plan
|Job
|Cpu cores
|GPU
|Cost/hour
|-
|1 core no GPU
|1
|0
|$0.018/hour
|-
|16 cores no GPU
|16
|0
|$0.288/hour
|-
|4 cores + 1 GPU
|4
|1
|$0.293/hour
|-
|16 cores + 1 GPU
|16
|1
|$0.334/hour
|-
|32 cores + 1 GPU
|32
|1
|$0.666/hour
|-
|32 cores + 2 GPU
|32
|2
|$0.756/hour
|}

c. '''Leasing node(s) (LNP)'''

Leasing node plan allows the users to lease the node(s) for the duration of the project. The minimum lease time is 30 days (one month), but leases of any length are possible. Discounts of 10% are given to users whose lease is longer than 90 days. Discounts cannot be combined. In difference to MAP and CODP the LNP users do not compete for resources and have full access to rented resources 24/7.

{| class="wikitable mw-collapsible"
|+Lease node(s) fees for MAP users
|Job (MAP users)
|Cpu cores
|GPU
|Cost/30 days
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$172.80
|-
|32 cores no GPU
|32
|0
|$264.96
|-
|16 cores + 2 GPU
|16
|1
|$302.40
|-
|32 cores + 2 GPU
|32
|2
|$475.20
|-
|40 cores + 8 GPU
|40
|8
|$760.0
|-
|64 cores + 8 GPU
|64
|8
|$950.40
|}

{| class="wikitable mw-collapsible"
|+Fees for lease a node(s) for non -MAP users
|Job (non-MAP users)
|Cpu cores
|GPU
|Cost/month
|-
|1 core no GPU
|1
|0
|NA
|-
|16 cores no GPU
|16
|0
|$249.82
|-
|32 cores no GPU
|32
|0
|$497.64
|-
|16 cores + 1 GPU
|16
|2
|$443.23
|-
|32 cores + 2 GPU
|32
|2
|$886.64
|-
|40 cores + 8 GPU
|40
|8
|$1399.68
|}

d. '''Condo Ownership (COP)'''

Condo describes a model when user(s) own a node/server managed by HPCC. Only full time faculty can own condo node. Condo nodes are fully integrated into HPCC infrastructure. The owners pay only HPCC’s infrastructure support operational fee which includes only proportional part of licenses and materials need for day-to-day operations. The fees are reviewed twice a year and currently are $0.003 per CPU hour and $0.02 per GPU hour. Condo owners can “borrow” (upon agreement) free of charge any node(s) from condo stack and can also lease (for higher fee – see below) their own nodes to non-condo users. The minimum let time is 30 days. The fees collected from non-condo users offset payments of the owner.
{| class="wikitable mw-collapsible"
|+Condo owners costs per year
|Type of condo node
|Cpu cores
|GPU
|Cost/year
|-
|Large hybrid SXM
|128
|8
|$4518.92
|-
|Small hybrid
|48
|2
|$1540.54
|-
|Medium compute
|96
|0
|$2464.86
|-
|Large compute
|128
|0
|$3286.49
|}

Condo owners can contract their node(s) to other non-condo users. Renting period is unlimited with min. length of 30 days. The table below shows the payments the non-condo users recompense the condo owners. These fees are accumulated in owners account(s) and do offset the owner’s duties. Discount of 10% is applied for leases longer than 90 days.
{| class="wikitable mw-collapsible"
|+Type of nodes and lease fees for condo nodes
!Type of node
!Renters cost/month
!Long term (90+ days) rent cost/month
!CPU/node
!CPU type
!GPU/node
!GPU type
!GPU interface
|-
|Laghe Hybrid
|$602.52
|$564.86
|128
|EPYC, 2.2 GHz
|8
|A100/80
|SXM
|-
|Small Hybrid
|$205.41
|$192.57
|48
|EPYC, 2.8 GHz
|2
|A40, A30, L40
|PCIe v4
|-
|Medium Non GPU
|$328.65
|$308.11
|96
|EPYC, 4.11GHz
|48
|None
|NA
|-
|Lagre Non GPU
|$438.20
|$410.81
|128
|EPYC, 2.0 GHz
|128
|None
|NA
|}

=== Free time ===
Any new project '''from colleges/centers that participate in MAP-B or MAP-C''' is entitled to get free '''11520 CPU hours and 1440 GPU hours.''' Users under '''MAP-A''' are not entitled for free time. The free compute hours are intended to help to establish a project and thus are shared for all members of the project. Thus compute free hours can be used either by PI or by any number of project's members. It is important to note that free time is per project not per user account, so any project can have free time only once. External collaborators to CUNY are not normally eligible for free time. Additional hours beyond free time are charged with MAP plan rates. '''Please contact CUNY-HPCC director for further details.'''

== Support for research grants ==
'''All proposals dated on Jan 1st 2026 ( 01/01/26 ) and later''' that require computational resources '''must include budget for cost recovery fees at CUNY-HPCC.''' For a project the PI can choose between:

* lease the node(s), That is useful option for well defined projects and those with high computational component requiring 100% availability of the computational resource.
* use "on-demand" resources. That is flexible option good for experimental projects or exploring new areas of study. The downgrade is that resources are shared among all users under fair share policy. Thus immediate access to resource cannot be guaranteed.
* participate in CONDO tier. That is most beneficial option in terms of availability of resources and level of support. It fits best the focused research of group(s) (e.g. materials science).

In all cases the PI can use the appropriate rates listed above to establish correct budget for the proposal. PI should '''contact the Director of CUNY-HPCC Dr. Alexander Tzanov''' (alexander.tzanov@csi.cuny.edu) and discuss the project's computational requirements including optimal and most economical computational workflows, suitable hardware, shared or own resources, CUNY-HPCC support options and any other matter concerning correct and optimal computational budget for the proposal.

== Partitions and jobs ==
The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in '''partitions'''. The table below describes the partitions and their limitations. The users are granted permissions house one or other partition and corresponding QOS key. The table below shows the limitations of the partitions (in progress).
{| class="wikitable mw-collapsible"
|+
!Partition
!Max cores/job
!Max jobs/user
!Total cores/group
!Time limits
!Tier
!
!GPU types
!
|-
|partnsf
|128
|50
|256
|240 Hours
|Advanced
|
|K20m, V100/16, A100/40
|
|-
|partchem
|128
|50
|256
|No limit
|Condo
|
|A100/80, A30
|
|-
|partcfd
|96
|50
|96
|No limit
|Condo
|
|A40
|
|-
|partsym
|96
|50
|96
|No limit
|Condo
|
|A30
|
|-
|partasrc
|48
|16
|16
|No limit
|Condo
|
|A30
|
|-
|partmatlabD
|128
|50
|256
|240 Hours
|Advanced
|
|V100/16,A100/40
|
|-
|partmatlabN
|384
|50
|384
|240 Hours
|Advanced
|
|None
|
|-
|partphys
|96
|50
|96
|No limit
|Condo
|
|L40
|
|}

* '''partnsf''' is the main partition with assigned resources across all sub-servers. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.
* '''partchem''' is CONDO partition.
* '''partphys''' is CONDO partition
* '''partsym''' is CONDO partition
* '''partasrc''' is CONDO partition
* '''partmatlabD''' partition allows to run MATLAB's Distributes Parallel Server across main cluster.
* '''partmatlabN''' partition to access large matlab node with 384 cores and 11 TB of shared memory. It is useful to run parallel Matlab jobs with Parallel ToolBox
* '''partdev''' is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

== Hours of Operation ==
In order to maximize the use of resources HPCC applies “rolling” maintenance scheme across all systems. When downtime is needed, HPCC will notify all users a week or more in advance (unless emergency situation occur). Typically, the fourth Tuesday mornings in the month from 8:00AM to 12PM is normally reserved (but not always used) for scheduled maintenance. Please plan accordingly. Unplanned maintenance to remedy system related problems may be scheduled as needed out of above mentioned days. Reasonable attempts will be made to inform users running on those systems when these needs arise. Note that users are strongly encouraged to use checkpoints in their jobs.

== User Support ==
Users are strongly encouraged to read this Wiki carefully before submitting ticket(s) for help. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPCC systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications
and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY
community in parallel programming techniques, HPC computing architecture, and the essentials of using our
systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

[mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]

== Warnings and modes of operation ==

1. hpchelp@csi.cuny.edu is for questions and accounts help communication '''only''' and does not accept tickets unless ticketing system is not operational. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. '''E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address.''' Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. '''Do not send questions to individual CUNY HPC Center staff members directly.''' These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared
to other HPC Centers of similar size '''our staff is extremely lean'''. Please make full use of the tools that we have provided (especially
the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using
our systems will be predictably good and productive.

== User Manual ==

The old version of the user manual provides PBS not SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM scheduler so users must check and use only the updated brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.

Administrative Information

2026-03-05T19:07:33Z

Alex: /* User accounts policies */

__TOC__
==How to get an account==

=== Definitions and procedures ===
CUNY-HPCC operates on cost recovery scheme which requires all accounts to be associated with research project(s) or to be class accounts. The research accounts are sponsored by Principle Investigator (PI). A '''Principal Investigator (PI) at CUNY is defined as the lead researcher responsible for the design, execution, and management of a research project, ensuring compliance with regulations and overseeing the project's financial aspects. PI is a faculty member or a qualified researcher who has the authority to apply for research funding and manage the project.''' The procedure to open an account is as follows:

'''Step 1.''' Creation of sponsor account (PI account) - '''form A or B below.''' At that step the PI must create account for him/her and provide information about project title, funding and duration. The request of resources is not mandatory.

'''Step 2.''' Upon creating account the PI will get unique code which has to be shared with members of a group (students and post docs) that require account on HPCC.

'''Step 3.''' Members of research group (lab) and academic collaborators can apply for account at CUNY-HPCC by using form C,D, E or F It is mandatory to use a code mentioned in Step 2 (from CUNY PI) in these forms.

'''Step 4.''' PI should assign students to his project.

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from '''F,''' and '''G''' account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator e-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''A,C,E and F''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''A,C,E and F''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Administrative Information

2026-03-05T19:04:19Z

Alex: /* Definitions and procedures */

__TOC__
==How to get an account==

=== Definitions and procedures ===
CUNY-HPCC operates on cost recovery scheme which requires all accounts to be associated with research project(s) or to be class accounts. The research accounts are sponsored by Principle Investigator (PI). A '''Principal Investigator (PI) at CUNY is the lead researcher responsible for the design, execution, and management of a research project, ensuring compliance with regulations and overseeing the project's financial aspects. PI is a faculty member or a qualified researcher who has the authority to apply for research funding and manage the project.''' The procedure to open an account is as follows:

'''Step 1.''' Creation of sponsor account (PI account) - '''form A or B below.''' At that step the PI must create account for him/her and provide information about project title, funding and duration. The request of resources is not mandatory.

'''Step 2.''' Upon creating account the PI will get unique code which has to be shared with members of a group (students and post docs) that require account on HPCC.

'''Step 3.''' Members of research group (lab) and academic collaborators can apply for account at CUNY-HPCC by using form C,D, E or F It is mandatory to use a code mentioned in Step 2 (from CUNY PI) in these forms.

'''Step 4.''' PI should assign students to his project.

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from F, and G account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator e-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''a,c,e, and f''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''a,c,e and f''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)

Administrative Information

2026-03-05T18:49:38Z

Alex: /* How to get an account */

__TOC__
==How to get an account==

=== Definitions and procedures ===
CUNY-HPCC operates on cost recovery scheme which requires all accounts to be associated with research project(s) or to be class accounts. The research accounts are sponsored by Principle Investigator (PI). A '''Principal Investigator (PI) at CUNY is the lead researcher responsible for the design, execution, and management of a research project, ensuring compliance with regulations and overseeing the project's financial aspects. PI is a faculty member or a qualified researcher who has the authority to apply for research funding and manage the project.''' CUNY-HPCC

===Accounts overview===
All users of HPCC resources must register with HPCC for an account described in a table below. A user account is issued to an '''''individual user'''.'' Accounts are '''not to be shared'''. HPCC '''will communicate only via CUNY e-mails. to users from groups A to E.''' HPCC will communicate with users from F, and G account types via users' verified work account, CC to the CUNY collaborator (for F only). In addition if resources are available and per discretion of CUNY-HPCC director, external to CUNY researchers can obtain external research account (type G) at CUNY-HPCC by renting HPC resources and paying in advance the full cost recovery fee. Please contact the HPCC director for details.
{| class="wikitable sortable mw-collapsible"
|+
!User accounts for:
!Type
!Conditions
!Renuwal cycles
!Conditions
!Mandatory conditions
|-
|Faculty, Research Staff
|A
|Renews every year at the beginning of Fall semester
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Adjunct Faculty
|B
|Renews every semester (Fall/Spring)
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Doctoral Graduate Students
|C
|Renews every year the beginning of Fall semester
|14
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Master Students
|D
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|CUNY EID, Valid CUNY e-mail
|-
|Undergraduate Students
|E
|Renews every semester (Fall/Spring)
|8
|Non renewed accounts are disabled in 7 days from renewal date, data and home directory are removed after 15 days. No backup of any data.
|CUNY EID, Valid CUNY e-mail
|-
|Academic Collaborators
|F
|Renews once a year (Fall) for the duration of a project
|Unlimited
|Non renewed accounts are disabled in 15 days from renewal date, data in home directory is removed after 90 days. Backup data have rollover time of 30 days.
|other institution EID, work e-mail and valid CUNY collaborator e-mail

|-
|Public and Private Sector Partners
|G
|No Renewal. Good only for the duration of contract.
|NA
|Account expires at the date of expiring of the contract.
|state/federal ID, verified work e-mail.'''Advanced pay of full cost for rented resource.'''
|}

Users who missed renewal for less than 90 days should communicate with HPCC via e-mail to '''hpchelp@csi.cuny.edu''' for account recovery. All users must inform HPCC for changes it their academic status. It is mandatory to specify information (or NA) on all points from below list. Please do not forget to provide information about past and pending '''publications''' and funded projects and information about your local available resources (local servers and workstations/desktops only). Think carefully about the resources needed and try to estimate as accurate as possible. Note that '''by applying for and obtaining an account, the user agrees to the HPCC End User Policy (EUP) and Mandatory Security Requirements for Access (MSRA).'''
{| class="wikitable sortable mw-collapsible"
|+Required Information for opening of HPCC account. Please provide information in all fields and/or mark NA when needed.
!
! rowspan="26" |
|-
|'''''<big>For All CUNY Faculty, Staff And Graduate Students</big>''' '''<big>(A ,B,C,D)</big>'''''
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
|-
|''CUNY EID and valid CUNY e-mail (e.g. John A. Smith 22341356 jsmith@csi.cuny.edu):''
|-
|CUNY ''Academic status ( faculty, adjunct faculty, PhD student, MS student, research staff):''
|-
|'''''Primary''' Affiliation within CUNY - campus name and Department ( e.g Hunter College, Biology):''
|-
|'''''Secondary''' CUNY affiliation if any. Provide campus name and Department (e.g. Graduate Center, Biology):''
|-
|Name, department and college affiliation of PI/Advisor (e.g. John Smith, Biology, Hunter College):
|-
|If out of College of Staten Island provide description of local resources available.
:Please state NONE if you do not have access to local computational resources. Otherwise provide:
::College (e.g. Hunter)
::Type of resource (e.g. Department cluster):
:::''- number of nodes:''
:::''- number of cores per node:''
:::''-- memory per node:''
:::''- total number of GPU for server:''
:::''- number of GPU per node:''
:::''-'' ''type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|Resources needed for the project:
::- CPU Cores (e.g 1000):
::- GPU options (e.g 2 x V100/16 GB):
::- V100/16 GB -
::- V100/32 GB -
::- L40/48 GB -
::- A30/24 GB -
::- A40/24 GB -
::- A100/40 GB -
::- A100/80 GB -
::- Storage Space (above 50 GB)
::- Backup of data (Y/N):
::- Archive of data (Y/N):
|-
|'''Title of the project:'''
|-
|Short Description of the project (up to 100 words):
|-
|Funding sources of the project (e.g. NSF grant #, CUNY):
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conference presentations, posters and talks.''
|-
|Number of refereed publications relevant to the project:
|-
|Pending publication relevant to the project:
|-
|<big>'''''For All External (not CUNY) Project Collaborators and Researchers (F,G)'''''</big>
|-
|''Full name as stated at the state/federal ID or EID from other Academic Institution (e.g. John Samuel Doe):''
|-
|''Affiliation outside CUNY if any(e.g. Rutgers University) and valid professional e-mail: ( e.g. John Doe, Rutgers University, jd@rutgers.edu):''
|-
|''Department at NON CUNY Academic Institution (e.g. MIS Rutgers):''
|-
|Non CUNY-email (collaborator/external contact):
|-
|Status of the collaborator (Academic: e.g. Professor; Partner: e.g. NVIDIA lab):
|-
|Status of the external researcher(s) (e.g. principal architect NVIDIA):
|-
|Resources needed (example: Cores 100; Time 10 000 hours Memory per core 8 GB, GPU cores 2 GPU hours 100 Storage 100GB):
|-
|Description of available local resources. Please state NONE if you do not have access to local computational resources. Otherwise provide:
''type of computational (cluster, advanced workstation):''

''- number of nodes:''

''- number of cores per node:''

''-- memory per node:''

''- total number of GPU for server:''

''- number of GPU per node:''

''- type of GPU (list of all types e.e. 2 x K20m, 4 x K80):''
|-
|''Consent that HPCC will be cited properly (see our wiki for details) in all your published work including conferences and talks.''
|-
|''<big>'''For All CUNY Graduate and Undergraduate Classes (E)'''</big>''
!
|-
|''Full name as stated at the CUNY ID card (e.g. John Samuel Doe):''
!
|-
|''CUNY EID and valid CUNY e-mail (e.g. 22341356):''
!
|-
|'''Valid CUNY e-mail.''' Public emails are not accepted (e.g. azho@cix.csi.cuny.edu):
!
|-
|Class ID (e.g. CS 220):
!
|-
|Class Section (e.g. 02):
!
|-
|College (e.g. Baruch College):
!
|-
|Name of the Professor:
!
|-
|Term (e.g. Fall 2025):
!
|}
Upon creation every research user account is provided with 50 GB (and max of 10000 files on /global/u) home directory mounted as /global/u/<userid>. If required, a user may request an increase in the size of their home directory. The HPC Center will endeavor to satisfy reasonable requests. If you expect to have more than 10 000 files please zip several small files into larger single zip file. Please keep only wrangled information in your space in order to optimize use of the existing storage.

Student class accounts ( group d)) are ~~blue:provided~~ with 10 GB home directory. Please note that class accounts and data will be deleted 30 days after the semester ends (unless otherwise agreed upon). Students are responsible for backing up their own data prior to the end of the semester.

When a user account is established, only the user has read/write to his files. The user can change his UNIX permissions to allow others in his group to read/write to his file.

Please be sure to notify the HPC Center if user accounts need to be removed or added to a specific re group. Please read below a policies for accounts. Note that accounts are not timeless and non accessed and non active accounts are removed ( see below).

=== User accounts policies ===
CUNY HPCC applies strict security standards in user accounts management. HPCC uses “account periods”. Account period is '''one year''' for accounts in '''a,c,e, and f''' type of accounts and '''one semester''' for '''b''' and '''d''' type of accounts. All accounts are periodically reviewed and inactive accounts are removed. All student accounts will expire automatically and will be removed after each semester unless student’s advisor request extension of the student’s account. All user accounts for groups '''a,c,e and f''' must be renewed once a year by Sept 30th. All user accounts in groups '''b''' and '''d''' must be renewed within 2 weeks after each semester . Accounts not accessed for one account period and/or not renewed are automatically disabled/locked and will be deleted 60 days from locking. Delete of particular account means unrecoverable remove of all data associated with that account.

===Reset Password ===

Users must use automatic password reset system. Click on [https://hpcauth1.csi.cuny.edu/reset/ Reset Password]. Upon resetting the users will get their individual security token on the e-mail address registered with HPCC.

===Close of account===
If a user would like to close their account, please contact the CUNY at HPCHelp@csi.cuny.edu.
Supervisors that would like to modify the access of their researchers and/or students working for them should contact the HPC to remove, add or modify access.
User accounts that are not accessed or renewed for more than a year and one day will be purged along with any data associated with the account. User accounts that are not renewed on time will be locked and users must contact HPCC to get access recovered.

=== Message of the day (MOTD) ===
Users are encouraged to read the "Message of the day” (MOTD), which is displayed to the user upon logging onto a system. The MOTD provides information on scheduled maintenance time when systems will be unavailable and/or important changes in the environment that are of import to the user community. The MOTD is the HPC Center’s only efficient mechanism for communicating to the broader user community as bulk email messages are often blocked by CUNY SPAM filters.

=== Required citations ===
The CUNY HPC Center appreciates the support it has received from the National Science Foundation (NSF). It is the policy of NSF that researchers who are funded by NSF or who make use of facilities funded by NSF acknowledge the contribution of NSF by including the following citation in their papers and presentations:

'''''This research was supported, in part, under National Science Foundation Grants: CNS-0958379, CNS-0855217, ACI-1126113 and OAC-2215760 (2022) and the City University of New York High Performance Computing Center at the College of Staten Island.'''''

The HPC Center, therefore, requests its users to follow this procedure as it helps the Center to demonstrate that NSF’s investments aided the research and educational missions of the University.

== Reporting requirements ==
The Center reports on its support of the research and educational community to both funding agencies and CUNY on an annual basis. Citations are an important factor that is included in these reports. Therefore, is mandatory users to send copies of research papers developed, in part, using the HPC Center resources to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu]. Accounts for users who violate that requirement may not be renewed. Reporting results obtained with HPC resources also helps the Center to keep abreast of user research requirement directions and needs.

== Funding of computational resources and storage ==
Systems at HPC Center are purchased with grants from National Science Foundation (NSF), grants from NYC, a grant from DASNY and a grant from CUNY's office of the CIO. In addition all systems in condo tier are purchased with direct funds from research groups. The largest financial support comes from '''NSF MRI grants (more than 80% of all funding).''' CUNY own investment constitute '''8.6%''' of all funds. Here is the list of all grants for CUNY-HPCC.

:'''PFSS and GPU Nodes:''' NSF Grant OAC-2215760 (operational)
:'''DSMS''' NSF Grant ACI-1126113 (server is partially retired)
:'''BLUE MOON''', Grant NYC 042-ST030-015 (operational)
:'''CRYO,''' Grant DASNY 208684-000 OP (operational)
:'''ANDY''', NSF Grant CNS-0855217 and the New York City Council through the efforts of Borough President James Oddo ( server is fully retired)
:'''APPEL''', New York State Regional Economic Development Grant through the efforts of State Senator Diane Savino (operational)
:'''PENZIAS''', The Office of the CUNY Chief Information Officer ( Server is partially retired)
:'''SALK''', NSF Grant CNS-0958379 and a New York State Regional Economic Development Grant (Server is fully retired)