Main Page

The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The CUNY-HPCC supports computational research and computational intensive courses offered at all CUNY colleges in fields such as Computer Science, Engineering, Bioinformatics, Chemistry, Materials Science, Genetics, Computational Biology and others. HPCC provides educational outreach to local schools and supports undergraduates who work in the research programs of the host institution (REU program from NSF). The primary mission of HPCC is:

• To enable advanced research and scholarship at CUNY colleges by providing faculty, staff, and students with access to high-performance computing, adequate storage resources and visualization resources;
• To provide CUNY faculty and their collaborators at other universities, CUNY research staff and CUNY graduate and undergraduate students with expertise in scientific computing, parallel scientific computing (HPC), software development, advanced data analytics, data driven science and simulation science, visualization, advanced database engineering, and others.
• Leverage the HPC Center capabilities to acquire additional research resources for CUNY faculty, researchers and students in existing and major new programs.
• Create opportunities for the CUNY research community to win grants from national funding institutions and to develop new partnerships with the government and private sectors.

Research Computing Infrastructure

The research computing infrastructure is depicted in the figure below. In order to support various types of research projects the CUNY-HPCC support variety of computational architectures. All computational resources are organized in 3 tiers - Condominium Tier, Basic Tier and Advanced Tier. The computational architectures participating in each tier are discussed below. All tiers access 2 separate file systems: 1) Global Storage (GS) (a global file system) and 2) /scratch file system. The GS is mounted only on login nodes and is proposed to keep user data (home directories) and project data (project directories). The /scratch file system is mounted on login nodes and on all computational nodes (in all tiers). Thus GS holds long lasting user data (executables, scripts and data), while /scratch is used to hold provisional data required by particular simulation(s). Consequently, jobs can be started only from /scratch and never from GS storage. It is important to remember that /scratch is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:

1. data in /scratch are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in /scratch.
2. /scratch undergoes regular file purging when either or both conditions are satisfied:
   1. load of the /scratch file system reaches 71%.
   2. there is/are inactive file(s) older than 60 days.
3. only data in GS are protected and recoverable.

Upon registering with HPCC every user will get 2 directories:

• /scratch/<userid> – this is temporary workspace on the HPC systems

• /global/u/<userid> – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data

• In some instances a user will also have use of disk space on the DSMS in /cunyZone/home/<projectid> (IRods).

The /global/u/<userid> directory has quota (see below for details) while the /scratch/<userid> do not have. However the /scratch space is cleaned up following the rules described below. There are no guarantees of any kind that files in /scratch will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called 'chizen. The Data Transfer Node called Cea allows file transfer from/to remote sites directly to/from /global/u/<userid> or to/from /scratch/<userid>

Condominium Tier

Condominium tier organizes resources purchased and owned by faculty, but maintained by HPCC. The participation in this tier is voluntary. Several faculty/research groups can combine finds to purchase and consequently share the hardware (node). All nodes in this tier must meet certain hardware specifications including to be fully warranted for time life of the node(s) in order to be accepted. If you want to participate in condominium please sent a request mail to hpchelp@csi.cuny.edu and consult HPCC before making a purchase. Condominium tier:

• Promotes vertical and horizontal collaboration between research groups;
• Makes possible to utilize small amounts of research money or "left-over" money wisely and to obtain advanced resources;
• Helps researchers to conduct large scope high quality research including collaborative projects leading to successful grants with high impact.

The resources in this tier are organized as sub-cluster and all jobs are submitted via main master head node (MHN) but use separate partition with different QOS settings. The condo tier members benefit from professional support from HPCC security and maintenance. Upon approval (from the node owner) any idle node(s) can be used by any other member(s). For instance a member can borrow (for agreed time) node with more advanced GPUs than those installed on his/her own node(s). The owners of the equipment are responsible for any repair costs for their node(s).

In sum the benefits of condo are:

• 5 year lifecycle - condo resources will be available for a duration of 5 years.
• Access to more cpu cores than purchased and access to resources which are not purchased.
• Support - HPCC staff will install, upgrade,secure and maintain condo hardware throughout its lifecycle.

HPC systems and their architectures

The HPC Center operates variety of architectures in order to support complex and demanding workflows. The deployed systems include: distributed memory (also referred to as “cluster”) computers, symmetric multiprocessor (also referred as SMP) and shared memory (also reffred as NUMA machines).

Computational Systems:

SMP servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU. Currently, HPCC operates several detached SMP servers named Math, Cryo and Karle. Karle is a server which does not have GPU and is used for visualizations, visual analytics and interactive MATLAB/Mathematica jobs. Math is a condominium server without GPU as well. Cryo (CPU+GPU server) is specialized server with eight (8) NVIDIA V100 (32G) GPU designed to support large scale multi-core multi-GPU jobs.

Cluster is defined as a single system comprizing a set of SMP servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. Each SMP member of the cluster is called a node. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU. The main cluster at HPCC is a hybrid (CPU+GPU) cluster called Penzias. Sixty six (66) of Penzias nodes have 2 x GPU K20m, and the 3 fat nodes (nodes with large number of CPU-cores and memory) of the cluster do not have GPU. In addition HPCC operates the cluster Herbert dedicated only to education.

Distributed shared memory computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the NUMA systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the NUMA server called Appel. This server does not have GPU.

Infrastructure systems:

o Master Head Node (MHN) is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus.

o Chizen is a redundant gateway server which provides access to protected HPCC domain.

o Cea is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. Cea is accessible directly (not only via Chizen), but allows only limited set of shell commands.

Table 1 below provides a quick summary of the attributes of each of the systems available at the HPC Center.

Master Head Node	System	Type	Type of Jobs	Nodes	CPU Cores	GPUs	Mem/node	Mem/core	Chip Type	GPU Type
MHN	Penzias	Hybrid Cluster	Sequential & Parallel jobs w/wo GPU	66	16	2	64 GB	4 GB	SB, EP 2.20 GHz	K20m GPU, PCIe
			Sequential & Parallel jobs	1	24	-	1500 GB	62 GB	HL, 2.30 GHz	-
					36	-	768 GB	21 GB		-
					24	-	768 GB	32 GB		-
	Appel	NUMA	Massive Parallel, sequential	1	384	-	11 TB	28 GB	IB, 3 GHz	-
	Cryo	SMP	Sequential and Parallel jobs, with GPU	1	40	8	1500 GB	37 GB	SL, 2.40 GHz	V100 (32GB) GPU, XSM
	Blue Moon	Hybrid Cluster	Sequential and Parallel jobs w/wo GPU	24	32	-	192 GB	6 GB	SL, 2.10 GHz	-
				2	32	2				V100(16GB) GPU, PCIe
	Karle	SMP	Visualization, MATLAB/Mathematica	1	36*	-	768 GB	21 GB	HL, 2.30 GHz	-
	Chizen	Gateway	2	-						-

Note: SB = Sandy Bridge, HL = Haswell, IB = Ivy Bridge, SL = Skylake

Partitions and jobs

The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in partitions. The main partition is called production. This is routing partition which distributes the jobs in several sub-partitions depend on job’s requirements. Thus the serial job submitted in production will land in partsequential partition. No SLURM Pro scripts should be ever used and all existing SLURM scripts must be converted to SLURM before use. The table below shows the limitations of the partitions.

Partition	Max cores/job	Max jobs/user	Total cores/group	Time limits
production	128	50	256	240 Hours
partedu	16	2	216	72 Hours
partmath	128	128	128	240 Hours
partmatlab	1972	50	1972	240 Hours
partdev	16	16	16	4 Hours

o production is the main partition with assigned resources across all servers (except Math and Cryo).It is routing partition so the actual job(s) will be placed in proper sub-partition automatically. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.

o partedu partition is only for education. Assigned resources are on educational server Herbert. Partedu is accessible only to students (graduate and/or undergraduate) and their professors who are registered for a class supported by HPCC. Access to this partition is limited by the duration of the class.

o partmatlab partition allows to run MATLAB's Distributes Parallel Server across main cluster. Note however that parallel toolbox programs can be submitted via production partition, but only as thread parallel jobs.

o partdev is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

Hours of Operation

The second and fourth Tuesday mornings in the month from 8:00AM to 12PM are normally reserved (but not always used) for scheduled maintenance. Please plan accordingly.
Unplanned maintenance to remedy system related problems may be scheduled as needed. Reasonable attempts will be made to inform users running on those systems when these needs arise.

User Support

Users are encouraged to read this Wiki carefully. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPC Center systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications and run scripts among them. Still, there are some differences, particularly with the SGI (ANDY) and Cray (SALK) systems.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY community in parallel programming techniques, HPC computing architecture, and the essentials of using our systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

 hpchelp@csi.cuny.edu 

Warnings and modes of operation

1. hpchelp@csi.cuny.edu is for questions and accounts help communication only and does not accept tickets. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address. Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. Do not send questions to individual CUNY HPC Center staff members directly. These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared to other HPC Centers of similar size our staff is extremely lean. Please make full use of the tools that we have provided (especially the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using our systems will be predictably good and productive.

User Manual

An old version of the user manual can be downloaded at: http://cunyhpc.csi.cuny.edu/publications/User_Manual.pdf. Note that this manual provides SLURM batch scripts as examples. Currently CUNY-HPCC uses SLURM so users must check the brief SLURM manual distributed with new accounts or ask CUNY-HPCC for a copy of the latter.