Main Page: Difference between revisions

From HPCC Wiki
Jump to navigation Jump to search
 
(26 intermediate revisions by the same user not shown)
Line 5: Line 5:
[[Image:hpcc-panorama3.png]]
[[Image:hpcc-panorama3.png]]


The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314.  The CUNY-HPCC supports computational research and computational intensive courses offered at all CUNY colleges in fields such as Computer Science, Engineering, Bioinformatics, Chemistry, Physics,Materials Science, Genetics, Computational Biology, Finance and others.  HPCC  provides educational outreach to local schools and supports undergraduates who work in the research programs of the host institution (REU program from NSF). The primary mission of HPCC is:  
The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314.  The CUNY-HPCC supports computational research and computational intensive courses on graduate and undergraduate level offered at all CUNY colleges in fields such as Computer Science, Engineering, Bioinformatics, Chemistry, Physics,Materials Science, Genetics, Computational Biology, Finance and others.  HPCC  provides educational outreach to local schools and supports undergraduates who work in the research programs of the host institution (e.g. REU program from NSF). The primary mission of HPCC is:  


* To enable advanced research and scholarship at CUNY colleges by providing faculty, staff, and students with access to high-performance computing, adequate storage resources and visualization resources;
* To enable advanced research and scholarship at CUNY colleges by providing faculty, staff, and students with access to high-performance computing, adequate storage resources and visualization resources;
* To enable advanced education and cross disciplinary  education by providing flexible and scalable resources;
* To provide CUNY faculty and their collaborators at other universities, CUNY research staff and CUNY graduate and undergraduate students with expertise in scientific computing, parallel scientific computing (HPC), software development, advanced data analytics, data driven science and simulation science, visualization, advanced database engineering, and others.  
* To provide CUNY faculty and their collaborators at other universities, CUNY research staff and CUNY graduate and undergraduate students with expertise in scientific computing, parallel scientific computing (HPC), software development, advanced data analytics, data driven science and simulation science, visualization, advanced database engineering, and others.  
* Leverage the HPC Center capabilities to acquire additional research resources for CUNY faculty, researchers and students in existing and major new programs.
* Leverage the HPC Center capabilities to acquire additional research resources for CUNY faculty, researchers and students in existing and major new programs.
* Create opportunities for the CUNY research community to win grants from national funding institutions and to develop new partnerships with the government and private sectors.
* Create opportunities for the CUNY research community to win grants from national funding institutions and to develop new partnerships with the government and private sectors.
CUNY-HPPC is voting member of '''Coalition for Academic Scientific Computation (CASC)'''. Originally formed in the 1990s as a small group of the heads of national supercomputing centers, CASC expanded to more than 100 member institutions representing many of the nation’s most forward-thinking universities and computing centers. CASC includes the leadership of large academic computing centers such as TACC or San Diego SC and recently attracts a greater diversity of smaller institutions such as non-R1s, HBCUs, HSIs, TCUs, etc. CASC’s mission as to be  “''dedicated to advocating for the use of the most advanced computing technology to accelerate scientific discovery for national competitiveness, global security, and economic success, as well as develop a diverse and well-prepared 21st century workforce.”''
== CUNY-HPCC - Democratization of Research ==
In last few years the model of cloud computing (called also computing-on-demand) made the promise that anyone, no matter where the user is, could leverage almost unlimited computing resources. This computing supposed to “democratize” research and level the playing field, as it were. Unfortunately that is not entirely true (for now) because the cloud computing even available to nearly anyone, from nearly anywhere remains comparatively expensive to local resources and lacks the flexibility and accessibility of local support tailored to education and research offered by the local research HPCC. Indeed, every computational environment has limitations and a learning curve such that students and faculty coming from variety of  backgrounds and having might feel crushed and helpless without  close and personalized local support. in this sense the carefully designed, user centered, academically focused  CUNY-HPCC has the transformative capability for rapidly evolving computation and data-driven research, and creates opportunities for vast collaboration and convergence research activities and thus provides the real democratization of the research. 
== Pedagogical value of CUNY- HPCC ==
CUNY-HPPC supports whole variety of  classes on graduate and undergraduate level from all CUNY-colleges, Graduate Center and institutes. It is important to mention that CUNY-HPCC impact goes beyond the STEM disciplines. Thus the CUNY-HPCC:
[[File:NAnoBio6.jpg|right|frameless|Dr Alexander Tzanov, the director of CUNY-HPCC speaks on NanoBioNYC workshop]]
* '''Allows to conduct analysis of datasets that are too large to work with easily on personal devices, or that cannot easily be shared or disseminated.''' These data sets are not coming only from STEM fields (i.e. finance, economics, linguistics etc.). Facilitating these analyses provides the students with opportunity to interact in real time with  increasingly large amounts of data, enabling them to gather important skills and experiences.
* '''CUNY-HPCC provides collaborative space for entire courses. T'''he multi-user capabilities and environment of HPCC facilitates collaborative work among learners and promotes more complex closer to reality learning problems.
* Large computational and visualization capabilities of '''HPCC is enabler for applying analytical techniques too large for personal devices.''' Students can  run unattended parameter sweeps or workflows in order to explore the problem in detail. That self exploration has proven positive effect on learning
* '''Use of CUNY-HPCC resources provides students with needed pre request skills and knowledge''' they may need later when explore larger HPC environments. For instance the CUNY-HPCC workflow and environment is extremely close to the environment of other research centers of ACCESS resources.
* '''CUNY-HPCC participates in educational programs such as NSF funded NanoBioNYC Ph.D. traineeship program at  CUNY.''' This program is focused on developing groundbreaking bio-nanoscience solutions to address urgent human and planetary health issues and preparing students to become tomorrow’s leaders in diverse STEM careers.


==Research Computing Infrastructure==
==Research Computing Infrastructure==
[[File:HPCC structure last.png|thumb|741x741px|                                                                                                                                    '''<big>                                                          The Organization of HPCC resources</big>''']]
[[File:HPCC structure last.png|thumb|682x682px|                                                                                                                                    '''<big>                                                          The Organization of HPCC resources</big>''']]
The research computing infrastructure is depicted in the figure below. In order to support various types of research projects the CUNY-HPCC support variety of computational architectures.  All computational resources are organized in '''3''' tiers - '''''Condominium Tier, Basic Tier''''' and '''''Advanced Tier.''''' In addition the HPCC operates the equipment purchased with '''NSF MRI''' grant. The computational architectures participating in each tier are discussed below. All tiers access 2 separate file systems: 1) Global Storage (GS)  (a global file system) and 2) '''/scratch f'''ile system. The '''GS''' is mounted only on login nodes and is proposed to keep user data (home directories) and project data (project directories). The '''/scratch''' file system is mounted on login nodes and on all computational nodes (in all tiers). Thus '''GS''' holds long lasting user data (executables, scripts and data), while '''/scratch i'''s used to hold provisional data required by particular simulation(s).  Consequently, jobs can be started only from '''/scratch''' and '''never from'''  '''GS''' storage.  It is important to remember that '''/scratch''' is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:   
The research computing infrastructure is depicted in the figure on right. In order to support various types of research projects the CUNY-HPCC support variety of computational architectures.  All computational resources are organized in '''3''' tiers - '''''Condominium Tier (CT), Free Tier''''' (FT) and '''''Advanced Tier (AT).'''''  Servers in FT and AT are attached to 2 separate file systems '''/scratch''' and  Global Storage ('''GS''') - '''/global/u/''', while the Arrow and servers in CT are attached to independent hybrid parallel file system ('''HPFS - not shown'''). Servers in CT and Arrow the /scratch and /global/u are placed on different tiers of the same '''HPFS'''.  Currently, only data on basic and advanced tier (using GS)  are protected by backup. The '''HPFS''' (serving condo tier) is '''not connected to tape backup silo''', due to lack of funding.
 
=== Storage systems ===
The HPCC uses 3 file systems. The scratch file system is small fast file system mounted on all nodes from basic and advanced tier. The files on that file system are '''not backup ed and are not protected.''' The file system does not have quota to allow to submit large jobs but it will be automatically purged if the load of the file system exceeds 70% or file(s) are not accessed for 60 days whatever comes first. The GS holds user home directories and is backup ed once per 24 hours. That is why is mandatory for users to keep all their valuable data in GS not in scratch.
 
The HPFS is the new hybrid file system with capacity of 2.1 PB obtained with NSF grant OAC-2215760. That file system is mounted on all condo servers and also on 2 nodes obtained from the same NSF grant. This file system combines scratch and home directories and is '''not backup ed.'''
 
==== Run jobs on HPCC resources ====
Despite of a tier and file system used all jobs at HPCC must:
 
# Start from user's directory on '''scratch''' file system '''- /scratch/<userid>''' . Jobs cannot be started from users home directories - '''/global/u/<userid>'''
# Use SLURM job submission system (job scheduler) '''.''' All jobs submission scripts written for other job scheduler(s) (i.e. PBS pro) must be converted to SLURM syntax.
 
All usefull users' data must be kept in user's home directory  on '''/global/u/<userid>.'''  For CT servers and for Arrow the user's home directories reside on '''HPFSS''' while for servers in AT and FT the data will be stored on older '''GS.'''  Note that '''GS and HPFS''' file systems <u>are not connected</u>. Data on any '''/scratch''' file system can be purged at any time nor are protected by tape backup.  The '''/scratch''' file system is mounted on login nodes and on all computational nodes. Consequently, jobs can be started only from '''/scratch''' directory and '''never from'''  '''/global/u/<userid> directory'''.  It is important to remember that '''/scratch''' is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:
#data in '''/scratch''' are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in '''/scratch.'''
#'''/scratch''' undergoes regular and automatic file purging when either or both conditions are satisfied:
##load of the '''/scratch''' file system reaches '''71%.'''
##there is/are inactive file(s) older than 60 days.
#only data in '''/global/u/<user_id>'''  are protected and recoverable.
#data in HPFS are nor recoverable both in /scratch and /global/u/<user_id>
 
Upon  registering with HPCC every user will get 2 directories:
 
:• '''<font face="courier">/scratch/<font color="red"><userid></font></font>''' – this is temporary workspace on the HPC systems
:• '''<font face="courier">/global/u/<font color="red"><userid></font></font>''' – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data
:• In some instances a user will also have use of disk space on the DSMS in '''<font face="courier">/cunyZone/home/<font color="red"><projectid></font></font>''' (IRods).
The '''/global/u/<userid>''' directory has quota (see below for details) while  the '''/scratch/<userid>''' do not have. However the '''/scratch''' space is cleaned up  following the rules described below. There are no guarantees of any kind that files in '''/scratch'''  will be preserved during the hardware crashes or cleaning up.  Access to all HPCC resources is provided by bastion host called '<nowiki/>'''''chizen'''''. The Data Transfer Node called '''Cea''' allows file transfer from/to remote sites directly to/from '''<font face="courier">/global/u/<font color="red"><userid></font></font>''' or to/from  '''<font face="courier">/scratch/<font color="red"><userid></font></font>'''
{| class="wikitable sortable"
|+
!Server
!Tier
!Scratch
!Data Storage
!Number of nodes
!Cores per node
!Memory per node
!GPU per node
!Total cores/cluster
!Total GPU per cluster
!Interconnect
|-
|Penzias
|FT
|Lustre FS
|GPFS FS (GS)
|60
|16
|64 GB
|2 x K20m/5GB PCI 2
|960
|120
|IB - 40 Gbps
|-
|Blue Moon
|AT
|Lustre FS
|GPFS FS (GS)
|24
|32
|192 GB
|2 x V100/16 GB PCI 3
|832
|4
|IB - 100 Gbps
|-
|Karle
|AT
|Lustre FS
|GPFS FS (GS)
|1
|36
|768 GB
|none
|36 (72 hyperthread)
|none
|IB - 40 Gbps
|-
|Appel
|AT
|Lustre FS
|GPFS FS (GS)
|1
|384
|11 TB
|none
|384
|none
|IB - 40 Gbps
|-
|Cryo
|AT
|Lustre FS
|GPFS FS (GS)
|1
|40
|1,5 TB
|8 x V100/32GB SXM
|40
|8
|IB - 40 Gbps
|-
|Arrow
|AT
|HPFS NVMe
|HPFS spin
|2
|128
|512 GB
|8 X A100/80GB
|128
|16
|IB - 200 Gbps
|-
|Math
|CT
|HPFS NVMe
|HPFS spin
|1
|128
|512 GB
|2 X A40/48GB
|128
|2
|B - 200 Gbps
|-
|Phys
|CT
|HPFS NVMe
|HPFS spin
|1
|64
|512 GB
|none
|64
|none
|B - 200 Gbps
|-
|Chem
|CT
|HPFS NVMe
|HPFS spin
|2
|64
|512 GB
|2 x A30/48GB PCI 3 + 8 x A100/40GB SXM
|192
|10
|B - 200 Gbps
|-
|ASRC
|CT
|HPFS NVMe
|HPFS spin
|1
|64
|512 GB
|2 x A30/24GB
|64
|2
|B - 200 Gbps
|-
|Eng
|CT
|HPFS NVMe
|HPFS spin
|1
|128
|512 GB
|1 x A40/48GB
|128
|1
|B - 200 Gbps
|}
The computational architectures participating in each tier are discussed below. All tiers access 2 separate file systems: 1) Global Storage (GS)  (a global file system) and 2) '''/scratch f'''ile system. The '''GS''' is mounted only on login nodes and is proposed to keep user data (home directories) and project data (project directories). The '''/scratch''' file system is mounted on login nodes and on all computational nodes (in all tiers). Thus '''GS''' holds long lasting user data (executables, scripts and data), while '''/scratch i'''s used to hold provisional data required by particular simulation(s).  Consequently, jobs can be started only from '''/scratch''' and '''never from'''  '''GS''' storage.  It is important to remember that '''/scratch''' is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:   


#data in '''/scratch''' are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in '''/scratch.'''
#data in '''/scratch''' are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in '''/scratch.'''
Line 30: Line 217:


===<u>Condominium  Tier</u>===
===<u>Condominium  Tier</u>===
Condominium tier organizes resources purchased and owned by faculty, but maintained by HPCC. The participation in this tier is '''strictly voluntary'''. Several faculty/research groups can combine finds to purchase and consequently share the hardware (a node or several nodes). All nodes in this tier must meet certain hardware specifications including to be fully warranted for time life of the node(s) in order to be accepted. If you want to participate in condominium please sent a request mail to hpchelp@csi.cuny.edu and consult HPCC before making a purchase.  Condominium tier:  
Condominium tier (called '''condo''') organizes resources purchased and owned by faculty, but maintained by HPCC. The participation in this tier is '''strictly voluntary'''. Several faculty/research groups can combine finds to purchase and consequently share the hardware (a node or several nodes). All nodes in this tier must meet certain hardware specifications including to be fully warranted for time life of the node(s) in order to be accepted. If you want to participate in condominium please sent a request mail to hpchelp@csi.cuny.edu and consult HPCC before making a purchase.  Condominium tier:  


*Promotes vertical and horizontal collaboration between research groups;
*Promotes vertical and horizontal collaboration between research groups;
Line 36: Line 223:
*Helps researchers to conduct large scope high quality research including collaborative projects leading to successful grants with high impact.
*Helps researchers to conduct large scope high quality research including collaborative projects leading to successful grants with high impact.


The resources in this tier are organized as '''sub-cluster''' (Condo) and all jobs are submitted via main master head node (MHN) but use separate partitions with different QOS settings. The condo tier members benefit from professional support from HPCC security and maintenance. Upon approval (from the node owner)  any idle node(s) can be used by any other member(s). For instance a member can borrow (for agreed time) node with more advanced GPUs than those installed on his/her own node(s). The owners of the equipment are responsible for any repair costs for their node(s). Other users may rent any of the described below condo resources if agreed with owners. The unused cycles can be shared with other members of the community.        
==== Access to condo resources ====
The resources are available only for condo owners and their groups. The users registered with condo must use the main login node of Arrow server. To access their own node the condo users must  specify their own private partition. In addition there are partitions which operate over two or more nodes owned by condo members.  Condo users may us  (private partition) to access their node. he condo tier members benefit from professional support from HPCC security and maintenance. Upon approval (from the node owner)  any idle node(s) can be used by any other member(s). For instance a member can borrow (for agreed time) node with more advanced GPUs than those installed on his/her own node(s). The owners of the equipment are responsible for any repair costs for their node(s). Other users may rent any of the described below condo resources if agreed with owners. The unused cycles can be shared with other members of the community.


In sum the benefits of condo are:     
In sum the benefits of condo are:     
Line 50: Line 238:
* '''To include in their research and instrumentation grants money''' for computing used to cover operational expences (non-tax-levy expenses) of the HPCC.
* '''To include in their research and instrumentation grants money''' for computing used to cover operational expences (non-tax-levy expenses) of the HPCC.
The table below summarized the available resources
The table below summarized the available resources
{| class="wikitable"
{| class="wikitable sortable"
|+Resources in Condominium Tier (Condo cluster):
|+Resources in Condominium Tier (Arrow cluster)
384 cores and 8 Ampere class GPU
!Number of nodes
!Number of nodes
!Cores/node
!Cores/node
Line 60: Line 247:
!Interconnect
!Interconnect
!Use
!Use
!Private partition
|-
|-
|2
|2
Line 65: Line 253:
| 2 x AMD EPYC
| 2 x AMD EPYC
|256 GB
|256 GB
| 2 x A30
| 2 x A30 24 GB, PCIe gen 3
|100 Gbps Infiniband EDR
|100 Gbps Infiniband EDR
|Number Crunching
|Number Crunching
|parchem, partasrc
|-
|-
|1
|1
Line 76: Line 265:
|100 Gbps infiniband EDR
|100 Gbps infiniband EDR
|Number Crunching  
|Number Crunching  
|parthphys
|-
|-
|2
|2
Line 81: Line 271:
| 2 x AMD EPYC
| 2 x AMD EPYC
|512 GB
|512 GB
| 2 x A40
| 2 x A40 48 GB, PCIe gen 3
|100 Gbps Infiniband EDR
|100 Gbps Infiniband EDR
|Number Crunching
|Number Crunching
|parting, partmath
|-
|1
|128
|2 x AMD EPYC
|512 GB
|8 x A100 40GB, SXM
|200 Gbps Infiniband HDR
|Number Crunching
|parched
|}
|}


==='''<u>Advanced Tier</u>''' ===
==='''<u>Advanced Tier</u>''' ===
The advanced tier holds the resources used for more advanced or large scale research. This tier provides nodes with Volta class GPUs with 16 GB and 32 GB on board. The table below summarizes the resources.  
The advanced tier holds the resources used for more advanced or large scale research. This tier provides nodes with Volta class GPUs with 16 GB and 32 GB on board. The table below summarizes the resources.  
{| class="wikitable"
{| class="wikitable sortable"
|+Resources in Advanced Tier (Blue Moon, Cryo, Appel):  
|+Resources in Advanced Tier (Blue Moon, Cryo, Appel):  
1256 cores and 12 Volta class GPU
1256 cores and 12 Volta class GPU
Line 104: Line 304:
|2 x Intel X86_64
|2 x Intel X86_64
|192 GB
|192 GB
|2 x V100 (16)
|2 x V100 (16 GB) PCIe gen 3
|100 Gbps Infiniband EDR
|100 Gbps Infiniband EDR
|Number Crunching
|Number Crunching
Line 122: Line 322:
|2 x Intel X86_64
|2 x Intel X86_64
| 1500 GB
| 1500 GB
|8 x V100 (32)
|8 x V100 (32 GB) SXM
|100 Gbps Infiniband EDR
|100 Gbps Infiniband EDR
|Number Crunching
|Number Crunching
Line 154: Line 354:
|2 x Intel X86_64
|2 x Intel X86_64
|64 GB
|64 GB
| 2 x K20m
| 2 x K20m, PCIe.2
|56 Gbps Infiniband
|56 Gbps Infiniband
| Number crunching, Sequential jobs, General computing, Distributed Parallel Matlab
| Number crunching, Sequential jobs, General computing, Distributed Parallel Matlab
Line 160: Line 360:
|-
|-
| 1
| 1
|72*
|36/72*
|2 x Intel X86_64
|2 x Intel X86_64
|768 GB
|768 GB
Line 189: Line 389:
|2 x AMD EPYC
|2 x AMD EPYC
|HDR 100 Gbps
|HDR 100 Gbps
|Molecular modeling, Data science, Number Crunching, Materials Science
|Molecular modeling, Data science, Number Crunching, Materials Science, AI, ML
|Arrow
|Arrow
|}
|}

Latest revision as of 17:31, 28 December 2023

CUNY-HPCC-HEADER-LOGO.jpg

Hpcc-panorama3.png

The City University of New York (CUNY) High Performance Computing Center (HPCC) is located on the campus of the College of Staten Island, 2800 Victory Boulevard, Staten Island, New York 10314. The CUNY-HPCC supports computational research and computational intensive courses on graduate and undergraduate level offered at all CUNY colleges in fields such as Computer Science, Engineering, Bioinformatics, Chemistry, Physics,Materials Science, Genetics, Computational Biology, Finance and others. HPCC provides educational outreach to local schools and supports undergraduates who work in the research programs of the host institution (e.g. REU program from NSF). The primary mission of HPCC is:

  • To enable advanced research and scholarship at CUNY colleges by providing faculty, staff, and students with access to high-performance computing, adequate storage resources and visualization resources;
  • To enable advanced education and cross disciplinary education by providing flexible and scalable resources;
  • To provide CUNY faculty and their collaborators at other universities, CUNY research staff and CUNY graduate and undergraduate students with expertise in scientific computing, parallel scientific computing (HPC), software development, advanced data analytics, data driven science and simulation science, visualization, advanced database engineering, and others.
  • Leverage the HPC Center capabilities to acquire additional research resources for CUNY faculty, researchers and students in existing and major new programs.
  • Create opportunities for the CUNY research community to win grants from national funding institutions and to develop new partnerships with the government and private sectors.

CUNY-HPPC is voting member of Coalition for Academic Scientific Computation (CASC). Originally formed in the 1990s as a small group of the heads of national supercomputing centers, CASC expanded to more than 100 member institutions representing many of the nation’s most forward-thinking universities and computing centers. CASC includes the leadership of large academic computing centers such as TACC or San Diego SC and recently attracts a greater diversity of smaller institutions such as non-R1s, HBCUs, HSIs, TCUs, etc. CASC’s mission as to be “dedicated to advocating for the use of the most advanced computing technology to accelerate scientific discovery for national competitiveness, global security, and economic success, as well as develop a diverse and well-prepared 21st century workforce.”

CUNY-HPCC - Democratization of Research

In last few years the model of cloud computing (called also computing-on-demand) made the promise that anyone, no matter where the user is, could leverage almost unlimited computing resources. This computing supposed to “democratize” research and level the playing field, as it were. Unfortunately that is not entirely true (for now) because the cloud computing even available to nearly anyone, from nearly anywhere remains comparatively expensive to local resources and lacks the flexibility and accessibility of local support tailored to education and research offered by the local research HPCC. Indeed, every computational environment has limitations and a learning curve such that students and faculty coming from variety of backgrounds and having might feel crushed and helpless without close and personalized local support. in this sense the carefully designed, user centered, academically focused CUNY-HPCC has the transformative capability for rapidly evolving computation and data-driven research, and creates opportunities for vast collaboration and convergence research activities and thus provides the real democratization of the research.

Pedagogical value of CUNY- HPCC

CUNY-HPPC supports whole variety of classes on graduate and undergraduate level from all CUNY-colleges, Graduate Center and institutes. It is important to mention that CUNY-HPCC impact goes beyond the STEM disciplines. Thus the CUNY-HPCC:

Dr Alexander Tzanov, the director of CUNY-HPCC speaks on NanoBioNYC workshop
  • Allows to conduct analysis of datasets that are too large to work with easily on personal devices, or that cannot easily be shared or disseminated. These data sets are not coming only from STEM fields (i.e. finance, economics, linguistics etc.). Facilitating these analyses provides the students with opportunity to interact in real time with increasingly large amounts of data, enabling them to gather important skills and experiences.
  • CUNY-HPCC provides collaborative space for entire courses. The multi-user capabilities and environment of HPCC facilitates collaborative work among learners and promotes more complex closer to reality learning problems.
  • Large computational and visualization capabilities of HPCC is enabler for applying analytical techniques too large for personal devices. Students can run unattended parameter sweeps or workflows in order to explore the problem in detail. That self exploration has proven positive effect on learning
  • Use of CUNY-HPCC resources provides students with needed pre request skills and knowledge they may need later when explore larger HPC environments. For instance the CUNY-HPCC workflow and environment is extremely close to the environment of other research centers of ACCESS resources.
  • CUNY-HPCC participates in educational programs such as NSF funded NanoBioNYC Ph.D. traineeship program at CUNY. This program is focused on developing groundbreaking bio-nanoscience solutions to address urgent human and planetary health issues and preparing students to become tomorrow’s leaders in diverse STEM careers.

Research Computing Infrastructure

The Organization of HPCC resources

The research computing infrastructure is depicted in the figure on right. In order to support various types of research projects the CUNY-HPCC support variety of computational architectures. All computational resources are organized in 3 tiers - Condominium Tier (CT), Free Tier (FT) and Advanced Tier (AT). Servers in FT and AT are attached to 2 separate file systems /scratch and  Global Storage (GS) - /global/u/, while the Arrow and servers in CT are attached to independent hybrid parallel file system (HPFS - not shown). Servers in CT and Arrow the /scratch and /global/u are placed on different tiers of the same HPFS. Currently, only data on basic and advanced tier (using GS) are protected by backup. The HPFS (serving condo tier) is not connected to tape backup silo, due to lack of funding.

Storage systems

The HPCC uses 3 file systems. The scratch file system is small fast file system mounted on all nodes from basic and advanced tier. The files on that file system are not backup ed and are not protected. The file system does not have quota to allow to submit large jobs but it will be automatically purged if the load of the file system exceeds 70% or file(s) are not accessed for 60 days whatever comes first. The GS holds user home directories and is backup ed once per 24 hours. That is why is mandatory for users to keep all their valuable data in GS not in scratch.

The HPFS is the new hybrid file system with capacity of 2.1 PB obtained with NSF grant OAC-2215760. That file system is mounted on all condo servers and also on 2 nodes obtained from the same NSF grant. This file system combines scratch and home directories and is not backup ed.

Run jobs on HPCC resources

Despite of a tier and file system used all jobs at HPCC must:

  1. Start from user's directory on scratch file system - /scratch/<userid> . Jobs cannot be started from users home directories - /global/u/<userid>
  2. Use SLURM job submission system (job scheduler) . All jobs submission scripts written for other job scheduler(s) (i.e. PBS pro) must be converted to SLURM syntax.

All usefull users' data must be kept in user's home directory  on /global/u/<userid>. For CT servers and for Arrow the user's home directories reside on HPFSS while for servers in AT and FT the data will be stored on older GS. Note that GS and HPFS file systems are not connected. Data on any /scratch file system can be purged at any time nor are protected by tape backup. The /scratch file system is mounted on login nodes and on all computational nodes. Consequently, jobs can be started only from /scratch directory and never from /global/u/<userid> directory. It is important to remember that /scratch is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:

  1. data in /scratch are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in /scratch.
  2. /scratch undergoes regular and automatic file purging when either or both conditions are satisfied:
    1. load of the /scratch file system reaches 71%.
    2. there is/are inactive file(s) older than 60 days.
  3. only data in /global/u/<user_id> are protected and recoverable.
  4. data in HPFS are nor recoverable both in /scratch and /global/u/<user_id>

Upon registering with HPCC every user will get 2 directories:

/scratch/<userid> – this is temporary workspace on the HPC systems
/global/u/<userid> – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data
• In some instances a user will also have use of disk space on the DSMS in /cunyZone/home/<projectid> (IRods).

The /global/u/<userid> directory has quota (see below for details) while the /scratch/<userid> do not have. However the /scratch space is cleaned up following the rules described below. There are no guarantees of any kind that files in /scratch will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called 'chizen. The Data Transfer Node called Cea allows file transfer from/to remote sites directly to/from /global/u/<userid> or to/from /scratch/<userid>

Server Tier Scratch Data Storage Number of nodes Cores per node Memory per node GPU per node Total cores/cluster Total GPU per cluster Interconnect
Penzias FT Lustre FS GPFS FS (GS) 60 16 64 GB 2 x K20m/5GB PCI 2 960 120 IB - 40 Gbps
Blue Moon AT Lustre FS GPFS FS (GS) 24 32 192 GB 2 x V100/16 GB PCI 3 832 4 IB - 100 Gbps
Karle AT Lustre FS GPFS FS (GS) 1 36 768 GB none 36 (72 hyperthread) none IB - 40 Gbps
Appel AT Lustre FS GPFS FS (GS) 1 384 11 TB none 384 none IB - 40 Gbps
Cryo AT Lustre FS GPFS FS (GS) 1 40 1,5 TB 8 x V100/32GB SXM 40 8 IB - 40 Gbps
Arrow AT HPFS NVMe HPFS spin 2 128 512 GB 8 X A100/80GB 128 16 IB - 200 Gbps
Math CT HPFS NVMe HPFS spin 1 128 512 GB 2 X A40/48GB 128 2 B - 200 Gbps
Phys CT HPFS NVMe HPFS spin 1 64 512 GB none 64 none B - 200 Gbps
Chem CT HPFS NVMe HPFS spin 2 64 512 GB 2 x A30/48GB PCI 3 + 8 x A100/40GB SXM 192 10 B - 200 Gbps
ASRC CT HPFS NVMe HPFS spin 1 64 512 GB 2 x A30/24GB 64 2 B - 200 Gbps
Eng CT HPFS NVMe HPFS spin 1 128 512 GB 1 x A40/48GB 128 1 B - 200 Gbps

The computational architectures participating in each tier are discussed below. All tiers access 2 separate file systems: 1) Global Storage (GS) (a global file system) and 2) /scratch file system. The GS is mounted only on login nodes and is proposed to keep user data (home directories) and project data (project directories). The /scratch file system is mounted on login nodes and on all computational nodes (in all tiers). Thus GS holds long lasting user data (executables, scripts and data), while /scratch is used to hold provisional data required by particular simulation(s). Consequently, jobs can be started only from /scratch and never from GS storage. It is important to remember that /scratch is not main storage for users' accounts (home directories), but a temporary storage used for job submission only. Thus:

  1. data in /scratch are not protected, preserved or backup-ed and can be lost at any time. CUNY-HPCC has no obligation to preserve user data in /scratch.
  2. /scratch undergoes regular file purging when either or both conditions are satisfied:
    1. load of the /scratch file system reaches 71%.
    2. there is/are inactive file(s) older than 60 days.
  3. only data in GS are protected and recoverable.

Upon registering with HPCC every user will get 2 directories:

/scratch/<userid> – this is temporary workspace on the HPC systems
/global/u/<userid> – space for “home directory”, i.e., storage space on the DSMS for program, scripts, and data
• In some instances a user will also have use of disk space on the DSMS in /cunyZone/home/<projectid> (IRods).

The /global/u/<userid> directory has quota (see below for details) while the /scratch/<userid> do not have. However the /scratch space is cleaned up following the rules described below. There are no guarantees of any kind that files in /scratch will be preserved during the hardware crashes or cleaning up. Access to all HPCC resources is provided by bastion host called 'chizen. The Data Transfer Node called Cea allows file transfer from/to remote sites directly to/from /global/u/<userid> or to/from /scratch/<userid>

Condominium Tier

Condominium tier (called condo) organizes resources purchased and owned by faculty, but maintained by HPCC. The participation in this tier is strictly voluntary. Several faculty/research groups can combine finds to purchase and consequently share the hardware (a node or several nodes). All nodes in this tier must meet certain hardware specifications including to be fully warranted for time life of the node(s) in order to be accepted. If you want to participate in condominium please sent a request mail to hpchelp@csi.cuny.edu and consult HPCC before making a purchase. Condominium tier:

  • Promotes vertical and horizontal collaboration between research groups;
  • Makes possible to utilize small amounts of research money or "left-over" money wisely and to obtain advanced resources;
  • Helps researchers to conduct large scope high quality research including collaborative projects leading to successful grants with high impact.

Access to condo resources

The resources are available only for condo owners and their groups. The users registered with condo must use the main login node of Arrow server. To access their own node the condo users must specify their own private partition. In addition there are partitions which operate over two or more nodes owned by condo members. Condo users may us (private partition) to access their node. he condo tier members benefit from professional support from HPCC security and maintenance. Upon approval (from the node owner) any idle node(s) can be used by any other member(s). For instance a member can borrow (for agreed time) node with more advanced GPUs than those installed on his/her own node(s). The owners of the equipment are responsible for any repair costs for their node(s). Other users may rent any of the described below condo resources if agreed with owners. The unused cycles can be shared with other members of the community.

In sum the benefits of condo are:

  • 5 year lifecycle - condo resources will be available for a duration of 5 years.

Access to more cpu cores than purchased and access to resources which are not purchased.

  • Support - HPCC staff will install, upgrade, secure and maintain condo hardware throughout its lifecycle.
  • Access to main application server
  • Access to HPC analytics

Responsibilities of condo memnbers

  • To share their resources (when idle or partially available) with other members of a condo;
  • To include in their research and instrumentation grants money for computing used to cover operational expences (non-tax-levy expenses) of the HPCC.

The table below summarized the available resources

Resources in Condominium Tier (Arrow cluster)
Number of nodes Cores/node Chip Memory/node GPU/node Interconnect Use Private partition
2 64 2 x AMD EPYC 256 GB 2 x A30 24 GB, PCIe gen 3 100 Gbps Infiniband EDR Number Crunching parchem, partasrc
1 64 2 x AMD EPYC 512 GB -- 100 Gbps infiniband EDR Number Crunching parthphys
2 128 2 x AMD EPYC 512 GB 2 x A40 48 GB, PCIe gen 3 100 Gbps Infiniband EDR Number Crunching parting, partmath
1 128 2 x AMD EPYC 512 GB 8 x A100 40GB, SXM 200 Gbps Infiniband HDR Number Crunching parched

Advanced Tier

The advanced tier holds the resources used for more advanced or large scale research. This tier provides nodes with Volta class GPUs with 16 GB and 32 GB on board. The table below summarizes the resources.

Resources in Advanced Tier (Blue Moon, Cryo, Appel): 1256 cores and 12 Volta class GPU
Number of Nodes Cores/node Chip Memory/node GPU/node Interconnect Use Association
2 32 2 x Intel X86_64 192 GB 2 x V100 (16 GB) PCIe gen 3 100 Gbps Infiniband EDR Number Crunching Blue Moon Cluster
24 32 2 x Intel X86_64 192 GB -- 100 Gbps Infiniband EDR Number Crunching Blue Moon Cluster
1 40 2 x Intel X86_64 1500 GB 8 x V100 (32 GB) SXM 100 Gbps Infiniband EDR Number Crunching Cryo
1 384 2 x Intel X86_64 11,000 GB -- 56 Gbps Infiniband QDR Number Crunching Appel

Basic tier

The basic tier provides resources for sequential and moderate size parallel jobs. The openMP jobs can be run only in a scope of single node. The distributed parallel jobs (MPI) can be run across cluster. This tier alsop support Matlab Parallel server which can be run across nodes. The users also can run GPU enabled jobs since this tier has 132 GPU Tesla K20m. Please note that these GPU are not supported by NVIDIA anymore. Many applications also may not support this GPU as well. The table below summaries the resources for their tier.

Resources in Free Tier (Penzias, Karle):1056 cores and 132 Tesla class GPU
Number of nodes Cores/node Chip Memory/node GPU/node Interconnect Use Association
66 16 2 x Intel X86_64 64 GB 2 x K20m, PCIe.2 56 Gbps Infiniband Number crunching, Sequential jobs, General computing, Distributed Parallel Matlab Penzias
1 36/72* 2 x Intel X86_64 768 GB -- 56 Gbps Infiniband Visualization, Matlab, Parallel Matlab (toolbox) Karle

* Hyperthread

Arrow cluster and hybrid storage (NSF grant 2023 equipment)

This equipment consist of large hybrid parallel file system and 2 computational nodes integrated in a cluster named Arrow. The file system has capacity of 2PB (Petabytes) and bandwidth of 35 GBps write and 50 GBps read. The computational nodes details are summarized in table below.

Resources in NSF grant equipment (Arrow): Total of 256 cores and 16 GPU Ampere A100/80GB GPU
Number of Nodes Cores/node GPU/node Memory/node Chip Interconnect Use Association
2 128 8 x A100/80GB 1024 GB 2 x AMD EPYC HDR 100 Gbps Molecular modeling, Data science, Number Crunching, Materials Science, AI, ML Arrow


HPC systems and their architectures

The HPC Center operates variety of architectures in order to support complex and demanding workflows. The deployed systems include: distributed memory (also referred to as “cluster”) computers, symmetric multiprocessor (also referred as SMP) and shared memory (also reffred as NUMA machines).

Computational Systems:

SMP servers have several processors (working under a single operating system) which "share everything". Thus all cpu-cores allocate a common memory block via shared bus or data path. SMP servers support all combinations of memory VS cpu (up to the limits of the particular computer). The SMP servers are commonly used to run sequential or thread parallel (e.g. OpenMP) jobs and they may have or may not have GPU. Currently, HPCC operates several detached SMP servers named Math, Cryo and Karle. Karle is a server which does not have GPU and is used for visualizations, visual analytics and interactive MATLAB/Mathematica jobs. Math is a condominium server without GPU as well. Cryo (CPU+GPU server) is specialized server with eight (8) NVIDIA V100 (32G) GPU designed to support large scale multi-core multi-GPU jobs.

Cluster is defined as a single system comprizing a set of SMP servers interconnected with high performance network. Specific software coordinates programs on and/or across those in order to perform computationally intensive tasks. Each SMP member of the cluster is called a node. All nodes run independent copies of the same operating system (OS). Some or all of the nodes may incorporate GPU. The main cluster at HPCC is a hybrid (CPU+GPU) cluster called Penzias. Sixty six (66) of Penzias nodes have 2 x GPU K20m, and the 3 fat nodes (nodes with large number of CPU-cores and memory) of the cluster do not have GPU. In addition HPCC operates the cluster Herbert dedicated only to education.

Distributed shared memory computer is tightly coupled server in which the memory is physically distributed, but it is logically unified as a single block. The system resembles SMP, but the number of cpu cores and the amounts of memory possible is far beyond limitations of the SMP. Because the memory is distributed, the access times across address space are non-uniform. Thus, this architecture is called Non Uniform Memory Access (NUMA) architecture. Similarly to SMP, the NUMA systems are typically used for applications such as data mining and decision support system in which processing can be parceled out to a number of processors that collectively work on a common data. HPCC operates the NUMA server called Appel. This server does not have GPU.

Infrastructure systems:

o Master Head Node (MHN) is a redundant login node from which all jobs on all servers start. This server is not directly accessible from outside CSI campus.

o Chizen is a redundant gateway server which provides access to protected HPCC domain.

o Cea is a file transfer node allowing transfer of files between users’ computers to/from /scratch space or to/from /global/u/<usarid>. Cea is accessible directly (not only via Chizen), but allows only limited set of shell commands.

Table 1 below provides a quick summary of the attributes of each of the systems available at the HPC Center.

CUNY-HPCC resources
System Type Type of Jobs Number of Nodes Cores/node Chip Type GPU/node Memory/node (GB)
Penzias Hybrid Cluster Sequential and parallel jobs with/without GPU 66 16 2 x Intel Sandy Bridge EP 2.20 GHz 2 x K20M (5 GB on board), PCIe 2.0 64 GB
Blue Moon Hybrid cluster Sequential and parallel jobs with/without GPU 24 (CPU) & 2 (CPU + GPU) 32 2 x Intel Skylake 2.10 GHz 2 x V100 (16 GB on board), PCIe 3.0 192 GB
Cryo SMP Sequential and parallel jobs with/without GPU 1 40 2 x Intel Skylake 2.10 GHz 8 x V100 (32 GB on board), XSM 1,500 GB
Appel NUMA Massive parallel and/or big data jobs without GPU 1 384 2 x Intel Ivy Bridge 3.0 GHz -- 11,000 GB
Fat node 1 Part of Penzias Big data jobs without GPU 1 24 2 x Intel Sandy Bridge 2.30 GHz -- 768 GB
Fat node 2 Part of Penzias Big data jobs without GPU 1 24 2 x Intel Sandy Bridge 2.30 GHz -- 1500 GB
Fat node 3 Part of Penzias Big data jobs without GPU 1 32 2 x Intel Haswell 2.30 GHz -- 768 GB
Condo 1 Part of Condo Sequential and parallel jobs with/without GPU 1 64 2 x AMD EPYC Roma, 2.10 GHz 2 x A30 256 GB
Condo 2 Part of Condo Sequential and parallel jobs with/without GPU 1 64 2 x AMD EPYC Roma, 2.10 GHz 2 x A30 256 GB
Condo 3 Part of Condo Sequential and parallel jobs with/without GPU 1 128 2 x AMD EPYC Roma, 2.10 GHz 1 x A40 512 GB
Condo 4 Part of Condo Sequential and parallel jobs with/without GPU 1 128 2 x AMD EPYC Roma, 2.10 GHz 2 x A40 512 GB
Condo 5 Part of Condo Sequential and parallel jobs with/without GPU 1 64 2 x AMD EPYC Roma, 2.10 GHz -- 512 GB
Karle SMP Visualization, Matlab, Mathematica 1 72* 2 x Intel Sandy Bridge 2.30 GHz -- 768 GB
Chizen SMP Gateway only 1 (redundant) -- -- -- 64 GB
MHN SMP Master Head Node 2 (redundant) -- -- -- --
Cea SMP File Transfer Node 1 -- -- -- --

* hyperthread

Partitions and jobs

The only way to submit job(s) to HPCC servers is through SLURM batch system. Any job despite of its type (interactive, batch, serial, parallel etc.) must be submitted via SLURM. The latter allocates the requested resources on proper server and starts the job(s) according to predefined strict fair share policy. Computational resources (cpu-cores, memory, GPU) are organized in partitions. The main partition is called production. This is routing partition which distributes the jobs in several sub-partitions depend on job’s requirements. Thus the serial job submitted in production will land in partsequential partition. No SLURM Pro scripts should be ever used and all existing SLURM scripts must be converted to SLURM before use. The table below shows the limitations of the partitions.

Partition Max cores/job Max jobs/user Total cores/group Time limits
production 128 50 256 240 Hours
partedu 16 2 216 72 Hours
partmath 128 128 128 240 Hours
partmatlab 1972 50 1972 240 Hours
partdev 16 16 16 4 Hours

o production is the main partition with assigned resources across all servers (except Math and Cryo).It is routing partition so the actual job(s) will be placed in proper sub-partition automatically. Users may submit sequential, thread parallel or distributed parallel jobs with or without GPU.

o partedu partition is only for education. Assigned resources are on educational server Herbert. Partedu is accessible only to students (graduate and/or undergraduate) and their professors who are registered for a class supported by HPCC. Access to this partition is limited by the duration of the class.

o partmatlab partition allows to run MATLAB's Distributes Parallel Server across main cluster. Note however that parallel toolbox programs can be submitted via production partition, but only as thread parallel jobs.

o partdev is dedicated to development. All HPCC users have access to this partition with assigned resources of one computational node with 16 cores, 64 GB of memory and 2 GPU (K20m). This partition has time limit of 4 hours.

Hours of Operation

The second and fourth Tuesday mornings in the month from 8:00AM to 12PM are normally reserved (but not always used) for scheduled maintenance. Please plan accordingly.
Unplanned maintenance to remedy system related problems may be scheduled as needed. Reasonable attempts will be made to inform users running on those systems when these needs arise.

User Support

Users are encouraged to read this Wiki carefully. In particular, the sections on compiling and running parallel programs, and the section on the SLURM batch queueing system will give you the essential knowledge needed to use the CUNY HPC Center systems. We have strived to maintain the most uniform user applications environment possible across the Center's systems to ease the transfer of applications and run scripts among them.

The CUNY HPC Center staff, along with outside vendors, offer regular courses and workshops to the CUNY community in parallel programming techniques, HPC computing architecture, and the essentials of using our systems. Please follow our mailings on the subject and feel free to inquire about such courses. We regularly schedule training visits and classes at the various CUNY campuses. Please let us know if such a training visit is of interest. In the past, topics have include an overview of parallel programming, GPU programming and architecture, using the evolutionary biology software at the HPC Center, the SLURM queueing system at the CUNY HPC Center, Mixed GPU-MPI and OpenMP programming, etc. Staff has also presented guest lectures at formal classes throughout the CUNY campuses.

If you have problems accessing your account and cannot login to the ticketing service, please send an email to:

 hpchelp@csi.cuny.edu 

Warnings and modes of operation

1. hpchelp@csi.cuny.edu is for questions and accounts help communication only and does not accept tickets. For tickets please use the ticketing system mentioned above. This ensures that the person on staff with the most appropriate skill set and job related responsibility will respond to your questions. During the business week you should expect a 48h response, quite often even same day response. During the weekend you may not get any response.

2. E-mails to hpchelp@csi.cuny.edu must have a valid CUNY e-mail as reply address. Messages originated from public mailers (google, hotmail, etc) are filtered out.

3. Do not send questions to individual CUNY HPC Center staff members directly. These will be returned to the sender with a polite request to submit a ticket or email the Helpline. This applies to replies to initial questions as well.

The CUNY HPC Center staff members are focused on providing high quality support to its user community, but compared to other HPC Centers of similar size our staff is extremely lean. Please make full use of the tools that we have provided (especially the Wiki), and feel free to offer suggestions for improved service. We hope and expect your experience in using our systems will be predictably good and productive.