Running Jobs - Revision history

Alex: /* SLURM Partitions and Quality of Service QOS */

2024-12-21T16:41:28Z

SLURM Partitions and Quality of Service QOS

← Older revision		Revision as of 16:41, 21 December 2024
Line 24:		Line 24:

	=== SLURM Partitions and Quality of Service QOS ===		=== SLURM Partitions and Quality of Service QOS ===
	The main partition which distributes jobs on other partitions is '''production'''. It is public. The condo tier is accessed via 7 private partitions (see table below). Users not listed in condo must use '''production''' partition for all job submission. The partition has time limit of 120 hours (currently). Note that time limit as well as number of jobs per group and per user are '''reviewed periodically''' and may change in order to maximize utilization f the resources. In addition the MHN supports '''partdev partition''' which has limit of 2 hours and is dedicated to development of the codes. In addition users in condo must use QOS identifier in order to submit jobs on proper SLURM partition.		The main partition which distributes jobs on other partitions is '''production'''. It is public. The condo tier can be accessed via 7 private partitions (see table below) with use of QOS (Quality of Service) identificator. Users not listed in condo must use '''production''' partition for all job submission. The partition has time limit of 120 hours (currently). Note that time limit as well as number of jobs per group and per user are '''reviewed periodically''' and may change in order to maximize utilization f the resources. In addition the MHN supports '''partdev partition''' which has limit of 2 hours and is dedicated to development of the codes. In addition users in condo must use QOS identifier in order to submit jobs on proper SLURM partition.
	{\| class="wikitable mw-collapsible"		{\| class="wikitable mw-collapsible"
	\|+Public SLURM partitions		\|+Public SLURM partitions (Basic and Advanced Tier Only)
	!Partition		!Partition
	!Max cores/job		!Max cores/job
Line 64:		Line 64:
	\|}		\|}

	=== QOS access ===		=== QOS access and private partitions (Condo Tier Only) ===
	The access to each partition is controlled by Quality of service (QOS) function. Thus only users registered for a particular partition with matched QOS credentials will be allowed to run. Simply put, any job from unauthorized user will be rejected. Table below summarizes the information for partitions:		The access to each partition is controlled by Quality of service (QOS) function. Thus only users registered for a particular partition with matched QOS credentials will be allowed to run. Simply put, any job from unauthorized user will be rejected. Table below summarizes the information for partitions:
	{\| class="wikitable mw-collapsible"		{\| class="wikitable mw-collapsible"
	\|+Private SLURM partitions and QOS (~~condo tier only~~)		\|+Private SLURM partitions and QOS (Condo Tier Only)
	!Partition name		!Partition name
	!Type		!Type

Alex: /* SLURM Partitions and Quality of Service QOS */

2024-12-21T16:39:09Z

SLURM Partitions and Quality of Service QOS

@@ Line 25: / Line 25: @@
 === SLURM Partitions and Quality of Service QOS ===
 The main partition which distributes jobs on other partitions is '''production'''. It is public. The condo tier is accessed via 7 private partitions (see table below). Users not listed in condo must use '''production''' partition for all job submission. The partition has time limit of 120 hours (currently).  Note that time limit as well as number of jobs per group and per user are '''reviewed periodically''' and may change in order to maximize utilization f the resources. In addition the MHN supports '''partdev partition''' which has limit of 2 hours and is dedicated to development of the codes. In addition users in condo must use QOS identifier in order to submit jobs on proper SLURM partition.
 === QOS access ===
 The access to each partition is controlled by Quality of service (QOS) function. Thus only users registered for a particular partition with matched QOS credentials  will be allowed to run. Simply put, any job from unauthorized user will be rejected. Table below summarizes the information for partitions:
-{| class="wikitable"
+{| class="wikitable mw-collapsible"
-|+
+|+Private SLURM partitions and QOS (condo tier only)
 !Partition name
 !Type

Alex: /* Advanced and Free Tier */

2024-12-21T02:05:29Z

Advanced and Free Tier

Show changes

Alex: /* Running jobs on Arrow and condo servers */

2023-10-10T04:42:57Z

Running jobs on Arrow and condo servers

← Older revision		Revision as of 04:42, 10 October 2023
Line 135:		Line 135:

	== Running jobs on Arrow and condo servers ==		== Running jobs on Arrow and condo servers ==
	Arrow server consist of 2 nodes purchased with NSF AOC grant funding. Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes share the same login node and are attached via 200 Gbps infiniband interconnect to the same hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying ~~files~~ system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.		Arrow server consist of 2 nodes purchased with NSF AOC grant funding. Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes share the same login node and are attached via 200 Gbps infiniband interconnect to the same hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying system software manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.

	=== Partitions and QOS access ===		=== Partitions and QOS access ===

Alex: /* Running jobs on Arrow and condo servers */

2023-10-10T04:41:50Z

Running jobs on Arrow and condo servers

← Older revision		Revision as of 04:41, 10 October 2023
Line 135:		Line 135:

	== Running jobs on Arrow and condo servers ==		== Running jobs on Arrow and condo servers ==
	Arrow server consist of 2 nodes purchased with NSF AOC grant funding. Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes ~~mount~~ the same login node and are attached via 200 Gbps infiniband interconnect to a hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.		Arrow server consist of 2 nodes purchased with NSF AOC grant funding. Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes share the same login node and are attached via 200 Gbps infiniband interconnect to the same hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.

	=== Partitions and QOS access ===		=== Partitions and QOS access ===

Alex: /* Running jobs on Arrow and condo servers */

2023-10-10T04:41:05Z

Running jobs on Arrow and condo servers

← Older revision		Revision as of 04:41, 10 October 2023
Line 135:		Line 135:

	== Running jobs on Arrow and condo servers ==		== Running jobs on Arrow and condo servers ==
	Arrow server consist of 2 nodes purchased with NSF AOC grant funding/ Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes mount the same login node and are attached via 200 Gbps infiniband interconnect to a hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.		Arrow server consist of 2 nodes purchased with NSF AOC grant funding. Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes mount the same login node and are attached via 200 Gbps infiniband interconnect to a hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.

	=== Partitions and QOS access ===		=== Partitions and QOS access ===

Alex: /* Running jobs on Arrow and condo servers */

2023-10-10T04:40:29Z

Running jobs on Arrow and condo servers

← Older revision		Revision as of 04:40, 10 October 2023
Line 135:		Line 135:

	== Running jobs on Arrow and condo servers ==		== Running jobs on Arrow and condo servers ==
	Arrow server and condo ~~servers~~ are attached via 200 Gbps infiniband interconnect to a ~~separate~~ hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.		Arrow server consist of 2 nodes purchased with NSF AOC grant funding/ Condo servers are purchased directly by faculty with their own research funds. Arrow nodes and condo nodes mount the same login node and are attached via 200 Gbps infiniband interconnect to a hybrid (NVMI + hard disks) fast file system called '''HPFFS''' which can provide speeds of 25-30GB/s write and 45-50 GB/s read. The '''/scratch''' and '''/global/u''' are part of the same '''HPFFS''' file system, but scratch is optimized for predominant access of the fast NVMI tier of '''HPFFS'''. The underlying files system manipulates the placement of the files to ensure the best possible performance for different file types. All jobs must start from '''<font face="courier"">/scratch/<font color="red"><userid></font></font>''' directory. Jobs <u>cannot be started</u> from user's home: '''/global/u/<font face="courier"><font color="red"><userid></font></font>''' It is important to mention that data on '''/global/u''' on '''HPFFS''' file system are not backup-ed since this equipment is not integrated in HPCC infrastructure. Every user home directory has a quote of 100 GB. The latter can be expanded by submitting motivated request to HPCC.

	=== Partitions and QOS access ===		=== Partitions and QOS access ===

Alex: /* Condo Tier and Arrow */

2023-10-10T04:36:03Z

Condo Tier and Arrow

← Older revision		Revision as of 04:36, 10 October 2023
Line 11:		Line 11:

	=== Condo Tier and Arrow ===		=== Condo Tier and Arrow ===
	At it was stated above all jobs must start from '''<font face="courier">/scratch/<font color="red"><userid></font></font>''' directory and the valuable data must be kept in '''/global/u/<font face="courier"><font color="red"><userid></font></font>'''. ~~The~~ '''/scratch''' and '''/global/u''' reside on the same HPFS file system over 200 Gbps Infiniband interconnect. The system software takes cares of optimal placement of the files. Note, that '''/global/u is not backed on tape at that time due to lack of funds.'''		At it was stated above all jobs must start from '''<font face="courier">/scratch/<font color="red"><userid></font></font>''' directory and the valuable data must be kept in '''/global/u/<font face="courier"><font color="red"><userid></font></font>'''. For Arrow and condo servers the '''/scratch''' and '''/global/u''' reside on the same HPFS file system over 200 Gbps Infiniband interconnect. The system software takes cares of optimal placement of the files. Note, that '''/global/u is not backed on tape at that time due to lack of funds.'''

	=== Copy/move files from/to server ===		=== Copy/move files from/to server ===

Alex: /* List of available modules */

2023-10-10T04:33:38Z

List of available modules

← Older revision		Revision as of 04:33, 10 October 2023
Line 138:		Line 138:

	=== Partitions and QOS access ===		=== Partitions and QOS access ===
	Arrow server has one public partition and '''seven private partitions'''. The public partition is open for all core users of NSF grant (). The private partitions are restricted to owners of the resources. The access to each partition is controlled by Quality of service (QOS) function. Thus only registered ~~users~~ for a particular partition with matched QOS credentials will be allowed to run. Simply put, any job from unauthorized user will be rejected. Table below summarizes the information for partitions:		Arrow server has one public partition and '''seven private partitions'''. The public partition is open for all core users of NSF grant (). The private partitions are restricted to owners of the resources. The access to each partition is controlled by Quality of service (QOS) function. Thus only users registered for a particular partition with matched QOS credentials will be allowed to run. Simply put, any job from unauthorized user will be rejected. Table below summarizes the information for partitions:
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+

Alex: /* Partitions and quality of service (QOS) */

2023-10-10T04:31:41Z

Partitions and quality of service (QOS)

@@ Line 372: / Line 372: @@
 ===Job(s) execution time===
 The job execution time is sum with them the job waits in  SLURM partition (queue) before being executed on node(s) and  actual running time on node. For the parallel corder the partition time (time job waits in partition) increases with increasing resources such as number of CPU-cores. On other hand the execution time (time on nodes) decreases with inverse of resources. Each job has its own "sweet spot" which minimizes the time to solution. Users are encouraged to run several test runs and to figure out what amount of asked resources works best for their job(s).
 === Working with QOS and partitions on Arrow===
 Every job submission script on Arrow must hold proper description of QOS and partition. For instance all jobs intended to use node n133 must have the following lines:<syntaxhighlight lang="abap">