James: Text replacement - "[pP][bB][sS]" to "SLURM"

2022-11-07T18:11:53Z

Text replacement - "[pP][bB][sS]" to "SLURM"

← Older revision		Revision as of 18:11, 7 November 2022
Line 48:		Line 48:
	<font face="courier">		<font face="courier">
	cd /scratch/<font color="red"><userid></font color>		cd /scratch/<font color="red"><userid></font color>
	mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">~~myPBS_Job~~</font color>		mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>
	cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/a.out ./		cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/a.out ./
	cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/<font color="red">mydatafile </font>./		cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/<font color="red">mydatafile </font>./
Line 56:		Line 56:
	<font face="courier">		<font face="courier">
	cd /scratch/<font color="red"><userid></font color>		cd /scratch/<font color="red"><userid></font color>
	mkdir <font color="red">~~myPBS_Job~~</font color> && cd <font color="red">~~myPBS_Job~~</font color>		mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>
	iget <font color="red">myProject</font color>/a.out		iget <font color="red">myProject</font color>/a.out
	iget <font color="red">myProject</font color>/<font color="red">mydatafile</font color>		iget <font color="red">myProject</font color>/<font color="red">mydatafile</font color>
Line 91:		Line 91:
	mv ./myoutputs /global/u/<font color="red"><userid></font color>/<font color="red">myProject</font color>/.		mv ./myoutputs /global/u/<font color="red"><userid></font color>/<font color="red">myProject</font color>/.
	cd ../		cd ../
	rm -rf <font color="red">~~myPBS_Job~~</font color>		rm -rf <font color="red">mySLURM_Job</font color>
	</font>		</font>

Line 98:		Line 98:
	iput ./myoutputs <font color="red">myProject</font color>/.		iput ./myoutputs <font color="red">myProject</font color>/.
	cd ../		cd ../
	rm -rf <font color="red">~~myPBS_Job~~</font color>		rm -rf <font color="red">mySLURM_Job</font color>
	</font>		</font>

James: Created page with "=Data Storage and Management System (DSMS)= TOC Key features of the '''DSMS''' system include: :• '''User''' home directories in a standard Unix file system called /global/u. :• Enhanced parallel scratch space on the HPC systems. :• '''Project''' directories in an Integrated Rule-Oriented Data-management System (iRODS) managed resource. Project directories exist in a “virtual file space” called '''cunyZone''' which contains a resource called '''Storage..."

2022-10-20T18:54:01Z

Created page with "=Data Storage and Management System (DSMS)= __TOC__ Key features of the '''DSMS''' system include: :• '''User''' home directories in a standard Unix file system called /global/u. :• Enhanced parallel scratch space on the HPC systems. :• '''Project''' directories in an Integrated Rule-Oriented Data-management System (iRODS) managed resource. Project directories exist in a “virtual file space” called '''cunyZone''' which contains a resource called '''Storage..."

New page

=Data Storage and Management System (DSMS)=

__TOC__

Key features of the '''DSMS''' system include:

:• '''User''' home directories in a standard Unix file system called /global/u.
:• Enhanced parallel scratch space on the HPC systems.
:• '''Project''' directories in an Integrated Rule-Oriented Data-management System (iRODS) managed resource. Project directories exist in a “virtual file space” called '''cunyZone''' which contains a resource called '''Storage Resource 1 (SR1)'''. For the purpose of this document, we will use the terminology SR1 to describe '''Project file space.'''
:• Automated backups.

The '''DSMS''' is the HPC Center’s primary file system and is accessible from all existing HPC systems, except for '''HERBERT''' . It will similarly be accessible from all future HPC systems.

The '''DSMS''' provides a 3-level data storage infrastructure: - '''HOME''' filesystem, '''SCRATCH''' filesystems, '''SR1''' (long-tern storage resource)

'''DSMS''' features are explained below.

=="Home" directories are on /global/u==
'''/global/u''' is a standard Unix file system that holds the home directories of individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<userid>''' and a 50 GB allocation of disk space for home directories on '''/global/u/<userid>'''. These '''home''' directories are on the '''DSMS''', not on the HPC systems, but can be accessed from any Center system. All home directories are backed up on weekly basis.

==/scratch==
Disk storage on the HPC systems is used only for '''scratch''' files. '''scratch''' files are temporary and are '''not backed up'''. '''/scratch''' is used by jobs queued for or in execution. Output from jobs may temporarily be located in '''/scratch'''.

In order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''/scratch''' from '''/global/u''' using UNIX commands and/or from '''SR1''' using '''iRODS''' commands.

Files in '''/scratch''' on a system are '''automatically purged''' when (1) usage reaches 70% of available space, or (2) file residence on scratch exceeds two weeks, whichever occurs first.

==“Project” directories==

'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands. In iRODS terminology, a “collection” is the equivalent of “directory”.

A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files. A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist. The PI is the individual responsible to the University or a granting agency for the “Project”. The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”.

Additional information on the '''DSMS''' is available in Section 4 of the User Manual 
http://www.csi.cuny.edu/cunyhpc/pdf/User_Manual.pdf

==Typical Workflow==
Typical workflows in are described below:

1. Copying files from a user’s home directory or from '''SR1''' to '''SCRATCH'''. 
If working with '''HOME''': 

cd /scratch/<userid>
mkdir mySLURM_Job && cd myPBS_Job
cp /global/u/<userid>/myProject/a.out ./
cp /global/u/<userid>/myProject/mydatafile ./


If working with '''SR1''': 

cd /scratch/<userid>
mkdir myPBS_Job && cd myPBS_Job
iget myProject/a.out
iget myProject/mydatafile


2. Prepare SLURM job script. Typical SLURM sript is similar to the following: 

#!/bin/bash
#SBATCH --partition production
#SBATCH -J test
#SBATCH --nodes 1
#SBATCH --ntasks 8
#SBATCH --mem 4000
echo "Starting…"

cd $SLURM_SUBMIT_DIR
mpirun -np 4 ./a.out ./mydatafile > myoutputs
echo "Done…"

Your SLURM may be different depending on your needs. Read section Submitting Jobs for a reference. 

3. Run the job 

sbatch ./mySLURM_script


4. Once job is finished, clean up '''SCRATCH''' and store outputs in your user home directory or in '''SR1'''. 

If working with '''HOME''': 

mv ./myoutputs /global/u/<userid>/myProject/.
cd ../
rm -rf myPBS_Job


If working with '''SR1''': 

iput ./myoutputs myProject/.
cd ../
rm -rf myPBS_Job


5. If output files are stored in '''SR1''' tag them with metadata. 

imeta addw -d myoutput zvalue 15 meters
imeta addw -d myoutput colorLabel RED

 

== iRODS ==

'''iRODS''' is the integrated Rule-Oriented Data-management System, a
community-driven, open source, data grid software solution. '''iRODS''' is
designed to abstract data services from data storage hardware and
provide users with hardware-agnostic way to manipulate data.

'''iRODS''' is a primary tool that is used by the CUNY HPCC users to
seamlessly access 1PB storage resource (further referenced as '''SR1'''
here) from any of the HPCC's computational systems.

Access to '''SR1''' is provided via so-called '''i-commands''':

<pre>iinit
ils
imv</pre>

Comprehesive list of i-commands with detailed description can be
obtained at [https://wiki.irods.org/index.php/icommands iRODS wiki].

To obtain quick help on any of the commands while being logged into
any of the HPCC's machines type '''i-command -h'''. For example:
<pre>
ils -h
</pre>

Following is the list of some of the most relevant '''i-commands''':

:'''iinit''' -- Initialize session and store your password in a scrambled form for automatic use by other icommands.
:'''iput''' -- Store a file
:'''iget''' -- Get a file
:'''imkdir''' -- Like mkdir, make an iRODS collection (similar to a directory or Windows folder)
:'''ichmod''' -- Like chmod, allow (or later restrict) access to your data objects by other users.
:'''icp''' -- Like cp or rcp, copy an iRODS data object
:'''irm''' -- Like rm, remove an iRODS data object
:'''ils''' -- Like ls, list iRODS data objects (files) and collections (directories)
:'''ipwd''' -- Like pwd, print the iRODS current working directory
:'''icd''' -- Like cd, change the iRODS current working directory
:'''ichksum''' -- Checksum one or more data-object or collection from iRODS space.
:'''imv''' -- Moves/renames an irods data-object or collection.
:'''irmtrash''' -- Remove one or more data-object or collection from a RODS trash bin.
:'''imeta''' -- Add, remove, list, or query user-defined Attribute-Value-Unit triplets metadata
:'''iquest''' -- Query (pose a question to) the ICAT, via a SQL-like interface

Before using any of the i-commands users need to identify themselves to the iRODS server running command
<pre>
# iinit
</pre>
and providing HPCC's password.

Typical workflow that involves operations on files stored in SR1
include storing/getting data to and from SR1, tagging data with
metadata, searching for data, sharing (setting permissions).

==== Storing data to SR1 ====

1. Create '''iRODS''' directory (aka 'collection'): 

# imkdir myProject

2. Store all files ''''myfile*'''' into this directory (collection): 

# iput -r <userid> myfile* myProject/.

3. Verify that files are stored: 

# ils
/cunyZone/home/<userid>:
C- /cunyZone/home/<userid>/myProject
# ils myProject
/cunyZone/home/<userid>/myProject:
myfile1
myfile2
myfile3


Symbol ''''C-'''' in the beginning of output of ''''ils'''' shows that listed item is a collection.

4. Combining ''''ils', 'imkdir', 'iput', 'icp', 'ipwd', 'imv'''' user can create iRODS directories and store files in them similarly to what is normally done with UNIX commands ''''ls', 'mkdir', 'cp', 'pwd', 'mv'''' etc...

==== Getting data from SR1 ====

1. To copy file from SR1 to current working directory run
# iget myProject/myfile1

2. Now listing current working directory should reveal '''myfile1''':
# ls
myfile1

3. Instead of individual files the whole directory (with
sub-directories) can be copied with ''''-r'''' flag (stands for
'recursive')
# iget -r myProject

NOTE: wildcards are not supported, therefore the command below will not work
# iget myProject/myfile*

=== Tagging data with metadata ===

iRODS provides users with extremely powerful mechanism of managing
data with metadata. While working with large datasets it's
sometimes easy to forget what is stored in this or the other file.
Metadata tags help organizing data in a very easy and reliable
manner.

Let's tag files from previous example with some metadata:
<pre>
# imeta add -d myProject/myfile1 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 comment "This is file number 1"
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 zvalue 10 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 comment "This is file number 2"
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 colorLabel BLUE
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 comment "This is file number 3"
AVU added to 1 data-objects
</pre>
Here we've tagged myfile1 with 3 metadata labels:

- zvalue 10 meters

- colorlabel RED

- comment "This is file number 1"

Similar tags were added to 'myfile2' and 'myfile3'

Metadata come in form of AVU -- Attribute|Value|Unit. As seen from
the above examples Unit is not necessary.

Let's list all metadata assigned to file 'myfie1':
<pre>
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: zvalue
value: 15
units: meters
----
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
</pre>

To remove an AVU assigned to a file run:
<pre>
# imeta rm -d myProject/myfile1 zvalue 15 meters
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
#
#
# imeta add -d myProject/myfile1 zvalue 15 meters
</pre>

Metadata may be assigned to directories as well:
<pre>
# imeta add -C myProject simulationsPool 1
# imeta ls -C myProject
AVUs defined for collection myProject:
attribute: simulationsPool
value: 1
units:
</pre>

Note the '-C' key that is used instead of '-d'.

=== Searching for data ===

Power of metadata becomes obvious when data needs to be found in
large collections. Here is an illustration of how easy this task is
done with iRODS via imeta queries:

# imeta qu -d zvalue = 15
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile3

We see both files that were tagged with label 'zvalue 10 meters'.
Here is different query:

# imeta qu -d colorLabel = RED
collection: /cunyZone/home/<userid></myProject
dataObj: myfile1
----
collection: /cunyZone/home/<userid>/myProject
dataObj: myfile2

Another powerful mechanism to query data is provided with 'iquest'.
Following is a number of examples that show 'iquest' capabilities:

iquest "SELECT DATA_NAME, DATA_SIZE WHERE DATA_RESC_NAME like 'cuny%'"
iquest "For %-12.12s size is %s" "SELECT DATA_NAME , DATA_SIZE WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest "SELECT COLL_NAME WHERE COLL_NAME like '/cunyZone/home/%' AND USER_NAME like '<userid>'"
iquest "User %-6.6s has %-5.5s access to file %s" "SELECT USER_NAME, DATA_ACCESS_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>'"
iquest " %-5.5s access has been given to user %-6.6s for the file %s" "SELECT DATA_ACCESS_NAME, USER_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<userid>>'"
iquest no-distinct "select META_DATA_ATTR_NAME"
iquest "select COLL_NAME, DATA_NAME WHERE DATA_NAME like 'myfile%'"
iquest "User %-9.9s uses %14.14s bytes in %8.8s files in '%s'" "SELECT USER_NAME, sum(DATA_SIZE),count(DATA_NAME),RESC_NAME"
iquest "select sum(DATA_SIZE) where COLL_NAME = '/cunyZone/home/<userid>'"
iquest "select sum(DATA_SIZE) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select sum(DATA_SIZE), RESC_NAME where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select order_desc(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select count(DATA_ID) where COLL_NAME like '/cunyZone/home/<userid>%'"
iquest "select RESC_NAME where RESC_CLASS_NAME IN ('bundle','archive')"
iquest "select DATA_NAME,DATA_SIZE where DATA_SIZE BETWEEN '100000' '100200'"

=== Sharing data ===

Access to the data can be controlled via 'ichmod' command. It's
behavior is similar to UNIX 'chmod' command. For example if there is a
need to provide user '''<userid>''' with read access to file
'''myProject/myfile1''' execute the following command:
ichmod read <userid1> myProject/myfile1

To see who has access to a file/directory use:
# ils -A myProject/myfile1
/cunyZone/home/<userid>/myProject/myfile1
ACL - <userid1>
#cunyZone:read object <userid>#cunyZone:own

In the above example user '''<userid1>''' has read access to the file and
user '''<userid>''' is an owner of the file.

Possible levels of access to a data object are null/read/write/own.

 

==Backups==

'''Backups.''' '''/global/u''' user directories and '''SR1''' Project files are backed up automatically to a remote tape silo system over a fiber optic network. Backups are performed daily.

If the user deletes a file from '''/global/u''' or '''SR1''', it will remain on the tape silo system for 30 days, after which it will be deleted and cannot be recovered. If a user, within the 30 day window finds it necessary to recover a file, the user must expeditiously submit a request to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu].

Less frequently accessed files are automatically transferred to the HPC Center robotic tape system, freeing up space in the disk storage pool and making it available for more actively used files. The selection criteria for the migration are age and size of a file. If a file is not accessed for 90 days, it may be moved to a tape in the tape library – in fact to two tapes, for backup. This is fully transparent to the user. When a file is needed, the system will copy the file back to the appropriate disk directory. No user action is required.
 

==Data retention and account expiration policy==

Project directories on SR1 are retained as long as the project is active. The HPC Center will coordinate with the Principal Investigator of the project before deleting a project directory. If the PI is no longer with CUNY, the HPC Center will coordinate with the PI’s departmental chair or Research Dean, whichever is appropriate.

For user accounts, current user directories under /global/u are retained as long as the account is active. If a user account is inactive for one year, the HPC Center will attempt to contact the user and request that the data be removed from the system. If there is no response from the user within three months of the initial notice, or if the user cannot be reached, the user directory will be purged.
 

==DSMS Technical Summary==

[[Image:dsms-summary.png]]

• '''SR1''' is tuned for high bandwidth, redundancy, and resilience. It is not optimal for handling large quantities of small files. If you need to archive more than a thousand of files on '''SR1''', please create a single archive using '''tar'''.

• A separate '''/scratch/<userid>''' exists on each system. On PENZIAS, SALK, KARLE, and ANDY, this is a Lustre parallel file system, on HERBERT it is NFS. These '''/scratch''' directories are visible on the login and compute nodes of the system only and on the data transfer nodes, but are not shared across HPC systems.

• '''/scratch/<userid>''' is used as a high performance parallel scratch filesystem, for example, temporary files (e.g. restart files) should be stored here.

• There are no quotas on '''/scratch/<userid>''', however any files older than 2 weeks are automatically deleted. Also, a cleanup script is scheduled to run every two weeks or whenever the /scratch disk space utilization exceeds 70%. Dot-files are generally left intact from these cleanup jobs.

• /scratch space is available to all users. If the scratch space is exhausted, jobs will not be able to run. Purge any files in '''/scratch/<userid>''', which are no longer needed, even before the automatic deletion kicks in.

• '''/scratch/<userid>''' directory may be empty when you login, you will need to copy any files required for submitting your jobs (submission scripts, data sets) from '''/global/u''' or from '''SR1'''. Once your jobs complete copy any files you need to keep back to '''/global/u''' or '''SR1''' and remove all files from /scratch.

• Do not use '''/tmp''' for storing temporary files. The file system where '''/tmp''' resides in memory is very small and slow. Files will be regularly deleted by automatic procedures.

• '''/scratch/<userid>''' is not backed up and there is no provision for retaining data stored in these directories.
 

==Good data handling practices==
===DSMS, i.e., /global/u and SR1===

• The '''DSMS''' is not an archive for non-HPC users. It is an archive for users who are processing data at the HPC Center. “Parking” files on the '''DSMS''' as a back-up to local data stores is prohibited.

• Do not store more than 1,000 files in a single directory. Store collections of small files into an archive (for example, tar). Note that for every file, a stub of about 4MB is kept on disk even if the rest of the file is migrated to tape, meaning that even migrated files take up some disk space. It also means that files smaller than the stub size are never migrated to tape because that would not make sense. Storing a large number of small files in a single directory degrades the file system performance.
 

===/scratch===

• Please regularly remove unwanted files and directories and avoid keeping duplicate copies in multiple locations. File transfer among the HPC Center systems is very fast. It is forbidden to use "touch jobs" to prevent the cleaning policy from automatically deleting your files from the '''/scratch''' directories. Use '''tar -xmvf''', not '''tar -xvf''' to unpack files. '''tar -xmvf''' updates the times stamp on the unpacked files. The '''tar -xvf''' command preserves the time stamp from the original file and not the time when the archive was unpacked. Consequently, the automatic deletion mechanism may remove files unpacked by '''tar –xvf''', which are only a few days old.

Dsms - Revision history

James: Text replacement - "[pP][bB][sS]" to "SLURM"