View source for Data Storage and Management System

This section describes the use of Data Storage Management System (DSMS) which serves only Penzias, Karle and Appel. This file system is separate from Global Hybrid File System (GHFS) which serves Arrow and all Condominium servers. 

__TOC__

Key features of the DSMS system include:

:•	'''User''' home directories in a standard Unix file system called /global/u.
:•	Enhanced parallel scratch space on the HPC systems.
:•	'''Project''' directories in an Integrated Rule-Oriented Data-management System (iRODS) managed resource.  Project directories exist in a “virtual file space” called '''cunyZone''' which contains a resource called '''Storage Resource 1 (SR1)'''.    For the purpose of this document, we will use the terminology SR1 to describe '''Project file space.'''
:•	Automated backups.


The '''DSMS''' is the HPC Center’s primary file system and is accessible from all existing HPC systems, except for '''HERBERT''' . It will similarly be accessible from all future HPC systems.   

The '''DSMS''' provides a 3-level data storage infrastructure: - '''HOME''' filesystem,  '''SCRATCH''' filesystems, '''SR1''' (long-tern storage resource)

'''DSMS''' features are explained below.


=="Home" directories are on <font face="courier">/global/u</font>==
'''<font face="courier">/global/u</font>''' is a standard Unix file system that holds the home directories of individual users. When users request and are granted an allocation of HPC resources, they are assigned a '''<font face="courier"><font color="red"><userid></font color></font>''' and a 50 GB allocation of disk space for home directories on '''<font face="courier">/global/u/<font color="red"><userid></font color></font>'''. These '''home''' directories are on the '''DSMS''', not on the HPC systems, but can be accessed from any Center system. All home directories are backed up on weekly basis.

==<font face="courier">/scratch</font>==
Disk storage on the HPC systems is used only for '''scratch''' files.  '''scratch''' files are temporary and are '''<font color="red">not backed up</font color>'''.  '''<font face="courier">/scratch</font>''' is used by jobs queued for or in execution.  Output from jobs may temporarily be located in '''<font face="courier">/scratch</font>'''.  

In order to submit a job for execution, a user must '''stage''' or '''mount''' the files required by the job to '''<font face="courier">/scratch</font>''' from '''<font face="courier">/global/u</font>''' using UNIX commands and/or from '''<font face="courier">SR1</font>''' using '''iRODS''' commands.

Files in '''<font face="courier">/scratch</font>''' on a system are '''automatically purged''' when (1) usage reaches 70% of available space, or (2) file residence on scratch exceeds two weeks, whichever occurs first.



==“Project” directories==

'''“Project”''' directories are managed through '''iRODS''' and accessible through iRODS commands, not standard UNIX commands.   In iRODS terminology, a “collection” is the equivalent of “directory”.

A '''“Project”''' is an activity that usually involves multiple users and/or many individual data files.  A '''“Project”''' is normally led by a “Principal Investigator” (PI), who is a faculty member or a research scientist.   The PI is the individual responsible to the University or a granting agency for the “Project”.  The PI has overall responsibility for “Project” data and “Project” data management. To establish a Project, the PI completes and submits the online “Project Application Form”.

Additional information on the '''DSMS''' is available in Section 4 of the User Manual <br />
http://www.csi.cuny.edu/cunyhpc/pdf/User_Manual.pdf



==Typical Workflow==
Typical workflows in are described below:

1. Copying files from a user’s home directory or from '''SR1''' to '''SCRATCH'''.<br />
If working with '''HOME''':<br />
<font face="courier">
   cd /scratch/<font color="red"><userid></font color>
   mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>
   cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/a.out ./
   cp /global/u/<font color="red"><userid></font>/<font color="red">myProject</font>/<font color="red">mydatafile </font>./
</font>

If working with '''SR1''':<br />
<font face="courier">
   cd /scratch/<font color="red"><userid></font color>
   mkdir <font color="red">mySLURM_Job</font color> && cd <font color="red">mySLURM_Job</font color>
   iget <font color="red">myProject</font color>/a.out 
   iget <font color="red">myProject</font color>/<font color="red">mydatafile</font color>
</font>


2. Prepare SLURM job script. Typical SLURM sript is similar to the following:<br />
<font face="courier">
   #!/bin/bash 
   #SBATCH --partition production 
   #SBATCH -J test 
   #SBATCH --nodes 1 
   #SBATCH --ntasks 8 
   #SBATCH --mem 4000
   echo "Starting…" 

   cd $SLURM_SUBMIT_DIR
   mpirun -np 4 ./a.out ./<font color="red">mydatafile</font color> > <font color="red">myoutputs</font color>
   echo "Done…"
</font>
Your SLURM may be different depending on your needs. Read section Submitting Jobs for a reference.<br />


3. Run the job <br />
<font face="courier">
   sbatch ./<font color="red">mySLURM_script</font color>
</font>


4. Once job is finished, clean up '''SCRATCH''' and store outputs in your user home directory or in '''SR1'''.<br />

If working with '''HOME''':<br />
<font face="courier">
   mv ./myoutputs /global/u/<font color="red"><userid></font color>/<font color="red">myProject</font color>/.
   cd ../
   rm -rf <font color="red">mySLURM_Job</font color>
</font>

If working with '''SR1''':<br />
<font face="courier">
   iput ./myoutputs <font color="red">myProject</font color>/. 
   cd ../
   rm -rf <font color="red">mySLURM_Job</font color>
</font>


5. If output files are stored in '''SR1''' tag them with metadata.<br />
<font face="courier">
   imeta addw -d myoutput zvalue 15 meters
   imeta addw -d myoutput colorLabel RED
</font>
<br />

== iRODS ==

'''iRODS''' is the integrated Rule-Oriented Data-management System, a
community-driven, open source, data grid software solution. '''iRODS''' is
designed to abstract data services from data storage hardware and
provide users with hardware-agnostic way to manipulate data. 


'''iRODS''' is a primary tool that is used by the CUNY HPCC users to
seamlessly access 1PB storage resource (further referenced as '''SR1'''
here) from any of the HPCC's computational systems.

Access to '''SR1''' is provided via so-called '''i-commands''':

<pre>iinit
ils
imv</pre>

Comprehesive list of i-commands with detailed description can be
obtained at [https://wiki.irods.org/index.php/icommands iRODS wiki].


To obtain quick help on any of the commands while being logged into
any of the HPCC's machines type <font face="courier">'''i-command -h'''</font>. For example:
<pre>
ils -h
</pre>
  
Following is the list of some of the most relevant '''i-commands''':
  
:'''<font face="courier">iinit</font>''' -- Initialize session and store your password in a scrambled form for automatic use by other icommands.
:'''<font face="courier">iput</font>''' --  Store a file
:'''<font face="courier">iget</font>''' --  Get a file
:'''<font face="courier">imkdir</font>''' --  Like mkdir, make an iRODS collection (similar to a directory or Windows folder)
:'''<font face="courier">ichmod</font>''' --  Like chmod, allow (or later restrict) access to your data objects by other users.
:'''<font face="courier">icp</font>''' --  Like cp or rcp, copy an iRODS data object
:'''<font face="courier">irm</font>''' --  Like rm, remove an iRODS data object
:'''<font face="courier">ils</font>''' --  Like ls, list iRODS data objects (files) and collections (directories)
:'''<font face="courier">ipwd</font>''' --  Like pwd, print the iRODS current working directory
:'''<font face="courier">icd</font>''' --  Like cd, change the iRODS current working directory
:'''<font face="courier">ichksum</font>''' --  Checksum one or more data-object or collection from iRODS space.
:'''<font face="courier">imv</font>''' --  Moves/renames an irods data-object or collection.
:'''<font face="courier">irmtrash</font>''' --  Remove one or more data-object or collection from a RODS trash bin.
:'''<font face="courier">imeta</font>''' --  Add, remove, list, or query user-defined Attribute-Value-Unit triplets metadata
:'''<font face="courier">iquest</font>''' --  Query (pose a question to) the ICAT, via a SQL-like interface


Before using any of the i-commands users need to identify themselves to the iRODS server running command
<pre>
# iinit
</pre>
and providing HPCC's password. 

Typical workflow that involves operations on files stored in SR1
include storing/getting data to and from SR1, tagging data with 
metadata, searching for data, sharing (setting permissions). 

==== Storing data to SR1 ====
 
1. Create '''iRODS''' directory (aka 'collection'):<br />
<font face="courier">
   # imkdir <font color="red">myProject</font color>
</font>
2. Store all files <font face="courier">''''myfile*''''</font face> into this directory (collection):<br />
<font face="courier">
   # iput -r <font color="red"><userid> myfile* myProject</font color>/.
</font>
3. Verify that files are stored:<br />
<font face="courier">
   # ils
   /cunyZone/home/<font color="red"><userid></font color>:
   C- /cunyZone/home/<font color="red"><userid></font color>/<font color="red">myProject</font color>
   # ils <font color="red">myProject</font color>
   /cunyZone/home/<font color="red"><userid></font color>/<font color="red">myProject</font color>:
      myfile1
      myfile2
      myfile3
</font>
   
Symbol '''<font face="courier">'C-'</font>''' in the beginning of output of '''<font face="courier">'ils'</font>''' shows that listed item is a collection.

4. Combining '''<font face="courier">'ils', 'imkdir', 'iput', 'icp', 'ipwd', 'imv''''</font> user can create iRODS directories and store files in them similarly to what is normally done with UNIX commands '''<font face="courier">'ls', 'mkdir', 'cp', 'pwd', 'mv''''</font> etc...

==== Getting data from SR1 ====

1. To copy file from SR1 to current working directory run
   # iget <font color="red">myProject</font color>/<font color="red">myfile1</font color>

2. Now listing current working directory should reveal '''myfile1''':
   # ls
   <font color="red">myfile1</font color>

3. Instead of individual files the whole directory (with
sub-directories) can be copied with ''''<font face="courier">-r</font>'''' flag (stands for
'recursive')
   # iget -r <font color="red">myProject</font color>


NOTE: wildcards are not supported, therefore the command below <u>will not work</u>
   # iget <font color="red">myProject</font color>/<font color="red">myfile</font color>*

=== Tagging data with metadata ===
   
iRODS provides users with extremely powerful mechanism of managing
data with metadata. While working with large datasets it's
sometimes easy to forget what is stored in this or the other file.
Metadata tags help organizing data in a very easy and reliable
manner.

Let's tag files from previous example with some metadata:
<pre>
# imeta add -d myProject/myfile1 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile1 comment "This is file number 1"
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 zvalue 10 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 colorLabel RED
AVU added to 1 data-objects
# imeta add -d myProject/myfile2 comment "This is file number 2"
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 zvalue 15 meters
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 colorLabel BLUE
AVU added to 1 data-objects
# imeta add -d myProject/myfile3 comment "This is file number 3"
AVU added to 1 data-objects
</pre>
Here we've tagged myfile1 with 3 metadata labels:

- zvalue 10 meters

- colorlabel RED

- comment "This is file number 1"
  
Similar tags were added to 'myfile2' and 'myfile3'

Metadata come in form of AVU -- Attribute|Value|Unit. As seen from
the above examples Unit is not necessary. 

Let's list all metadata assigned to file 'myfie1':
<pre>
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: zvalue
value: 15
units: meters
----
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
</pre>
 
To remove an AVU assigned to a file run:
<pre>
# imeta rm -d myProject/myfile1 zvalue 15 meters
# imeta ls -d myProject/myfile1
AVUs defined for dataObj myProject/myfile1:
attribute: colorLabel
value: RED
units:
----
attribute: comment
value: This is file number 1
units:
#
#
# imeta add -d myProject/myfile1 zvalue 15 meters
</pre>

Metadata may be assigned to directories as well:
<pre>
# imeta add -C myProject simulationsPool 1
# imeta ls -C myProject
AVUs defined for collection myProject:
attribute: simulationsPool
value: 1
units:
</pre>

Note the '-C' key that is used instead of '-d'.


=== Searching for data ===

Power of metadata becomes obvious when data needs to be found in
large collections. Here is an illustration of how easy this task is
done with iRODS via imeta queries:

 # imeta qu -d zvalue = 15
 collection: /cunyZone/home/<font color="red"><userid></font color>/myProject
 dataObj: myfile1
 ----
 collection: /cunyZone/home/<font color="red"><userid></font color>/myProject
 dataObj: myfile3


We see both files that were tagged with label 'zvalue 10 meters'.
Here is different query:
 
 # imeta qu -d colorLabel = RED
 collection: /cunyZone/home/<font color="red"><userid></font color></myProject
 dataObj: myfile1
 ----
 collection: /cunyZone/home/<font color="red"><userid></font color>/myProject
 dataObj: myfile2


Another powerful mechanism to query data is provided with 'iquest'. 
Following is a number of examples that show 'iquest' capabilities:
 
 iquest "SELECT DATA_NAME, DATA_SIZE WHERE DATA_RESC_NAME like 'cuny%'"
 iquest "For %-12.12s size is %s" "SELECT DATA_NAME ,  DATA_SIZE  WHERE COLL_NAME = '/cunyZone/home/<font color="red"><userid></font color>'"
 iquest "SELECT COLL_NAME WHERE COLL_NAME like '/cunyZone/home/%' AND USER_NAME like '<font color="red"><userid></font color>'"
 iquest "User %-6.6s has %-5.5s access to file %s" "SELECT USER_NAME,  DATA_ACCESS_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<font color="red"><userid></font color>'"
 iquest " %-5.5s access has been given to user %-6.6s for the file %s" "SELECT DATA_ACCESS_NAME, USER_NAME, DATA_NAME WHERE COLL_NAME = '/cunyZone/home/<font color="red"><userid></font color>>'"
 iquest no-distinct "select META_DATA_ATTR_NAME"
 iquest  "select COLL_NAME, DATA_NAME WHERE DATA_NAME like 'myfile%'"
 iquest "User %-9.9s uses %14.14s bytes in %8.8s files in '%s'" "SELECT USER_NAME, sum(DATA_SIZE),count(DATA_NAME),RESC_NAME"
 iquest "select sum(DATA_SIZE) where COLL_NAME = '/cunyZone/home/<font color="red"><userid></font color>'"
 iquest "select sum(DATA_SIZE) where COLL_NAME like '/cunyZone/home/<font color="red"><userid></font color>%'"
 iquest "select sum(DATA_SIZE), RESC_NAME where COLL_NAME like '/cunyZone/home/<font color="red"><userid></font color>%'"
 iquest "select order_desc(DATA_ID) where COLL_NAME like '/cunyZone/home/<font color="red"><userid></font color>%'"
 iquest "select count(DATA_ID) where COLL_NAME like '/cunyZone/home/<font color="red"><userid></font color>%'"
 iquest "select RESC_NAME where RESC_CLASS_NAME IN ('bundle','archive')"
 iquest "select DATA_NAME,DATA_SIZE where DATA_SIZE BETWEEN '100000' '100200'"

=== Sharing data ===

Access to the data can be controlled via 'ichmod' command. It's
behavior is similar to UNIX 'chmod' command. For example if there is a
need to provide user <font face="courier"><font color="red">'''<userid>'''</font color></font> with read access to file
'''<font face="courier">myProject/myfile1'''</font> execute the following command:
   ichmod read <font color="red"><userid1></font color> myProject/myfile1</font>

To see who has access to a file/directory use:
   # ils -A myProject/myfile1
   /cunyZone/home/<font color="red"><userid></font color>/myProject/myfile1
   ACL - <font color="red"><userid1></font color>
   #cunyZone:read object   <font color="red"><userid></font color>#cunyZone:own

In the above example user <font face="courier">'''<font color="red"><userid1></font color>'''</font> has read access to the file and
user '''<font face="courier"><font color="red"><userid></font color></font>''' is an owner of the file. 

Possible levels of access to a data object are null/read/write/own.


<br />

==Backups==

'''Backups.'''	'''<font face="courier">/global/u</font>''' user directories and '''<font face="courier">SR1</font>''' Project files are backed up automatically to a remote tape silo system over a fiber optic network.  Backups are performed daily. 

If the user deletes a file from '''<font face="courier">/global/u</font>''' or '''<font face="courier">SR1</font>''', it will remain on the tape silo system for 30 days, after which it will be deleted and cannot be recovered.   If a user, within the 30 day window finds it necessary to recover a file, the user must expeditiously submit a request to [mailto:hpchelp@csi.cuny.edu hpchelp@csi.cuny.edu].

Less frequently accessed files are automatically transferred to the HPC Center robotic tape system, freeing up space in the disk storage pool and making it available for more actively used files. The selection criteria for the migration are age and size of a file. If a file is not accessed for 90 days, it may be moved to a tape in the tape library – in fact to two tapes, for backup. This is fully transparent to the user. When a file is needed, the system will copy the file back to the appropriate disk directory. No user action is required.
<br />




==Data retention and account expiration policy==

Project directories on SR1 are retained as long as the project is active.  The HPC Center will coordinate with the Principal Investigator of the project before deleting a project directory.  If the PI is no longer with CUNY, the HPC Center will coordinate with the PI’s departmental chair or Research Dean, whichever is appropriate.

For user accounts, current user directories under /global/u are retained as long as the account is active.  If a user account is inactive for one year, the HPC Center will attempt to contact the user and request that the data be removed from the system.  If there is no response from the user within three months of the initial notice, or if the user cannot be reached, the user directory will be purged. 
<br />




==DSMS Technical Summary==
{| class="wikitable"
|+
!File Space
!Purpose
!Accessibility
!Quota
!Backups
!Purges
|-
|Scratch:
/scratch/<userid>
on *PENZIAS, ANDY, SALK, BOB*
|High Performance Parallel scratch filesystems. Work area for jobs, datasets, restart files, files to be pre-/post processed. Temporary space for data that will be removed within a short amount of time.
|Not globally accessible.
Separate /scratch/<userid> exists on each system. Visible on login and compute nodes of each system and on the data transfer nodes.
|None
|None
|Files older than 2 weeks are automatically deleted 
OR
when scratch filesystem reaches 70% utilization
|-
|Home:
/global/u/<userid>
|User home filespace. Essential data should be stored here, such as user's source code, documents, and data structures.
|Globally accessible on the login and on the data transfer nodes through native GPFS or NFS mounts
|Nominally 50 GB
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days.
| rowspan="2" |Not purged
|-
|Project:
/SR1/<PID>
|Project space allocations
|Accessible on the login and on the data transfer nodes. Accessible outside CUNY HPC Center through iRODS.
|Allocated according to project needs
|Yes, backed up nightly to tape. If the active copy is deleted, the most recent backup is stored for 30 days and retrievable on request, but the iRODS metadata may be lost.
|}
•	'''<font face="courier">SR1</font>''' is tuned for high bandwidth, redundancy, and resilience.  It is not optimal for handling large quantities of small files. If you need to archive more than a thousand of files on '''<font face="courier">SR1</font>''', please create a single archive using '''<font face="courier">tar</font>'''.

•	A separate '''<font face="courier">/scratch/<font color="red"><userid></font color></font>''' exists on each system.  On PENZIAS, SALK, KARLE, and ANDY, this is a Lustre parallel file system, on HERBERT it is NFS. These '''<font face="courier">/scratch</font>''' directories are visible on the login and compute nodes of the system only and on the data transfer nodes, but are not shared across HPC systems.

•	'''<font face="courier">/scratch/<font color="red"><userid></font color></font>''' is used as a high performance parallel scratch filesystem, for example, temporary files (e.g. restart files) should be stored here.

•	There are no quotas on '''<font face="courier">/scratch/<font color="red"><userid></font color></font>''', however any files older than 2 weeks are automatically deleted.  Also, a cleanup script is scheduled to run every two weeks or whenever the /scratch disk space utilization exceeds 70%.  Dot-files are generally left intact from these cleanup jobs.

•	/scratch space is available to all users. If the scratch space is exhausted, jobs will not be able to run. Purge any files in '''<font face="courier">/scratch/<font color="red"><userid></font color></font>''', which are no longer needed, even before the automatic deletion kicks in.

•	'''<font face="courier">/scratch/<font color="red"><userid></font color></font>''' directory may be empty when you login, you will need to copy any files required for submitting your jobs (submission scripts, data sets) from <font face="courier">'''/global/u'''</font> or from '''<font face="courier">SR1</font>'''.  Once your jobs complete copy any files you need to keep back to '''<font face="courier">/global/u</font>''' or '''<font face="courier">SR1</font>''' and remove all files from /scratch.

•	Do not use '''<font face="courier">/tmp</font>''' for storing temporary files. The file system where '''<font face="courier">/tmp</font>''' resides in memory is very small and slow. Files will be regularly deleted by automatic procedures.

•	'''<font face="courier">/scratch/<font color="red"><userid></font color></font>''' is not backed up and there is no provision for retaining data stored in these directories.
<br />

==Good data handling practices==
===DSMS, i.e., /global/u and SR1===

•	The '''DSMS''' is not an archive for non-HPC users. It is an archive for users who are processing data at the HPC Center.  “Parking” files on the '''DSMS''' as a back-up to local data stores is prohibited.  

•	Do not store more than 1,000 files in a single directory. Store collections of small files into an archive (for example, tar). Note that for every file, a stub of about 4MB is kept on disk even if the rest of the file is migrated to tape, meaning that even migrated files take up some disk space. It also means that files smaller than the stub size are never migrated to tape because that would not make sense.  Storing a large number of small files in a single directory degrades the file system performance. 
<br />

===/scratch===

•	Please regularly remove unwanted files and directories and avoid keeping duplicate copies in multiple locations. File transfer among the HPC Center systems is very fast. It is forbidden to use "touch jobs" to prevent the cleaning policy from automatically deleting your files from the '''<font face="courier">/scratch</font>''' directories. Use '''<font face="courier">tar -xmvf</font>''', not '''<font face="courier">tar -xvf</font>''' to unpack files.   '''<font face="courier">tar -xmvf</font>''' updates the times stamp on the unpacked files.  The '''<font face="courier">tar -xvf</font>''' command preserves the time stamp from the original file and not the time when the archive was unpacked. Consequently, the automatic deletion mechanism may remove files unpacked by '''<font face="courier">tar –xvf</font>''', which are only a few days old.