Using scratch space
Each cluster is equipped with several file systems that can be used to store files, for instance:
- the home file system
- a global scratch file system
- a local temporary file system
- a long-term storage file system
The home filesystem is dedicated to source code, configuration file, and small datasets. It is the working directory you land in when you connect (its full path can be shown with echo ~, and you can return there with simplr cd.) It is often built on reliable, well-proven technologies, and often restricted in terms of quota. The status of your disk quota and usage is often shown in the welcome screen when you connect. Alternatively, you can find about it by issuing the quota -s command. Long-term storage is also built on stable technologies, but quota there are less limited. The downside is that the long-term storage is often not directly connected to the cluster so it can be quite slow.
By contrast, a scratch filesystem is dedicated to storing large amounts of data in a very fast way. It uses more powerful technologies but at the expense of stability: data stored on a scratch filesystem can disappear without notice, and sometimes does! Consequently, data copied on a scratch filesystem should be temporary (for the duration of a job) and non-critical (either reconstructible or copied elsewhere.)
Scratch filesystems come in two flavours: global or local.
Global scratch file system.
A global scratch is a scratch space that is common to all compute nodes and to the frontend. On the CÉCI clusters, they are available through the $GLOBALSCRATCH environment variable. It is often built using a fast parallel filesystem such as Lustre, GPFS, or FraunhofferFS.
To use a global scratch, you might first need to create a directory by yourself. It is then common to name it after your login. On some clusters, it might already be created for you. Anyway, it is always safe to try to create it with
mkdir -p $GLOBALSCRATCH
on any CÉCI cluster.
Then you can copy there large input files if required. After that, it is custom in the job script to create a subdirectory with the job id, where all temporary data will be written, and to clean that directory when the job finishes, and after having copied results to another location either on the same filesystem if the results are to be consumed by a later job, or on another filesystem such as the home or a remote long-term storage.
Local scratch file system
By contrast to a global scratch, a local scratch is only visible from within the compute node it belongs to. On the CÉCI clusters, they are available through the $LOCALSCRATCH environment variable. Often they are built on top of a fast, but redundancy-lacking, RAID-0.
To use a local scratch, you will need to create a directory for your job in the local scratch space. It is custom to name it after the jobid:
mkdir -p $LOCALSCRATCH/$SLURM_JOB_ID
There you can write temporary results during your job, copy the results of interest back to the home directory, and then delete it at the end of the job script.
Final remarks
As scratch spaces are not meant to store data in the long term, you should expect them to be cleaned automatically after some time. This does not mean that you should not do it yourself though: always clean up after your job. This is especially important after a job crashes before the cleaning operations in the submission script had a chance to run.
You need to be careful about filenaming when using a parallel job to avoid overwriting files by two different parts (threads, or processes) of the program, especially on global filesystems.
Even if your data is small and the quota allow you working in your home directory, you should consider using scratch spaces as they are in general much faster. The global filesystems are much faster because they handle requests and store data in parallel, while the local filesystems are faster as they do not need to access the network and they are used by far less jobs at a time.