skip to primary navigationskip to content

Cambridge Service for Data Driven Discovery

University of Cambridge

Studying at Cambridge

 

Filesystems

Summary of available filesystems

Directory Explanation Backed up? Disk quotas? Files removed?
/home/<username>/   User home directory* Yes (hourly snapshots) Yes (40GB) No
/rds/user/<username>/hpc-work   HPC work user directories* No

Yes (1TB and 1 Million Files )

No automatic deletion currently
/local/   Per-node local disk** No Limited by partition size (57GB or 131GB) Deleted after job completion
/ramdisks/   Per-node tmpfs (in-memory)** No Limited to 50% node RAM Deleted after job completion
/usr/local/ Cluster-wide software Yes N/A (not for user storage) N/A (not for user storage)

*These are shared Lustre filesystems - files created on these are immediately visible to all nodes.

**These are private for each compute node - files created on these are destroyed after completion of the current job.

The /home directories are provided by an enterprise NFS storage system, whereas the /rds directories are provided by Lustre filesystems. The latter are performant and scalable, and it is on these that active data to be read or written by jobs should be placed (not under /home which is more suited to code, postprocessed results, reports etc).

Quotas

Each user has a limit on the amount of file storage he/she can use in the home directory, via quotas. Quotas on /home are currently set at 40 GB per user. Use the command quota to list your current usage - this reports for /home, /rds/user/<username>hpc-work and any RDS projects you may have. Current usages appear under the GiBytes heading (1GiByte=1024*1024KB). A * symbol next to this value indicates that you have met or exceeded your quota (listed in GiBytes under quota), and will need to reduce usage as soon as possible to below the quota value. 

There are 1TB quotas in force on /rds/usr/<username>/hpc-work. For some filesystems, the limit value is slightly greater than the quota value (the former is also known as the hard limit, whereas the latter is the soft limit). It is possible (but not desirable) to exceed the soft limit, but not the hard limit. Exceeding the soft limit is allowed for 7 days, but if the over-quota condition is not corrected after 7 days, creation of new files and directories on the filesystem will become impossible. This happens immediately the soft limit is met if the hard and soft limits are equal. It is essential to resolve this situation as quickly as possible to prevent job errors and (in the case of /home) more subtle malfunctions of the account.

Additionally /rds/usr/<username>/hpc-work has a quota limit on the number of files that can be created, which is limited to 1 Million files.

More RDS storage can be purchased - full details of this service are accessible here.

Backups

ZFS snapshots of /home are being made hourly, daily and weekly. The snapshots are replicated to a second site, but note that the number retained falls off with time (for example the daily snapshots are retained for 2 weeks). Thus we cannot always retrieve deleted files (particularly short-lived files). Hence please be careful when deleting! If you need to retrieve a lost or damaged file, you should have the full pathname, and know, as closely as possible, when the file was damaged and when the file was created. With this information it is possible to search /home/.zfs/snapshot and browse the best snapshot for a good version of the file (which can be copied back to your home directory). For example,

 /home/.zfs/snapshot/snap-hourly-1-2017-11-22-122003/abc123

contains an image of the /home/abc123 directory as it was at 12:20 on 22nd November 2017.

No backups are made of data in the RDS directories.

The /usr/local directory

A number of system-wide applications and libraries are installed under /usr/local/. There are several important subdirectories:

/usr/local/Cluster-Apps and /usr/local/software

These directories contain cluster applications and libraries. Older applications are stored under /usr/local/Cluster-Apps. Each application is installed under its own directory & version, and usually has an 'Environment Module' file. Modules are used for setting up compile-time and run-time environments conveniently, and for switching between different versions of the same application/library in a flexible way.

/usr/local/Cluster-Config

This directory contains module files, licenses for cluster software (e.g. Intel compilers), and configuration files for specific cluster software packages (e.g. scheduler software).

/usr/local/Cluster-Docs/SLURM

This directory contains template job submission scripts.