6. Best Practices for Svante Use¶

This section aims to address some of the most common questions about Svante usage.

6.1. Use of login node (aka “head node”) vs. file servers vs. compute nodes¶

6.1.1. Login node usage¶

The main purpose of login nodes svante-login.mit.edu and svante.mit.edu is precisely that: it allows one to log in (see How to log into Svante) and “look around”, or submit or monitor jobs in SLURM, and acts as a gateway to interactive sessions on file servers or compute nodes. Proper uses of the login node include listing directories and looking at files, copying small files, editing, submitting jobs (by you, not by an AI agent; see Section 6.8), and running simple scripts (requiring trivial processing).

Do not run RAM-intensive jobs on the login node, nor do massive copying or deleting of files, nor tasks which use many cores or access many files. DO NOT RUN PYTHON, MATLAB ETC. ON THE LOGIN NODE. Such use will incur the wrath of the Systems Administrator and Executive Director, and other cluster users who might be frustrated by how slowly the login node is responding to simple typed commands. Most work on svante should occur while logged into a file server or on a compute node; see below. There is no local disk space on the login node (/home is presently housed on fs01).

Improper use of the login node will not be tolerated.

6.1.2. File server usage¶

The main purpose of file servers is to store model output, large data sets, and/or any large output related to research projects. As such, the following computational activities are appropriate while logged into file servers:

Copying and deleting files and directories - Ideally, massive copying or deleting of files should be done on the local machine where disks are attached. If copying a large directory from file server A to B, ssh first to A where you can access the disks locally as /d0, for example (as discussed in Section 2.3), thus only requiring a remote /net/ pathway for the target specification.
Downloading external data - If you are downloading external data onto a file server, when logged into the external server please scp (or use your file transfer protocol of choice) directly to our machines svante2.mit.edu-svante12.mit.edu, rather than through the login node. For external data that might be of use to multiple groups (e.g., reanalysis or CMIP data), contact us about storing the data on fs01 (i.e., to store on /net/fs01/data/). Note: at present svante4, svante6, svante9, and svante12 have 10g connections to the external world, versus 1g on the other servers. Your data transfer rate will likely be much faster if you scp to one of the 10g servers.

At this time Svante is not set up as a Globus endpoint, but users can follow these steps using Globus Connect Personal:
1. wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
2. tar xzf globusconnectpersonal-latest.tgz
3. cd globusconnectpersonal-3.3.1/
4. ./globusconnectpersonal -setup (Follow the steps prompted here to set it up)
5. ./globusconnectpersonal -start &
Then when you are done transferring the data you can use

./globusconnectpersonal -stop
Data analysis using Python, MATLAB, GrADS, fortran analysis codes, etc. - Any interactive data analysis (i.e. opening, reading, processing, and/or writing new data files) should typically occur while logged into one of the file servers, where the data are located, particularly if the analysis involves heavy file input/output. Accessing data and files on disks locally is MUCH faster and superior than using a remote /net/ pathway. Moreover, the bandwidth for remote /net/ pathway data transfer is limited, and remote file access during periods of heavy cluster usage can result your data analysis routines running very slowly. Note if you must load a massive data set into memory for data analysis, RAM capacity makes fs08 or fs12 your best choice (see Table 2.1); otherwise, for routine computational analysis, use a file server where your supervisor or PI has data stored.
Running small single core or multi-core executables - file servers typically have significant computational horsepower, with 8-24 cores, thus making them a simple way to interactively run a single-core executable without having to bother with SLURM. This can even be done with small MPI or shared-memory jobs using a few cores. The only caveat here is that you are effectively sharing the file server with your fellow group researchers, and they may suffer if you monopolize the server’s cores or RAM; if in doubt, you might want to give them a head’s up.

6.1.3. Compute node usage¶

The main purpose of the compute nodes is to run large model simulations, ensembles of model simulations, ensembles of data analysis scripts, or compile code. Although possible to run small (or short) jobs on file servers interactively, typically we do not want to see people running production jobs on the file server nodes. The SLURM scheduler distributes submitted jobs among our pool of compute nodes; see Section 4 for detailed instructions. It is certainly acceptable however to use a compute node interactively instead of a file server for data analysis (especially if the task involves significant number crunching, as the compute nodes are generally fastest at this); see Section 4.5 on how to get on a compute node interactively for this purpose. Interactive sessions on compute nodes are also recommended for compiling code.

6.2. Where should I compile code for running on Svante?¶

Users should typically compile on the node type where they intend to run (see below for exception regarding compiling on svante.mit.edu). Most compilers are smart enough to query the local machine type and will tailor the compilation toward that specific architecture (note, one can usually also force machine-specific compilations using compiler options). What we’ve found is that executables compiled on the oldest machines will often run on anything newer, but compilations on the newest machine only execute there, giving ‘illegal instruction’ on older machines. There are potential speed boosts by compiling on the node type where you plan to run, although how much, if any, is unclear and likely depends on the code. Our advice: if you have a big MPI job that you’d only run on the EDR nodes, then compile it on an EDR node. Otherwise, compile jobs with intent to run on EDR or FDR on a FDR node. For something to run on HDR, most typically it must be compiled on HDR.

Do not compile on svante-login.mit.edu. We’ve even purposely broken some compilers on svante-login; the login node requires extra security and runs a different OS than EDR (or HDR) compute nodes, and thus is problematic for compilation. As a technical note, we very rarely upgrade the OS on compute nodes (say, once a decade); this makes it much easier to maintain working codes over long time periods.

Compiling code to run on HDR and HDR200 nodes is ok however on svante.mit.edu; This login node is configured exactly as a HDR or HDR200 node, albeit quite limited in both speed and RAM. Note that resulting executables are identical whether compiled on svante.mit.edu. a HDR node, or a HDR200 node.

What about compiling for file servers? If you intend to run code on a file server, first suggestion would be to compile on a FDR node and see if it runs on the file server. If it doesn’t, then try to compile and run on the file server where you intend to run the code. You may need assistance from svante-admin@mit.edu to get this working properly (computing environment and operating system varies across file server nodes).

Many of the models we run on Svante are fundamentally chaotic, and as such, obtaining reproducible output is usually only possible if one uses the exact same executible on identical hardware. Almost any variation will result in different output; this may just be in the insignificant digits of a double precision variable, or more significant. Because our compute nodes are not directly accessible through the internet (and thus don’t require frequent security updates), to the extent possible we “freeze” the operating system and computing environment so that machine-precision reproduction is possible. However, at some point in time upgrades are unavoidable. If obtaining machine-precision reproducible results over long time periods is critical to your work, you should discuss your requirements with us, as we cannot guarantee the duration we can maintain a frozen compute node environment.

6.3. Which compute nodes should I use: HDR, HDR200, EDR, or FDR?¶

Svante’s compute nodes are organized into partitions by InfiniBand (IB) generation (see Table 2.1). The partitions are meant for different kinds of work, and sticking to that helps keep the cluster functional for everyone.

Partition	Intended use	Rules
HDR / HDR200 (48 cores/node)	Multi-node MPI jobs get priority. Parallel, single-node jobs (e.g., Python or MATLAB using many cores) are also welcome (preferentially on HDR, not HDR200).	Whole nodes only. Request all 48 cores on each node. Nodes are not shared between users.
EDR (32 cores/node) and FDR (16 cores/node)	General use: single-core and multi-core jobs, ensembles of single-core jobs, Python, MATLAB, GEOS-Chem and other shared-memory runs.	You may request one core or many. Several users’ jobs can share a node, so request only the cores and memory you need.

HDR and HDR200. These are our fastest nodes, with the fastest interconnect speed, and we want them used for what the interconnect is for: large MPI jobs spanning several nodes. HDR nodes are allocated whole, thus never shared across multiple users. Request all 48 cores per node, for example:

#SBATCH -p hdr
#SBATCH -N 2 --ntasks-per-node=48

Do not run small jobs on HDR nodes, and don’t use a few cores while leaving the rest idle. Note that jobs cannot span HDR and HDR200 partitions, and that HDR/HDR200 nodes run a different Linux version from EDR/FDR, so code usually needs to be recompiled (see Section 6.2).

EDR and FDR. These nodes are for everything else. Single-core jobs, small parallel jobs, and ensembles belong here, and we strongly suggest (and in fact insist) that ensembles of single-core jobs run here and not on HDR. Because multiple users share these nodes, request only what you need (see Section 4.2). Some applications, including Python and MATLAB, can effectively ignore their SLURM allocation; if you aren’t sure how many cores your job will use in practice, ask us.

Choosing between EDR and FDR. EDR is faster than FDR in both CPU and IB speed. If your job is a MPI job with a lot of communication between nodes, prefer EDR (or use HDR). If it spends most of its time on computation, FDR is only a slight disadvantage. For shared-memory codes such as GEOSChem classic, the core count per node limits you: 32 on EDR or 16 (c041-c060) on FDR.

6.4. Where should I write model output?¶

If you know in advance that you need to keep run output for an extended period, it may make sense to write run output directly to your file server space although beware of possible bottlenecks with too many people attempting to access disks on a single file server at one time. We’ve also noticed that a single user copying many streams of data to and from a file server can effectively grind all other access on this server to a near standstill, so be forewarned. Using local /scratch space on the compute nodes for run output is strongly encouraged, especially if run output need not be saved permanently. If you are writing large amounts of data, e.g. 1 TB or more, give svante-admin@mit.edu a head’s up and we’ll clean out old files on /scratch to ensure sufficient space.

Do not write model output to /home, or write other data there for temporary storage. Besides the obvious consideration that /home is rather limited in size (500GB per user), the offsite backup of /home is run daily, and if folks are constantly deleting and repopulating their home spaces, the backup is unlikely to complete its daily cycle, and thus fail to properly backup many users.

6.5. Limits on compute usage¶

To keep Svante usable for everyone, each user’s running jobs are limited as follows:

Core usage limits. Core limits are separate for different partitions:

Table 6.1 Maximum running cores per user¶
Partition	Maximum running cores per user
EDR	320 cores (10 full nodes)
HDR and HDR200 (combined)	576 cores (12 full nodes)
FDR	No limit

The EDR and HDR/HDR200 limits are counted separately, and FDR usage does not count toward either one. These limits may change depending on current cluster usage.

Walltime limits. The maximum walltime for a single job depends on the partition:

Table 6.2 Maximum walltime by partition¶
Partition	Maximum walltime
EDR	4 days
HDR	5 days
HDR200	7 days
FDR	No limit

Walltime limits are subject to change. If a specific job needs a longer walltime, contact the Executive Director after you submit it. Long jobs should also write restart files regularly, so a job that hits its limit (or a node failure) can be continued by a follow-up job and not started over.

These limits are a ceiling, not a target. Using the full core limit is more than your fair share of a cluster this size. If you need to run at or near this limit for an extended period, please talk to the Executive Director first. We can often help you plan, suggest the best partition, or arrange a reservation (see below).

There is no limit to storage on file server spaces other than the disk partition size itself. That being said, individual users and research groups are responsible for monitoring their file server space usage. Ideally, disks should not go above 95-97% capacity; in fact, disk speed performance starts to decline somewhere > 90% filled. Filling up a file server disk 100% is painful, i.e. costly in terms of tech support hours. To check free disk space: df -h /d0 to get information on the /d0 partition when logged into a file server, for example.

In general, an individual’s accountability for using the cluster “responsibly” is proportional to the magnitude of their computational task. In other words, for small jobs on few cores using few files, the repercussions of running in a clumsy manner are usually minimal, whereas a poorly conceived setup for huge jobs with copious output can effectively make the cluster a miserable user environment for all cluster users. If you are unsure about any significant undertaking, please consult with the Executive Director beforehand; if necessary, they will consult with our tech support for the best way to set up your work.

6.6. Reservations¶

A reservation sets aside specific compute nodes for a specific purpose and time. Reservations are created only by the system administrators; to request one, contact the Executive Director. There are two kinds.

Production run reservations. Certain research groups have priority status on Svante, and for them a reservation (typically 4-12 nodes, either created for an individual or the full group) is available for running large, time-consuming ensembles of runs.

The ensemble must be set up and tested before the reservation becomes active. A reservation is for production runs, not for debugging the setup.
Reserved nodes that sit idle for more than several hours are released from the reservation and returned to the general pool, so have your jobs fully tested and ready to start when the reservation begins.

Short-term debugging reservations. Any user may request a short-term reservation, on a case-by-case basis, for debugging. These are granted during periods when requests for compute nodes are queued (that is, when debugging in the regular queue would be slow).

Once a reservation is in place, you will be told its name. Submit your jobs to it by adding #SBATCH --reservation=«reservation_name» to your batch script (or --reservation=«reservation_name» on the srun/sbatch command line).

6.7. Courtesy and common sense¶

Svante is a small, shared facility. It works because users look out for one another and communicate within their research group, not because of technical limits. There are limited checks stopping one user from filling a file server, flooding the queue, or making the login node unusable, so we rely on you to use good judgment. In practice:

Use the right node for the job. The login node is for looking around, editing, and managing jobs. File servers are great for work using data stored locally (i.e., on that server). Compute nodes are for heavy computation. See Section 6.
Start small, then scale up. Test a new script on one case or a short run before launching an ensemble or a large MPI job. Check the output before you commit hundreds of core-hours.
Throttle your ensembles. When you submit array jobs, use the %N option (see array jobs) so you don’t overwhelm the cluster. This is preferable to looping sbatch in a script.
Ask for what you need, no more. Give an honest walltime estimate. Don’t take a full node for a one-core job, and don’t request more RAM than the job needs.
Watch your jobs. If a job is failing, hung, or producing garbage, cancel it. Don’t let it, or a script that resubmits it, run unattended.
Be gentle with shared storage. A single user copying or deleting huge amounts of data can bring a file server to a near standstill for everyone using it. Do big file operations on the server that holds the disks, not over /net/ paths. Avoid scanning large directory trees (find, du, ls -R) during busy periods. Keep an eye on disk usage, and give us a heads-up before writing 10 TB or more.
Keep /home quiet. Don’t use it for temporary storage, large installs, or anything you will repeatedly delete and recreate. Heavy churn can keep the daily backup from finishing.
Think before you delete or overwrite. Check exactly what a command will touch, especially ones that use wildcards, rm -r, or chmod -R. Deleted data cannot always be recovered.
Give a heads-up. If you are about to use a large share of a file server, warn your group. If you are about to do something unusual or large, ask the Executive Director first.
Clean up after yourself. Remove scratch files, stray processes, and abandoned interactive or screen sessions.

A good rule of thumb is: if you would hesitate to do something on a shared machine without telling your colleagues, tell them first.

6.8. AI agents and coding assistants¶

AI-assisted tools (for example, coding assistants in VS Code) can be useful for editing files, writing scripts, setting up environments, and analyzing output, and you are welcome to use them. These tools have no sense of courtesy, no feel for what is costly on a shared machine, and no stake in whether other users can get their work done. Courtesy and common sense is your job.

You are responsible for everything an AI agent does under your account. Treat its actions as your own. In particular:

Agents may not submit, modify, or cancel SLURM jobs (sbatch, srun, salloc, scancel, scontrol update). Review the script and the resource request yourself, then submit it yourself. Read-only monitoring such as squeue, sinfo, and seff is fine.
Agents must follow every guideline in this section, including those on the login node, file servers, /home, and /scratch. For example, an agent should not run Python or compile on svante-login, scan /net/ paths, start large copies or downloads, or write large files to /home.
No destructive commands without your explicit approval. This includes deleting, overwriting, moving or reorganizing directories, and recursive changes to permissions (including in public_html).
Supervise the agent. Review the commands it proposes before they run. Don’t leave agents running unattended or in a loop, and don’t give them your SSH keys or credentials. If an agent starts a long-running process, you are responsible for stopping it.
Be careful with remote-development tools. Remote sessions (for example, VS Code over SSH) start server processes and file watchers on the node you connect to. Avoid connecting to svante-login this way, and exclude large data directories from file watching and indexing.

Violations by an agent are treated as violations by the account holder, and may result in suspension or termination of your account. If you are unsure whether something is acceptable, ask us first.