Skip to content
Skills Library
Science & research · Learning & explanation

rseng-hpc-computing

/rseng-hpc-computing

Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring

Licence
MIT
Last repository change
23.09.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
2
URLs in SKILL.md
5
Deprecation

No signals detected. Repository changed on 23.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • SKILL.md zawiera komentarze HTML (2). Nie widać ich w podglądzie Markdown, a model je czyta.

What it does and when to use it

Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring

Original author description

Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring

How it works

  1. Batch jobs done right (SLURM)
  2. Software environments on clusters
  3. Parallelism: pick the right kind
  4. Cluster citizenship and data
  5. Working with this skill
  6. Related skills
Original SKILL.md headings · 124 lines, 650 words
SKILL.md preview
---
name: rseng-hpc-computing
description: >-
  Covers working effectively on high-performance computing clusters: writing
  and debugging SLURM job scripts, choosing and requesting resources honestly,
  running containers with Apptainer, using module systems and EESSI software
  stacks, MPI basics, checkpointing, and scaling from laptop to cluster
  reproducibly. Use when the user mentions a cluster, supercomputer, SLURM,
  sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime,
  or when a compute workload has outgrown a single machine.
  (Larger-than-memory data processing with Dask or Spark is
  rseng-big-data-processing; measuring scaling before requesting allocations is
  rseng-performance-profiling.)
license: CC-BY-4.0
metadata:
  version: 0.1.0
---

# Research computing on HPC clusters

HPC clusters are where much research code actually runs, and they
invert desktop habits: you do not run programs, you request resources
and submit batch jobs; you do not install software, you load modules
or bring containers; the login node is a shared hallway, not a
workstation. An agent's job is to translate the user's computation
into this model correctly and reproducibly - and to respect that
every cluster has local documentation that overrides generic advice.

## Batch jobs done right (SLURM)

A job script is a shell script with resource directives:

```bash
#!/bin/bash
#SBATCH --job-name=fit-model
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --output=logs/%x-%j.out

module load Python/3.12
srun python fit_model.py --config configs/run.yaml
```

Practices that matter:

- Request honestly: measure a pilot run, then set time/memory with
  modest headroom. Over-requesting wastes allocation and queue
  priority (and energy - rseng-green-computing); under-requesting
  kills jobs at 99%.
- Never compute on the login node; test with a short interactive
  allocation (srun --pty or salloc) instead.
- Parameter sweeps are job arrays (--array=0-99), not 100 submitted
  scripts; index into a config list with $SLURM_ARRAY_TASK_ID
  (externalized run configs pair naturally with rseng-workflows).
- Long jobs checkpoint: clusters preempt and nodes fail; save
  restartable state at intervals and make the script resume from the
  last checkpoint.
- Log usefully: %x-%j names, and `seff JOBID` after completion to
  compare requested vs used resources - feed that back into the next
  request.

## Software environments on clusters

Three layers, in order of preference for reproducibility:

1. Containers with Apptainer (formerly Singularity): the HPC-native
   container runtime - unprivileged, image-file based, GPU-aware.
   Build once (possibly from a Docker image), run identically on any
   cluster; pairs with rseng-reproducible-environments for the build
   recipe.
2. EESSI: a shared, optimized, ready-to-use scientific software
   stack streamed to any cluster or laptop - when available it gives
   identical software everywhere without building anything.
3. Module system (module avail / module load): the site's curated
   builds; record exact module versions in the job script - an
   unversioned `module load Python` is a reproducibility hole.

Never pip-install into the home directory as the primary strategy;
quota, node-architecture and reproducibility problems follow.

## Parallelism: pick the right kind

- Independent runs (sweeps, bootstraps, per-file processing): job
  arrays or a workflow engine - no MPI needed. This covers most
  research workloads.
- Single-node parallelism: threads/multiprocessing;
  --cpus-per-task, and pin BLAS/OpenMP thread counts to match.
- Multi-node tightly-coupled computation: MPI (mpi4py or native),
  launched with srun; requires the code to be written for it.
  Distributed data processing frameworks are their own path
  (rseng-big-data-processing).

Scaling discipline: measure strong/weak scaling on small counts
before requesting large allocations; doubling cores that yield 1.2x
wastes everyone's allocation
(rseng-performance-profiling for the measurement).

## Cluster citizenship and data

- Filesystems differ: home (small, backed up), project (shared),
  scratch (fast, purged) - stage data to scratch for jobs, move
  results back, and script the staging (rseng-data-management).
- Transfer with rsync (resumable) or the site's data-transfer nodes;
  never leave the only copy of results on scratch.
- Read the site's own docs first; when the user's cluster is known,
  prefer its conventions over anything generic, including this
  skill.

## Working with this skill

This skill is source-independent: its authority is the scheduler,
container and software-stack documentation linked below, plus each
cluster's local documentation.

Learn more (verified):
  - https://slurm.schedmd.com - SLURM documentation
  - https://apptainer.org - Apptainer container runtime
  - https://www.eessi.io/docs/ - EESSI shared software stack
  - https://carpentries-incubator.github.io/hpc-intro/ - HPC
    Carpentry introduction
  - https://coderefinery.github.io/TTT4HPC_parallel_workflows/ -
    CodeRefinery tuesday-tools lessons on HPC workflows

<!-- related-skills:begin -->

## Related skills

Check whether any of these applies before moving on:

- rseng-big-data-processing - distributed data framework path
- rseng-data-management - staging data across cluster filesystems
- rseng-green-computing - honest requests save energy
- rseng-performance-profiling - measure scaling before allocating
- rseng-reproducible-environments - containers and pinned modules
- rseng-workflows - sweeps via workflow engines

<!-- related-skills:end -->
Files in this package (2)
  • SKILL.md 6 KB
  • references.md 1 KB

Versions

The database has no other skills named rseng-hpc-computing.

Database snapshot: 13.11.2026 · package version 7353594e2522 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100singularity-apptainer
    Use when the user asks about Singularity, Apptainer, HPC containers, running containers on clusters, or converting Docker images for HPC environments. Creates definition files, builds and validates .sif images, configures MPI/GPU execution, and integrates container runs with Slurm.
  • 90hpc
    Manages HPC/Slurm jobs, GPU allocations, and Singularity containers. Use for slurm, sbatch, srun, gpu, partition, cluster queue, apptainer work.
  • 100domino-distributed-computing
    Work with Domino on-demand compute clusters - Apache Spark, Ray, Dask and Open MPI - attached to a Workspace or Job. Covers choosing a framework, the cluster and execution environments, attaching a cluster from the UI or the Jobs API (computeCluster), connecting from code,…
  • 100slurm-job-script-generator
    Generate correct, copy-pasteable SLURM sbatch job scripts and sanity-check HPC resource requests — configure nodes, MPI tasks, OpenMP threads, memory (per-node or per-cpu), GPUs, walltime, partitions, modules, and environment variables, with automatic detection of conflicting…
  • 100mn5-first-job
    Takes a new BSC MareNostrum 5 user from "I have an account" to a first job run, verified and pulled back, then to real GPU and CPU-array jobs. Use whenever the task mentions BSC, MareNostrum, MN5, the BSC AI Factory, EuroHPC, a supercomputer, HPC, SLURM, sbatch, a GPU…
  • 90code-hpc
    Running on Stanford's Sherlock cluster with SLURM the das_ani way - preflight submit.sh, module setup script, safe sbatch headers, Python-side thread control, GPU type constraints, TF32 pinned off, sync/fetch without data loss, requeue-safe resume. Use when writing or editing…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.