rseng-hpc-computing
/rseng-hpc-computing
Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring
- Author
- fdiblen ↗
- Repository
- rseng-agent-skills ↗
- Licence
- MIT
- Last repository change
- 23.09.2026
Assessment and security
- Files in this package
- 2
- Scripts
- 0
- HTML comments
- 2
- URLs in SKILL.md
- 5
No signals detected. Repository changed on 23.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- SKILL.md zawiera komentarze HTML (2). Nie widać ich w podglądzie Markdown, a model je czyta.
What it does and when to use it
Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring
Original author description
Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring
How it works
- Batch jobs done right (SLURM)
- Software environments on clusters
- Parallelism: pick the right kind
- Cluster citizenship and data
- Working with this skill
- Related skills
SKILL.md preview
---
name: rseng-hpc-computing
description: >-
Covers working effectively on high-performance computing clusters: writing
and debugging SLURM job scripts, choosing and requesting resources honestly,
running containers with Apptainer, using module systems and EESSI software
stacks, MPI basics, checkpointing, and scaling from laptop to cluster
reproducibly. Use when the user mentions a cluster, supercomputer, SLURM,
sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime,
or when a compute workload has outgrown a single machine.
(Larger-than-memory data processing with Dask or Spark is
rseng-big-data-processing; measuring scaling before requesting allocations is
rseng-performance-profiling.)
license: CC-BY-4.0
metadata:
version: 0.1.0
---
# Research computing on HPC clusters
HPC clusters are where much research code actually runs, and they
invert desktop habits: you do not run programs, you request resources
and submit batch jobs; you do not install software, you load modules
or bring containers; the login node is a shared hallway, not a
workstation. An agent's job is to translate the user's computation
into this model correctly and reproducibly - and to respect that
every cluster has local documentation that overrides generic advice.
## Batch jobs done right (SLURM)
A job script is a shell script with resource directives:
```bash
#!/bin/bash
#SBATCH --job-name=fit-model
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --output=logs/%x-%j.out
module load Python/3.12
srun python fit_model.py --config configs/run.yaml
```
Practices that matter:
- Request honestly: measure a pilot run, then set time/memory with
modest headroom. Over-requesting wastes allocation and queue
priority (and energy - rseng-green-computing); under-requesting
kills jobs at 99%.
- Never compute on the login node; test with a short interactive
allocation (srun --pty or salloc) instead.
- Parameter sweeps are job arrays (--array=0-99), not 100 submitted
scripts; index into a config list with $SLURM_ARRAY_TASK_ID
(externalized run configs pair naturally with rseng-workflows).
- Long jobs checkpoint: clusters preempt and nodes fail; save
restartable state at intervals and make the script resume from the
last checkpoint.
- Log usefully: %x-%j names, and `seff JOBID` after completion to
compare requested vs used resources - feed that back into the next
request.
## Software environments on clusters
Three layers, in order of preference for reproducibility:
1. Containers with Apptainer (formerly Singularity): the HPC-native
container runtime - unprivileged, image-file based, GPU-aware.
Build once (possibly from a Docker image), run identically on any
cluster; pairs with rseng-reproducible-environments for the build
recipe.
2. EESSI: a shared, optimized, ready-to-use scientific software
stack streamed to any cluster or laptop - when available it gives
identical software everywhere without building anything.
3. Module system (module avail / module load): the site's curated
builds; record exact module versions in the job script - an
unversioned `module load Python` is a reproducibility hole.
Never pip-install into the home directory as the primary strategy;
quota, node-architecture and reproducibility problems follow.
## Parallelism: pick the right kind
- Independent runs (sweeps, bootstraps, per-file processing): job
arrays or a workflow engine - no MPI needed. This covers most
research workloads.
- Single-node parallelism: threads/multiprocessing;
--cpus-per-task, and pin BLAS/OpenMP thread counts to match.
- Multi-node tightly-coupled computation: MPI (mpi4py or native),
launched with srun; requires the code to be written for it.
Distributed data processing frameworks are their own path
(rseng-big-data-processing).
Scaling discipline: measure strong/weak scaling on small counts
before requesting large allocations; doubling cores that yield 1.2x
wastes everyone's allocation
(rseng-performance-profiling for the measurement).
## Cluster citizenship and data
- Filesystems differ: home (small, backed up), project (shared),
scratch (fast, purged) - stage data to scratch for jobs, move
results back, and script the staging (rseng-data-management).
- Transfer with rsync (resumable) or the site's data-transfer nodes;
never leave the only copy of results on scratch.
- Read the site's own docs first; when the user's cluster is known,
prefer its conventions over anything generic, including this
skill.
## Working with this skill
This skill is source-independent: its authority is the scheduler,
container and software-stack documentation linked below, plus each
cluster's local documentation.
Learn more (verified):
- https://slurm.schedmd.com - SLURM documentation
- https://apptainer.org - Apptainer container runtime
- https://www.eessi.io/docs/ - EESSI shared software stack
- https://carpentries-incubator.github.io/hpc-intro/ - HPC
Carpentry introduction
- https://coderefinery.github.io/TTT4HPC_parallel_workflows/ -
CodeRefinery tuesday-tools lessons on HPC workflows
<!-- related-skills:begin -->
## Related skills
Check whether any of these applies before moving on:
- rseng-big-data-processing - distributed data framework path
- rseng-data-management - staging data across cluster filesystems
- rseng-green-computing - honest requests save energy
- rseng-performance-profiling - measure scaling before allocating
- rseng-reproducible-environments - containers and pinned modules
- rseng-workflows - sweeps via workflow engines
<!-- related-skills:end -->
Files in this package (2)
- SKILL.md 6 KB
- references.md 1 KB
Versions
The database has no other skills named rseng-hpc-computing.
Database snapshot: 13.11.2026 · package version 7353594e2522 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100singularity-apptainer
Use when the user asks about Singularity, Apptainer, HPC containers, running containers on clusters, or converting Docker images for HPC environments. Creates definition files, builds and validates .sif images, configures MPI/GPU execution, and integrates container runs with Slurm. - 90hpc
Manages HPC/Slurm jobs, GPU allocations, and Singularity containers. Use for slurm, sbatch, srun, gpu, partition, cluster queue, apptainer work. - 100domino-distributed-computing
Work with Domino on-demand compute clusters - Apache Spark, Ray, Dask and Open MPI - attached to a Workspace or Job. Covers choosing a framework, the cluster and execution environments, attaching a cluster from the UI or the Jobs API (computeCluster), connecting from code,… - 100slurm-job-script-generator
Generate correct, copy-pasteable SLURM sbatch job scripts and sanity-check HPC resource requests — configure nodes, MPI tasks, OpenMP threads, memory (per-node or per-cpu), GPUs, walltime, partitions, modules, and environment variables, with automatic detection of conflicting… - 100mn5-first-job
Takes a new BSC MareNostrum 5 user from "I have an account" to a first job run, verified and pulled back, then to real GPU and CPU-array jobs. Use whenever the task mentions BSC, MareNostrum, MN5, the BSC AI Factory, EuroHPC, a supercomputer, HPC, SLURM, sbatch, a GPU… - 90code-hpc
Running on Stanford's Sherlock cluster with SLURM the das_ani way - preflight submit.sh, module setup script, safe sbatch headers, Python-side thread control, GPU type constraints, TF32 pinned off, sync/fetch without data loss, requeue-safe resume. Use when writing or editing…