optimization-target
/optimization-target
Optymalizuje kod eksperymentów na klastrach Slurm, żeby wiele eksperymentów kończyło się szybciej bez zmiany wyników, i pisze raport.
- Autor
- mila-iqia ↗
- Repozytorium
- skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 12.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 2
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 2
Brak sygnałów. Repozytorium zmienione 12.11.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Optymalizuje kod eksperymentów na klastrach Slurm, żeby wiele eksperymentów kończyło się szybciej bez zmiany wyników, i pisze raport.
Oryginalny opis (EN)
Optimize experiment code and compute usage on Slurm clusters so that many experiments finish as quickly as possible without changing or compromising their results. Use when asked to speed up, profile, or scale experiment or training code, improve GPU utilization or SM occupancy (SMO), pack multiple experiments into one job allocation, tune throughput (samples/tokens per second), or write the optimization report for Mila's IDT team.
Jak działa
- Invariants
- About hyperparameter optimization
- Metrics/KPIs
- Optimization budget
- Job packing
- Optimization report
- Sending the report to the IDT team
Podgląd SKILL.md
---
name: optimization-target
description: >-
Optimize experiment code and compute usage on Slurm clusters so that many
experiments finish as quickly as possible without changing or compromising
their results. Use when asked to speed up, profile, or scale experiment or
training code, improve GPU utilization or SM occupancy (SMO), pack multiple
experiments into one job allocation, tune throughput (samples/tokens per
second), or write the optimization report for Mila's IDT team.
---
# Code and resource optimization
The context is that we want to run many experiments using this code on clusters to which this user has access, and which use Slurm to allocate resources. We want all of these experiments to finish as quickly as possible, without changing or compromising their results. This entails finding ways to accelerate individual experiments and/or packing experiments when possible.
Furthermore, we want to produce a report about our findings and optimizations which can be sent to Mila's IDT team.
## Invariants
The optimization of the codebase should have **no impact on the results of the experiment**. Given the same seed and hyperparameters, final artifacts such as model weights and training/validation/test error should be the same, either bit-for-bit or within a margin of tolerance. While working, fidelity of intermediate artifacts such as batch loss is a good proxy.
If the algorithm is non-deterministic, or if some changes can improve performance significantly but would influence the RNG or introduce minor numerical deviations, this is likely OK as long as your changes preserve the intent and semantics of the experiment: if not precisely, you must compute qualitatively the same thing. Nonetheless, ask the user if you should proceed.
## About hyperparameter optimization
It is possible that the best way to maximize resource utilization is to change certain hyperparameters (notably increasing the batch size). Tread carefully! Changing hyperparameter optimization changes experimental results. Raise this point to the user early and ask them what the acceptable hyperparameter ranges are.
## Metrics/KPIs
The metric to optimize depends on the kind of experiment we're running. Walltime is the simplest, and the most direct representation of what we wish to achieve. However, it can take a long time, which makes it hard to iterate on. Ideally we want to optimize a good proxy for it such as throughput, for instance the number of training samples or tokens per second.
## Optimization budget
Take note that your optimization work uses the same resource budget as the experiments. Spending a lot of time iterating on minor performance gains may be unproductive unless a large enough volume of experiments is planned to make up for it.
## Job packing
If an experiment requires significantly less resources than the minimum or usual GPU allocations on the cluster, it would be a great idea to pack them: run multiple experiments in parallel in the same job allocation. You can check experimentally how the time to run N experiments in a given allocation scales to determine the optimal N, or at least a good N.
## Optimization report
You should maintain a report in `optimization-report.md`, unless otherwise specified. Update it regularly as you work, even if some claims are as-of-yet unverified. Do not write anything private or confidential in that file. The report should only describe the essentials, although more details can be written in an annex at the end at your discretion.
The IDT team is interested in this report and may ask for it, especially if we call on their expertise to further optimize the code. They may not have access to the codebase or the data. In that light, when it is possible, please include in your report:
* **Skill version**: v1.0
* **Agent information**: Your model and harness, with versions, and effort level
* **The user's username**
* **Summary of the codebase**
* What it does. Training a model? LLM inference? Running a RL environment? What datasets? CPU-bound or GPU-bound?
* The software stack (pytorch, huggingface, hydra, etc. and also Mila tools like cluv)
* On which clusters experiments were run, which GPU types
* **Issues**
* Problems that you flagged in the code
* Fixes you made and the gains obtained
* Changes or fixes rejected by the user, and why
* **SMO feedback** (optional)
* If the user's experiments were flagged as "wasteful" by the IDT team, that means SMO was under a certain threshold, because SMO is the proxy they use to identify wastage. That is not a perfect proxy and some workloads may legitimately have low SMO and that's okay, but the IDT team cannot tell from their dashboard. So they would like some feedback.
* You can measure SMO with [milalib](https://github.com/mila-iqia/milalib) (`uvx milalib monitor -i 10 -m sm_occupancy` to probe every 10s), but keep in mind that we currently have a time resolution of ~30 seconds on profiling metrics.
* If SMO was low prior to optimization: why was it low?
* If SMO is *still* low after your efforts: is that to be expected with this workload?
* **Feedback**
* Anything you learned while working that would be useful to optimize other codebases, similar or not (or to avoid wasting their time)
## Sending the report to the IDT team
Always ask for the user's permission before sending a report. Tell them where to read the report prior to sending.
If permission is given, upload the file to this write-only endpoint (note: no more than 500 KB and it's rate-limited)
```bash
curl -w '\nStatus: %{http_code}' \
-X POST "https://sarc.mila.quebec/upload-report?category=codebase-optimization" \
-F "file=@optimization-report.md"
```
Don't persevere if you cannot upload the report, just say it failed.
Pliki w paczce (2)
- SKILL.md 6 KB
- quiz.yaml 2 KB
Wersje
W bazie nie ma innych skilli o nazwie optimization-target.
Migawka bazy z 13.11.2026 · wersja paczki ca9befc6b2b1 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 86run-experiments
Uruchamia kod eksperymentów na GitHub Actions lub Seyval i zwraca wyniki do repozytorium wraz z pochodzeniem danych. - 100experiment-bridge
Przechodzi od planu eksperymentów do uruchomienia: implementuje kod z EXPERIMENT_PLAN.md, wdraża go na GPU i zbiera początkowe wyniki. - 100experiment-bridge
Zamienia plan eksperymentów z EXPERIMENT_PLAN.md w działający kod, wdraża go na GPU i zbiera wstępne wyniki przed automatycznym przeglądem. - 100experiment-bridge
Łączy odkrywanie pomysłów z przeglądem: czyta EXPERIMENT_PLAN.md, implementuje kod eksperymentów, uruchamia je na GPU i zbiera pierwsze wyniki. - 100mila-sarc
Pobiera przez HTTP z API SARC dane o zadaniach, użytkownikach, klastrach i zużyciu zasobów Mila oraz DRAC, w tym wykorzystanie GPU i marnowane zasoby. - 100aris-experiment-bridge
Łączy odkrywanie pomysłów z automatyczną recenzją: czyta EXPERIMENT_PLAN.md, implementuje kod eksperymentów, wdraża go na GPU i zbiera wstępne wyniki.