Skip to content
Skills Library
Programming · Testing

benchmark

/benchmark

Run scalex performance benchmarks, profiling, and timing analysis. Use this skill whenever the user asks to benchmark scalex, measure performance, profile index/query times, compare before/after performance of a change, investigate bottlenecks, or mentions \"benchmark\", \"perf\", \"how fast\", \"timing\", \"hyperfine\", \"profile\", \"flame graph\", \"profiling\", \"--timings\", \"slow\", \"bottleneck\", \"regression\", \"memory\", \"heap\", \"GC\", \"allocation\". Also use proactively after implementing performance improvements to verify gains. Covers 6 layers: built-in --timings, hyperfine

Repository
scalex ↗
Licence
MIT
Last repository change
18.09.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
2
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 18.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 2 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/bench-compare.sh, scripts/bench.sh

What it does and when to use it

Run scalex performance benchmarks, profiling, and timing analysis. Use this skill whenever the user asks to benchmark scalex, measure performance, profile index/query times, compare before/after performance of a change, investigate bottlenecks, or mentions \"benchmark\", \"perf\", \"how fast\", \"timing\", \"hyperfine\", \"profile\", \"flame graph\", \"profiling\", \"--timings\", \"slow\", \"bottleneck\", \"regression\", \"memory\", \"heap\", \"GC\", \"allocation\". Also use proactively after implementing performance improvements to verify gains. Covers 6 layers: built-in --timings, hyperfine

Triggers from the description

benchmark\perf\how fast\timing\hyperfine\profile\flame graph\profiling\
Original author description

Run scalex performance benchmarks, profiling, and timing analysis. Use this skill whenever the user asks to benchmark scalex, measure performance, profile index/query times, compare before/after performance of a change, investigate bottlenecks, or mentions \"benchmark\", \"perf\", \"how fast\", \"timing\", \"hyperfine\", \"profile\", \"flame graph\", \"profiling\", \"--timings\", \"slow\", \"bottleneck\", \"regression\", \"memory\", \"heap\", \"GC\", \"allocation\". Also use proactively after implementing performance improvements to verify gains. Covers 6 layers: built-in --timings, hyperfine

How it works

  1. Overview
  2. Decision guide
  3. Layer 1: --timings flag
  4. Layer 2: hyperfine benchmarks (bench.sh)
  5. Layer 3: async-profiler flame graphs
  6. Layer 4: JFR (Java Flight Recorder)
  7. Layer 5: Microbenchmarks (src/bench.scala)
  8. Layer 6: Memory profiling (bench.sh memory)
  9. Native image profiling
  10. Performance budget (from AGENTS.md)
Original SKILL.md headings · 363 lines, 1,752 words
SKILL.md preview
---
name: benchmark
description: "Run scalex performance benchmarks, profiling, and timing analysis. Use this skill whenever the user asks to benchmark scalex, measure performance, profile index/query times, compare before/after performance of a change, investigate bottlenecks, or mentions \"benchmark\", \"perf\", \"how fast\", \"timing\", \"hyperfine\", \"profile\", \"flame graph\", \"profiling\", \"--timings\", \"slow\", \"bottleneck\", \"regression\", \"memory\", \"heap\", \"GC\", \"allocation\". Also use proactively after implementing performance improvements to verify gains. Covers 6 layers: built-in --timings, hyperfine benchmarks, async-profiler flame graphs, JFR recording, microbenchmarks, and memory profiling."
---

## Overview

Scalex has a multi-layered profiling and benchmarking system. Pick the right layer for the situation:

| Layer | Tool | When to use | Works in native? |
|-------|------|-------------|-------------------|
| 1. `--timings` | Built-in flag | Quick phase breakdown, first look at any perf question | Yes |
| 2. hyperfine | `bench.sh` | Reproducible before/after comparison with statistics | Yes |
| 3. async-profiler | `profiling/profile.sh` | Deep CPU/alloc/lock flame graphs to find hotspots | JVM only |
| 4. JFR | `profiling/scalex.jfc` | GC pressure, file I/O patterns, thread utilization | JVM only |
| 5. Microbenchmarks | `src/bench.scala` | Isolate per-function cost with warmup + statistics | JVM only |
| 6. Memory profiling | `bench.sh memory` | Heap usage, GC pressure, peak memory across scenarios | JVM only |

## Decision guide

**"Where is time spent?"** → Start with `--timings` (Layer 1)

**"Is this change faster?"** → Use hyperfine before/after (Layer 2), optionally with `bench-compare.sh`

**"Why is parsing slow?"** → async-profiler CPU flame graph (Layer 3)

**"Why are allocations high?"** → async-profiler alloc or JFR ObjectAllocationSample (Layer 3/4)

**"Is there GC pressure?"** → JFR (Layer 4)

**"How fast is extractSymbols on one file?"** → Microbenchmark (Layer 5)

**"How much memory does indexing use?"** → Memory profiling (Layer 6)

**"Is there a memory leak or GC regression?"** → Memory profiling before/after (Layer 6)

---

## Layer 1: `--timings` flag

The fastest way to see where time goes. Works in both JVM and native image. Prints to stderr.

```bash
# Cold index phase breakdown
rm -rf benchmark/scala3/.scalex
./scalex index benchmark/scala3 --timings

# Warm index
./scalex index benchmark/scala3 --timings

# Query with bloom/text-search breakdown
./scalex refs benchmark/scala3 Compiler --timings

# JVM mode
./mill run index benchmark/scala3 --timings
```

### Phases reported

**Index phases:** `git-ls-files`, `cache-load`, `oid-compare`, `parse`, `cache-save`, and lazy `build-*` lookup phases

**Query phases** (refs/imports/coverage): `bloom-screen`, `text-search`

Durations are exclusive per thread: nested phases are subtracted from their parent. `command` and `render` expose query and output work. `request-total` measures elapsed application request time, including otherwise uninstrumented work, but not runtime startup before the entry point. Use hyperfine for complete process latency. Concurrent phases can overlap, so summing phases is not a wall-clock measurement. Batch mode reports the shared index load, then a separate request total for each query.

### Reading the output

```
Timings:
  git-ls-files          12.3 ms  ( 1%)
  cache-load            45.2 ms  ( 5%)
  oid-compare            3.1 ms  ( 0%)
  parse                782.0 ms  (80%)
  index-build           89.4 ms  ( 9%)
  cache-save            42.1 ms  ( 4%)
  request-total        974.1 ms
```

- **parse > 70%**: Scalameta parsing dominates — look at parallelism, parser options, or reducing parsed file count
- **cache-load > 20%**: Index deserialization — check bloom skip, file size, buffering
- **cache-save > 10%**: Index serialization — check if save is running unnecessarily (when `parsedCount == 0`)
- **bloom-screen > text-search**: Bloom filter is slow — check expected element count, FPP
- **text-search >> bloom-screen**: Text scanning dominates — check candidate count (bloom too permissive?)

---

## Layer 2: hyperfine benchmarks (`bench.sh`)

Reproducible, statistical benchmarks using [hyperfine](https://github.com/sharkdp/hyperfine) against the Scala 3 compiler repo (~17.7K files).

### Prerequisites

- `hyperfine` installed (`brew install hyperfine`)
- Native scalex binary built (`./build-native.sh`)
- The scala3 repo is cloned automatically on first run

### Running

```bash
# Full suite (cold + warm + query + diverse + timings)
.agents/skills/benchmark/scripts/bench.sh

# Individual modes
.agents/skills/benchmark/scripts/bench.sh cold
.agents/skills/benchmark/scripts/bench.sh warm
.agents/skills/benchmark/scripts/bench.sh query
.agents/skills/benchmark/scripts/bench.sh diverse    # miss, heavy refs, fuzzy, grep, hierarchy
.agents/skills/benchmark/scripts/bench.sh timings    # --timings output for cold/warm/refs
.agents/skills/benchmark/scripts/bench.sh memory     # heap usage, GC pressure (JVM only)

# Custom runs/binary
BENCH_RUNS=10 SCALEX_BIN=./target/scalex .agents/skills/benchmark/scripts/bench.sh
```

### Before/after comparison

```bash
# 1. Benchmark current state
BENCH_EXPORT=benchmark/results/before.json .agents/skills/benchmark/scripts/bench.sh

# 2. Make changes, rebuild
./build-native.sh

# 3. Benchmark new state
BENCH_EXPORT=benchmark/results/after.json .agents/skills/benchmark/scripts/bench.sh

# 4. Compare (flags >5% regressions)
.agents/skills/benchmark/scripts/bench-compare.sh benchmark/results/before.json benchmark/results/after.json
```

`bench-compare.sh` exits non-zero if any benchmark regressed >5%.

### Typical ranges (Apple M3 Max)

| Metric | Range | Bottleneck |
|--------|-------|------------|
| Cold index | 3-5s | Scalameta parsing (CPU-bound, parallel) |
| Warm index | 0.8-1.0s | OID compare + index load |
| Query (any) | 1.2-1.5s | Index deserialization from disk |
| refs (heavy symbol) | 2-4s | Text search across candidate files |

---

## Layer 3: async-profiler flame graphs

Reveals call-stack-level CPU hotspots. No code changes needed — JVM agent only.

### Prerequisites

```bash
brew install async-profiler
# Or set AP_HOME to your installation
```

### Running

```bash
# CPU flame graph of cold index
./profiling/profile.sh benchmark/scala3

# Wall-clock (includes I/O wait — useful for parallelStream bottlenecks)
./profiling/profile.sh benchmark/scala3 wall

# Allocation hotspots (where objects are created)
./profiling/profile.sh benchmark/scala3 alloc

# Lock contention (parallelStream synchronization)
./profiling/profile.sh benchmark/scala3 lock
```

Output: `profiling/profile-<event>.html` — open in browser for interactive flame graph.

### What to look for

- **CPU**: Wide bars in Scalameta parsing → specific parser methods. Wide bars in `parallelStream` infrastructure → overhead from small task granularity.
- **Alloc**: Hot allocation sites → potential for object reuse or structural changes.
- **Lock**: Contention in `ConcurrentLinkedQueue.add()` → batch results instead of per-item add.
- **Wall**: I/O wait in `Files.readAllLines` or `Files.readString` → potential for memory-mapped I/O.

---

## Layer 4: JFR (Java Flight Recorder)

Built into the pinned JDK. Near-zero overhead. Best for GC, file I/O, and thread analysis.

### Running

```bash
# Record with custom config
SCALEX_JAVA_OPTS="-XX:StartFlightRecording=filename=profiling/scalex.jfr,settings=profiling/scalex.jfc,duration=60s" \
  ./mill run index benchmark/scala3

# Quick summary
jfr summary profiling/scalex.jfr

# Specific events
jfr print --events jdk.ObjectAllocationSample profiling/scalex.jfr | head -100
jfr print --events jdk.GarbageCollection profiling/scalex.jfr
jfr print --events jdk.FileRead profiling/scalex.jfr | head -50
jfr print --events jdk.ThreadPark profiling/scalex.jfr | head -50

# GUI analysis
open profiling/scalex.jfr  # Opens in JDK Mission Control
```

### Custom config

`profiling/scalex.jfc` is tuned for scalex — it enables allocation sampling, GC events, file I/O (>1ms threshold), and thread parking/monitor events.

---

## Layer 5: Microbenchmarks (`src/bench.scala`)

Isolate per-function costs with warmup and statistical measurement.

### Running

```bash
# Specific benchmark
./mill bench.run extract-single benchmark/scala3
./mill bench.run bloom-build benchmark/scala3
./mill bench.run persistence-load benchmark/scala3
./mill bench.run search benchmark/scala3
./mill bench.run refs benchmark/scala3

# All benchmarks
./mill bench.run all benchmark/scala3

# Custom warmup/iterations
./mill bench.run extract-single benchmark/scala3 --warmup 3 --iterations 10
```

### Available benchmarks

| Benchmark | What it measures |
|-----------|-----------------|
| `extract-single` | `extractSymbols` on the largest file |
| `extract-batch` | `extractSymbols` on 100 files (sequential AND parallel) |
| `bloom-build` | `buildBloomFilterFromSource` on a large source |
| `persistence-load` | `IndexPersistence.load` with and without bloom deserialization |
| `search` | `WorkspaceIndex.search("Compiler")` warm |
| `refs` | `findReferences("Phase")` warm |
| `index-cold` | Full cold index including map building |

Reports: mean, median, p99, stddev, min, max per benchmark.

---

## Layer 6: Memory profiling (`bench.sh memory`)

Measures heap usage, GC pressure, and peak memory across three scenarios: cold index (full parse), warm index (cache load), and refs query. Uses JVM GC logging via `-Xlog:gc*` — requires Mill (JVM mode), not native binary.

### Running

```bash
# Full memory profile (cold + warm + refs)
.agents/skills/benchmark/scripts/bench.sh memory
```

Uses the checked-in Mill launcher. Does not require hyperfine or native binary.

### Output

Reports per-scenario: phase timings (from `--timings`), then memory stats:

```
--- Cold index (full parse, ~17.7k files) ---
  git-ls-files            60.7 ms  ( 1%)
  parse                 5576.5 ms  (94%)
  cache-save             228.7 ms  ( 4%)
  total                 5952.2 ms

  Peak pre-GC heap:        790 MB
  Heap at exit (used):     676 MB
  Heap at exit (committed): 1184 MB
  GC pauses:               34
  Total GC pause time:     185.3 ms
```

Ends with a summary table:

```
=== Memory Summary ===

Scenario             Peak Heap      Exit Used    Exit Commit     GC #
--------             ---------      ---------    -----------     ----
Cold index              790 MB       676.2 MB      1184.0 MB       34
Warm index              256 MB       271.8 MB       584.0 MB        2
refs Phase               53 MB        29.7 MB        56.0 MB        0
```

### Reading the output

- **Peak pre-GC heap**: Highest live heap before any GC — the true high-water mark. This is the number that determines minimum `-Xmx` for constrained environments.
- **Heap at exit (used)**: Retained heap at process exit — the steady-state footprint of the loaded index + results.
- **Heap at exit (committed)**: OS-committed memory — what the JVM actually reserved. Higher than "used" because G1 keeps headroom.
- **GC pauses**: Number of young-gen collections. High count during cold index is normal (Scalameta ASTs are short-lived).
- **Total GC pause time**: Sum of all GC pauses. Should be small relative to wall time (<5%).

### What to watch for

- **Cold peak > 1 GB**: `parallelStream` is creating too many concurrent ASTs. Consider batching files in chunks.
- **Warm used > 400 MB**: Index deserialization is holding too much data. Check if lazy maps are working.
- **Refs used > 100 MB**: Text search is accumulating results. Check if bloom filtering is effective.
- **GC time > 10% of wall time**: GC pressure is impacting performance. Look at allocation hotspots (async-profiler alloc, Layer 3).

### Baseline ranges (scala3 corpus, 17.7k files, ~38k symbols)

| Scenario | Peak Heap | Exit Used | GC Pauses |
|----------|-----------|-----------|-----------|
| Cold index | 700-900 MB | 600-700 MB | 30-70 |
| Warm index | 200-300 MB | 250-300 MB | 1-3 |
| refs query | 30-60 MB | 25-35 MB | 0-1 |

### Ad-hoc memory profiling

For one-off measurements without the script:

```bash
# Run any scalex command with GC logging to a temp file
SCALEX_JAVA_OPTS="-Xlog:gc*=info:file=/tmp/scalex-gc.log" \
  ./mill run overview benchmark/scala3

# Check peak heap
grep -o '[0-9]*M->' /tmp/scalex-gc.log | sed 's/M->//' | sort -n | tail -1
# Check exit heap
grep "garbage-first" /tmp/scalex-gc.log | tail -1
```

---

## Native image profiling

async-profiler and JFR don't work with GraalVM native images. Options:

```bash
# macOS Instruments (Time Profiler)
xcrun xctrace record --template "Time Profiler" --launch -- ./scalex index scala3

# --timings always works
./scalex index scala3 --timings
```

---

## Performance budget (from AGENTS.md)

When evaluating changes:

- **Index times**: Accept <5% regression
- **Index size**: 0% growth for non-index features, <10% if schema changes
- **Query latency**: No regression
- **Feature gate**: "Is this better than grep, or does it introduce a worst case?"

The `bench-compare.sh` script automates the regression check against these thresholds.

## Typical workflow for a performance change

1. `--timings` to identify which phase to optimize
2. `BENCH_EXPORT=before.json bench.sh` to capture baseline
3. If deeper analysis needed: async-profiler flame graph or JFR
4. Make the change
5. `--timings` to verify phase improvement
6. `BENCH_EXPORT=after.json bench.sh` to capture new numbers
7. `bench-compare.sh before.json after.json` to check for regressions
8. Microbenchmarks if isolating a specific function
Files in this package (3)
  • SKILL.md 13 KB
  • scripts/bench-compare.sh 3 KB
  • scripts/bench.sh 9 KB

Versions

RepositoryScoreLicence
This page · nguyenyou/scalex100MIT
workerstack/repo100MIT
YuDefine/nuxt-supabase-starter100MIT
hoangsonww/Claude-Code-Agent-Monitor100MIT
RainingDaemons/agents100MIT
wwylon/devkit100MIT
serejaris/personal-corp-os100MIT
pekral/cursor-rules100MIT
peabrane/peapods100MIT

Database snapshot: 13.11.2026 · package version 0228edf103b6 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100benchmark-controls
    Benchmark control hazards: mutated controls, disagreeing harnesses, VM lock scaling, hyperfine 2.0 exports. Use when benchmarking, comparing timings, A/B runs, hyperfine.
  • 95hyperfine-benchmarking
    Use this skill to benchmark shell commands reliably with warmup runs, statistical summaries, and exportable artifacts using hyperfine.
  • 100review-performance
    Use when the user explicitly asks for a performance review, benchmark review, profiling review, perf audit, pprof analysis, allocation review, latency regression check, hot-path review, throughput review, "is this fast enough", or "review perf". This skill is opt-in only — never…
  • 100flame-graph-generator
    Generate flame graph generator operations. Auto-activating skill for Performance Testing. Triggers on: flame graph generator, flame graph generator Part of the Performance Testing skill category. Use when working with flame graph generator functionality. Trigger with phrases…
  • 88dev-loop-performance-profiler
    Use when the user asks about performance profiling, application performance, profiling tools, performance optimization, bottleneck analysis, or performance testing. Do NOT use for: debugging bugs (dev-loop-debugging-strategy), or code review (dev-loop-code-review).
  • 100python-performance
    Profiles Python code for performance bottlenecks and memory issues. Use when Python code is slow or when profiling for optimization before a release.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.