Skip to content
Skills Library
Data & analytics · Analysis

validation

/validation

Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.

Licence
MIT
Last repository change
07.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 07.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/psr_dsr.py

What it does and when to use it

Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.

Original author description

Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.

How it works

  1. Inputs
  2. Procedure
  3. Checks
  4. Output
  5. Explaining the result
Original SKILL.md headings · 49 lines, 424 words
SKILL.md preview
---
name: validation
description: Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.
license: MIT
compatibility: Requires Python 3.9 or later with numpy and pandas.
metadata:
  owner: research
  version: "1.0"
---

# Strategy validation

Answers one question: is this backtest result real? Uses the probabilistic Sharpe ratio (Bailey and López de Prado, 2012) and the deflated Sharpe ratio (Bailey and López de Prado, 2014). Every number comes from `scripts/psr_dsr.py`.

## Inputs

- `returns.csv`: a date column, then periodic returns of the selected strategy (net of costs, excess of cash if available). Complete, no gaps.
- The number of trials: every variation tested on the way to this result, including parameter sets, universes, and rules that were discarded. This input is required. If the user does not know it, ask for their best honest count and state it in the answer.
- The dispersion of results across trials, one of:
  - the returns of all trials (`--trial-returns`), or
  - their per period Sharpe ratios (`--trial-sharpes`), or
  - the variance of per period Sharpe ratios (`--sr-variance`) together with `--trials`.

## Procedure

1. Confirm the selected strategy, the return frequency, and the number of trials.
2. Run the validation:
   ```bash
   python scripts/psr_dsr.py --returns returns.csv --column STRAT \
     --trial-returns all_trials.csv --periods-per-year 252
   ```
   or, without trial returns:
   ```bash
   python scripts/psr_dsr.py --returns returns.csv --column STRAT --trials 40 --sr-variance 0.0004
   ```
3. Read the JSON. If `passed` is false, report the failed check and stop.
4. Report the Sharpe ratio, PSR, the threshold SR0, DSR, and whether DSR clears the confidence level.

## Checks

- At least 60 observations.
- The trial count is at least the number of trials supplied.
- The variance term in the PSR formula is positive.

## Output

`results.sharpe_per_period`, `results.sharpe_annualized`, `skewness`, `kurtosis` (3 for normal returns), `psr` against the benchmark, `sr0_per_period` and `sr0_annualized` (the Sharpe ratio the best of N trials would reach by chance), `dsr`, `passes`, and `min_track_record_length` in periods, plus `checks` and `passed`.

## Explaining the result

- Lead with DSR: the probability that the true Sharpe ratio exceeds what the best of N luckless trials would show. A DSR of 0.95 or more clears the default bar.
- Compare the annualized Sharpe with the annualized SR0, so the reader sees how high the bar was raised by the search.
- Negative skew and fat tails lower PSR and DSR for the same Sharpe ratio. Say so when they matter.
- Minimum track record length is the number of periods needed for DSR to reach the confidence level at the current Sharpe ratio.
- A failing DSR does not prove the idea is wrong. It shows the evidence is not yet strong enough given the search behind it.

For formulas and conventions, see [references/methodology.md](references/methodology.md).
Files in this package (3)
  • SKILL.md 3 KB
  • references/methodology.md 2 KB
  • scripts/psr_dsr.py 6 KB

Versions

RepositoryScoreLicence
This page · ruujOnChain/HedgeFund-Skills100MIT
mtwuser/zero-cloud-gravity100MIT
hamzabellouch/agent-skills100MIT
minhuw/claude-writer100MIT
richard-kim-79/archora-skills100MIT
matteocervelli/llms95MIT
openai/codex-security90Apache-2.0
agentic-swe/agentic-swe90MIT

Database snapshot: 13.11.2026 · package version 805dfae64b41 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100skill-backtest-overfit
    Detect backtest overfitting and selection bias from multiple testing. Use when a user has a backtest / factor result and asks whether the Sharpe is real, whether a strategy is overfit, or wants to validate results before trusting them. Computes Deflated Sharpe Ratio, Probability…
  • 95ml4t-deflated-sharpe
    Adjust the Sharpe ratio for multiple testing bias when selecting from many trials. Use when reporting strategy performance after parameter or model search.
  • 100sharpe-ratio-non-iid-corrections
    Sharpe ratio inference for ONE strategy under non-Normal, serially correlated returns. Implements the ρ-corrected (AR(1), skew, kurtosis) Sharpe variance of López de Prado, Lipton & Zoonekynd (2026, Eq 2-3) and the single-test tools built on it: PSR, MinTRL, critical Sharpe…
  • 100portfolio-robustness
    Stress-test a jquantstats result rather than just computing it — is this Sharpe real, does it survive execution delay, trading costs, autocorrelation, outliers, or a different sub-period? Use when asked whether a backtest holds up, how sensitive it is to costs or lag, whether…
  • 100backtest-strategy
    Runs backtests on trading strategies using historical market data. Calculates performance metrics including Sharpe ratio, maximum drawdown, win rate, total return, and generates equity curves. Trigger when the user requests backtesting, strategy simulation, or performance evaluation.
  • 100quant-research
    Research systematic trading signals to the standard of a professional quant desk, where most backtests are false discoveries. Use when the user asks to find alpha, build or backtest a trading strategy or factor, evaluate a signal, compute a Sharpe ratio or information…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.