validation
/validation
Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.
- Author
- ruujOnChain ↗
- Repository
- HedgeFund-Skills ↗
- Licence
- MIT
- Last repository change
- 07.11.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 1
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 07.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 1 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/psr_dsr.py
What it does and when to use it
Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.
Original author description
Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.
How it works
- Inputs
- Procedure
- Checks
- Output
- Explaining the result
SKILL.md preview
---
name: validation
description: Tests whether a backtest result reflects skill or the luck of trying many variations, using the probabilistic Sharpe ratio and the deflated Sharpe ratio, which adjust for track length, skewness, fat tails, and the number of trials. Use when asked whether a backtest or Sharpe ratio is real, significant, overfit, or worth trusting, or how long a track record needs to be.
license: MIT
compatibility: Requires Python 3.9 or later with numpy and pandas.
metadata:
owner: research
version: "1.0"
---
# Strategy validation
Answers one question: is this backtest result real? Uses the probabilistic Sharpe ratio (Bailey and López de Prado, 2012) and the deflated Sharpe ratio (Bailey and López de Prado, 2014). Every number comes from `scripts/psr_dsr.py`.
## Inputs
- `returns.csv`: a date column, then periodic returns of the selected strategy (net of costs, excess of cash if available). Complete, no gaps.
- The number of trials: every variation tested on the way to this result, including parameter sets, universes, and rules that were discarded. This input is required. If the user does not know it, ask for their best honest count and state it in the answer.
- The dispersion of results across trials, one of:
- the returns of all trials (`--trial-returns`), or
- their per period Sharpe ratios (`--trial-sharpes`), or
- the variance of per period Sharpe ratios (`--sr-variance`) together with `--trials`.
## Procedure
1. Confirm the selected strategy, the return frequency, and the number of trials.
2. Run the validation:
```bash
python scripts/psr_dsr.py --returns returns.csv --column STRAT \
--trial-returns all_trials.csv --periods-per-year 252
```
or, without trial returns:
```bash
python scripts/psr_dsr.py --returns returns.csv --column STRAT --trials 40 --sr-variance 0.0004
```
3. Read the JSON. If `passed` is false, report the failed check and stop.
4. Report the Sharpe ratio, PSR, the threshold SR0, DSR, and whether DSR clears the confidence level.
## Checks
- At least 60 observations.
- The trial count is at least the number of trials supplied.
- The variance term in the PSR formula is positive.
## Output
`results.sharpe_per_period`, `results.sharpe_annualized`, `skewness`, `kurtosis` (3 for normal returns), `psr` against the benchmark, `sr0_per_period` and `sr0_annualized` (the Sharpe ratio the best of N trials would reach by chance), `dsr`, `passes`, and `min_track_record_length` in periods, plus `checks` and `passed`.
## Explaining the result
- Lead with DSR: the probability that the true Sharpe ratio exceeds what the best of N luckless trials would show. A DSR of 0.95 or more clears the default bar.
- Compare the annualized Sharpe with the annualized SR0, so the reader sees how high the bar was raised by the search.
- Negative skew and fat tails lower PSR and DSR for the same Sharpe ratio. Say so when they matter.
- Minimum track record length is the number of periods needed for DSR to reach the confidence level at the current Sharpe ratio.
- A failing DSR does not prove the idea is wrong. It shows the evidence is not yet strong enough given the search behind it.
For formulas and conventions, see [references/methodology.md](references/methodology.md).
Files in this package (3)
- SKILL.md 3 KB
- references/methodology.md 2 KB
- scripts/psr_dsr.py 6 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · ruujOnChain/HedgeFund-Skills | 100 | MIT |
| mtwuser/zero-cloud-gravity | 100 | MIT |
| hamzabellouch/agent-skills | 100 | MIT |
| minhuw/claude-writer | 100 | MIT |
| richard-kim-79/archora-skills | 100 | MIT |
| matteocervelli/llms | 95 | MIT |
| openai/codex-security | 90 | Apache-2.0 |
| agentic-swe/agentic-swe | 90 | MIT |
Database snapshot: 13.11.2026 · package version 805dfae64b41 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100skill-backtest-overfit
Detect backtest overfitting and selection bias from multiple testing. Use when a user has a backtest / factor result and asks whether the Sharpe is real, whether a strategy is overfit, or wants to validate results before trusting them. Computes Deflated Sharpe Ratio, Probability… - 95ml4t-deflated-sharpe
Adjust the Sharpe ratio for multiple testing bias when selecting from many trials. Use when reporting strategy performance after parameter or model search. - 100sharpe-ratio-non-iid-corrections
Sharpe ratio inference for ONE strategy under non-Normal, serially correlated returns. Implements the ρ-corrected (AR(1), skew, kurtosis) Sharpe variance of López de Prado, Lipton & Zoonekynd (2026, Eq 2-3) and the single-test tools built on it: PSR, MinTRL, critical Sharpe… - 100portfolio-robustness
Stress-test a jquantstats result rather than just computing it — is this Sharpe real, does it survive execution delay, trading costs, autocorrelation, outliers, or a different sub-period? Use when asked whether a backtest holds up, how sensitive it is to costs or lag, whether… - 100backtest-strategy
Runs backtests on trading strategies using historical market data. Calculates performance metrics including Sharpe ratio, maximum drawdown, win rate, total return, and generates equity curves. Trigger when the user requests backtesting, strategy simulation, or performance evaluation. - 100quant-research
Research systematic trading signals to the standard of a professional quant desk, where most backtests are false discoveries. Use when the user asks to find alpha, build or backtest a trading strategy or factor, evaluate a signal, compute a Sharpe ratio or information…