PRO RESEARCH / STRATEGY VALIDATION
An attractive backtest can be manufactured without fabricating a single trade. Repeated parameter selection, indicator trials, survivorship filters and data revisions are enough to create apparent edge. Robust validation asks whether a fully specified decision rule survives new observations.
Research status: Educational framework only. No verified live quote, broker fill, investment recommendation or commercial endorsement.
Separate research, selection and evaluation
A training window is where models and parameters may be fitted. A validation window is where limited choices may be compared. A final untouched holdout is not a place to keep improving the strategy after disappointing results. Once a researcher sees the holdout and adjusts parameters, it becomes part of the research process, not an independent proof.
For time-series strategies, overlapping labels and forward returns cause leakage across adjacent folds. Use a purge/embargo interval appropriate to the longest label horizon and market data publication lag. Random row-wise cross-validation is often unsuitable.
Include actual trading feasibility
Suppose an illustrative training estimate is +0.30R per trade but the independent period yields +0.09R before omitted implementation costs. If realistic spread and slippage are 0.12R, holdout net is −0.03R. A profitable in-sample figure does not override that result.
Where fill probability is uncertain, model no-fill scenarios and missed opportunities. A backtest that assumes fills only for winning signals is not a defensible execution model.
A researcher’s reproducibility ledger
- Immutable source dataset and provider licence, including timezone and adjustments.
- Feature and label definitions as of their observation time.
- Training/validation/holdout windows and purge duration.
- Parameter search space and number of attempted variants.
- Execution model, contract rules, fees, taxes excluded and reasons.
- Experiment timestamp, code version, reviewer and confidence limits.
When to stop
Do not declare success from one small profitable holdout. Examine stability through different market sessions, volatility conditions, fees, contract roll schedules and sample endpoints. The number of strategies tested also matters: the best of hundreds of trials requires much stronger skepticism.
Model failure is useful information. Archive rejected hypotheses so that next month’s apparently new idea cannot quietly reproduce an old discarded test.
Related tools
- Strategy validation lab — Measure holdout cost sensitivity
- Performance and methodology — How results should be represented
- Research workflow — Return to the evidence-first library
Editorial framework · Version 2026-11-05 · Hypothetical examples only · Financial markets involve material risk. Inputs and data rights must be verified independently.
ILLUSTRATIVE RESEARCH GRAPHIC · NOT LIVE DATA
A profit can vanish after realistic costs
Charts here explain a method. They do not use a licensed market feed, a live quote, realized investment returns or an audited performance history.

