Why professionals guard the holdout
Repeatedly consulting the holdout while tuning indicators turns a supposed out-of-sample segment into another training set. Record an initial research question, code/data versions, optimization search space, transaction assumptions and a frozen decision rule before opening the holdout. A subsequent strategy change needs another independent test rather than a rewritten history.
Even a chronological split is not always independent: overlapping trades, serial correlation, related instruments and liquidity conditions can make apparent sample size misleading. Evaluate execution realism and data provenance before claiming anything about performance.
