Curve fitting: making anything look good in hindsight
Given enough parameters and enough attempts, any set of rules can be made to fit any history. That fit predicts nothing.
Curve fitting is tailoring a method to a specific stretch of historical data so closely that it captures the accidents of that period rather than anything durable. The fitted version performs beautifully on the data it was fitted to and poorly on everything else.
The mechanism
Every adjustable choice is a degree of freedom: the lookback, the threshold, the filter, the instrument, the date range, the session hours. With enough of them, you can shape the output to match almost any target.
The trap is that this feels like work. Each individual change has a story attached — “the shorter period suits this instrument,” “overnight bars are noisy” — and each story is plausible. The problem is not any single decision. It is that they were all made while looking at the answer.
The tell. Ask whether the choice would have been made before seeing its effect on results. If not, it is a fit, however good the reasoning sounds afterwards.
In-sample and out-of-sample
The standard defence is to split the data. Develop on one portion, then test once on a portion never examined during development.
It only works if the second portion is genuinely untouched. Testing out-of-sample, being disappointed, adjusting, and testing again turns the reserved data into development data. After three rounds of that, there is no out-of-sample left.
Walk-forward
A stricter version. Develop on a window, test on the period immediately after, roll both forward, repeat. It approximates how a method would have been used in real time and is far harder to game — though nothing removes the problem entirely when the same person keeps choosing what to try next.
Signs you are looking at a fitted result
- Many parameters relative to the number of trades
- Sharp cliffs — a period of 20 works and 19 does not
- Rules with carve-outs (“except on Mondays,” “except in the first 30 minutes”) that arrived after the fact
- Results that hold on one instrument and collapse on similar ones
- A test window that begins or ends at a suspiciously convenient date
Why we mention this while selling indicators
Because the same scepticism applies to us. Any historical illustration — ours included — was produced with the benefit of knowing how the period turned out. That is unavoidable for anything shown on past data, and it is a reason to weight what you observe on your own charts above what any vendor puts on a landing page.