Backtesting in Trading: 5 Checks for Reliable Results

A smooth, steadily rising equity curve can look very convincing. But it is not proof that a strategy actually works.

Backtesting in trading shows how a Trading System would have performed on the historical data used for the test. The real question is how robust those results are and how much they may depend on the way the system was built, optimized, and tested.

This is a crucial distinction in systematic trading.

Andrea Unger, four-time winner of the World Cup Trading Championships(R), has always emphasized the importance of building strategies based on objective rules and thoroughly testing them before putting real money at risk.

No test can guarantee that a strategy will continue to make money in the future. Markets change, and any Trading System can go through losing periods or become less effective over time.

What we can do, however, is determine whether the results we see in a backtest are robust enough to justify further testing.

Here are five essential checks to perform before trusting backtesting in trading.

Check 1: Is the system at risk of overfitting?

Overfitting in trading is one of the most insidious problems in Trading System development.

It occurs when a strategy is fitted too closely to the historical data used to build it. Adding parameters, filters, and conditions can progressively improve the results of backtesting in trading, but that does not necessarily mean the strategy itself has improved.

Instead, we may have simply built a system that is very good at describing what has already happened.

For example, imagine starting with a relatively simple strategy and progressively adding filters based on volatility, day of the week, trading hours, or other indicators because each one removes some losing trades from the backtest.

Each modification may seem reasonable on its own. The risk, however, is that the overall set of rules becomes fitted to very specific characteristics of the historical sample.

This is why a good strategy should not be evaluated solely on the final performance of its backtest.

We also need to ask whether the rules have a logical basis, whether the number of parameters is reasonable, and, above all, whether small changes in those parameters produce completely different results.

A system that works only with an extremely precise combination of values may be far less robust than its equity curve suggests.

We explored overfitting in trading in a video on our YouTube channel.

Check 2: Has the system been tested out-of-sample?

One of the most important checks for identifying a reliable backtest is testing the strategy on data that was not used to build it.

This is the principle behind out-of-sample testing.

During the development of a Trading System, part of the historical data is used to build, test, and optimize the strategy. This is the in-sample dataset.

A separate portion of the historical data can be excluded from the development process and subsequently used to evaluate how the system performs on data that played no role in its construction.

This is the out-of-sample dataset.

The distinction matters because excellent in-sample results may simply be the consequence of excessive optimization.

If the strategy continues to show promising behavior on data that was not used during development, however, we have an additional indication of its robustness.

Of course, even good out-of-sample results cannot guarantee that the system will continue to work in the future.

But they help answer a fundamental question: did the strategy produce promising results only on the data used to build it, or was it able to maintain similar characteristics on an independent dataset?

Check 3: Is the sample large and representative enough?

Another fundamental consideration is the quantity and quality of the data used to evaluate the system.

Backtesting in trading based on only a small number of trades can produce excellent results simply due to chance.

Likewise, even an apparently large number of trades may not be sufficient if they all occurred during a very specific historical period.

There is no single minimum number of trades that applies to every Trading System.

The required sample size depends on many factors, including the strategy’s trading frequency, the market being traded, and the variability of its results.

This is why simply asking how many trades are included in the backtest is not enough.

It is also important to consider how long the period being analyzed is and whether it includes different market conditions.

A strategy tested only during a particularly favorable market environment may provide an unrepresentative picture of its overall behavior.

The larger and more diverse the sample, the more information we can gather about how the strategy behaves.

Even then, however, the amount of data does not replace the other checks: a very long backtest can still be overfitted or improperly constructed.

Check 4: Is the out-of-sample period truly “untouched”?

Simply having a period labeled as out-of-sample is not enough.

We also need to make sure that this data has not directly or indirectly influenced the development of the strategy.

For example, suppose we set aside the most recent years of historical data as our out-of-sample dataset.

We test the strategy on that period, find that the results are disappointing, and go back to change some parameters or rules. We then test the strategy again on the same data.

At that point, the data is no longer truly “new.”

The information obtained from the first test has already influenced our subsequent decisions.

This is often referred to as out-of-sample contamination, and it is particularly insidious because it can happen without us even realizing it.

For the validation to remain meaningful, the out-of-sample dataset should be defined before the final development phase and should not be repeatedly used to decide which changes to make to the strategy.

Otherwise, we risk gradually turning the out-of-sample data into another form of in-sample data.

The discipline used to keep the datasets separate is therefore just as important as the split itself.

Check 5: Does backtesting in trading account for slippage and real-world costs?

Another common mistake is evaluating a strategy without adequately accounting for the costs that will be encountered in live trading.

Commissions, spreads, and slippage can reduce a strategy’s performance, sometimes significantly.

Slippage, in particular, is the difference between the theoretical price at which the system is expected to execute an order and the price at which the order is actually filled in the market.

Its impact depends on several factors, including the instrument being traded, liquidity, order type, market conditions, and trading frequency.

This becomes particularly important when a strategy has a low average trade.

If the average profit per trade is small, even seemingly modest costs can absorb a significant portion of the statistical edge shown by backtesting in trading.

For this reason, a strategy’s results should be evaluated using realistic and, whenever possible, conservative assumptions about execution costs.

A Trading System that looks attractive only when commissions and slippage are ignored is unlikely to provide a realistic representation of what we can expect in live trading.

We explored this topic in more detail in our article on slippage and the differences between backtesting and live trading.

What do these 5 backtesting checks actually tell us?

Passing these checks does not mean that we have found a strategy that is guaranteed to make money in the future.

There are no such guarantees in trading.

What it does mean is that we have subjected the backtest to several important checks designed to determine whether the results appear robust enough to justify further validation.

We have tried to reduce the risk of overfitting in trading.

We have tested the strategy on data that was not used during development.

We have assessed whether the historical sample is sufficiently representative.

We have tried to keep the out-of-sample period genuinely independent.

And we have considered the impact of the costs we will face when moving from simulation to live trading.

Of course, these are not the only factors that can affect the reliability of backtesting in trading.

Data quality, the characteristics of the instrument, order execution assumptions, and many other factors can all have an impact on the results.

The goal, therefore, is not to eliminate every possible source of uncertainty, but to build a development and validation process that is as rigorous as possible.

What comes after validation? Monte Carlo simulation

Once a strategy has been built and validated, we can subject it to other types of analysis.

One of these is Monte Carlo simulation.

Its purpose is different from that of the checks we have just discussed.

A Monte Carlo simulation can be used, for example, to analyze how certain results might change when the sequence of trades is altered and to gain additional insight into the potential variability of performance and drawdowns.

It does not replace Trading System validation. Instead, it is a complementary tool that can help us analyze the strategy’s risk profile in greater depth.

If you would like to learn more, we covered this topic in our article dedicated to Monte Carlo simulation.

A strong equity curve can be a good starting point.

But before putting real money behind a strategy, what matters is understanding how those results were obtained and how well they hold up when the robustness of the system is put to the test.

It is precisely this focus on data, testing, and validation that distinguishes systematic strategy development from simply trying to find a setup that happened to work well in the past.

Frequently Asked Questions About Backtesting in Trading

How do I know if I have a reliable backtest?

There is no single test that can guarantee the reliability of backtesting in trading. However, there are several important checks you can perform: assess the risk of overfitting, use out-of-sample data, analyze a sufficiently large and representative sample, avoid out-of-sample contamination, and account for realistic costs and slippage in trading.

What can make a backtest unreliable?

Some of the most common problems include overfitting historical data, using an unrepresentative sample, failing to perform genuine out-of-sample testing, and failing to account for real-world trading costs. Other factors, such as data quality and the way order execution is simulated, can also affect the results.

How many trades do I need before I can trust backtesting in trading?

There is no single number that applies to every strategy. It depends on the system’s trading frequency, the market, the variability of its results, and the length of the historical period analyzed. In general, a larger sample provides more information, but the number of trades should always be evaluated alongside the other elements of backtesting in trading.

Does a very long backtest eliminate the need for out-of-sample testing?

No. The length of a backtest and out-of-sample testing serve two different purposes. A longer historical period allows us to observe how the strategy behaves under a wider range of market conditions, while out-of-sample testing evaluates the system on data that did not contribute to its development.