Backtesting trading strategies software

Check out my ebook on quant trading where I teach you how to build profitable systematic trading strategies with Python tools, from scratch. Take a look at my new ebook on advanced trading strategies using time series analysis, machine learning and Bayesian statistics, with Python and R. This article continues the series on quantitative trading, which started with the Beginner's Guide and Strategy Identification. Both of these longer, more involved articles have been very popular so I'll continue in this vein and provide detail on the topic of strategy backtesting.

I couldn't hope to cover all of those topics in one article, so I'm going to split them into two or three smaller pieces. What will we discuss in this section? I'll begin by defining backtesting and then I will describe the basics of how it is carried out. Then I will elucidate upon the biases we touched upon in the Beginner's Guide software Quantitative Trading.

Next I will present a comparison of the various available backtesting software options. In subsequent articles we will look at the details strategies strategy implementations that are often barely mentioned or ignored.

We will also consider how to make the backtesting process more realistic by including the idiosyncrasies of a trading exchange. Then we will discuss transaction costs and how to correctly model them in a backtest setting. We will end with a discussion on the performance of our backtests and finally provide an example of a common quant strategy, known as a mean-reverting pairs trade.

Let's begin by discussing what backtesting is and why we should carry it out in our algorithmic trading. Backtesting trading stands apart from other types of strategies classes because we can more reliably provide expectations about future performance from past performance, as a consequence of abundant data availability. The process by which this is carried out is known as backtesting.

In simple terms, backtesting is carried out by exposing your particular strategy algorithm to a stream of historical financial data, which leads to a set of trading signals. Each backtesting which we will mean here to be a 'round-trip' of two signals will have an associated profit or loss. That is the essence of the idea, although of course the "devil trading always in the details"! Backtesting provides a host of advantages trading algorithmic trading.

However, it is not always possible to straightforwardly backtest a strategy. In general, as the frequency of the strategy increases, it becomes harder to correctly model the microstructure effects of the market and exchanges.

This leads to less reliable backtests and thus a trickier evaluation of a chosen strategy. This is a particular problem where the execution system is the key to the strategy performance, as with ultra-high frequency algorithms.

Unfortunately, trading is fraught with biases of all types. We have touched upon some of these issues in previous articles, strategies we will now discuss them in depth.

There are strategies biases that can affect the performance of a backtested strategy. Unfortunately, these biases have a tendency to inflate the performance rather than detract from it. Thus you should always consider a backtest to be an idealised upper bound on the actual performance of the strategy. It is almost impossible to eliminate biases from algorithmic trading so it is our job to minimise them as best we can in order to make informed decisions about our algorithmic strategies.

There are software major strategies that I wish to discuss: Optimisation BiasLook-Ahead BiasSurvivorship Bias and Psychological Tolerance Bias. This is probably the most insidious of all backtest biases. It involves adjusting or introducing additional trading parameters until the strategy performance on trading backtest data set is very attractive.

However, once live the performance of the strategy can be markedly different. Another name for this bias is "curve fitting" or "data-snooping bias".

Optimisation bias is hard to eliminate as algorithmic strategies often involve many parameters. Optimisation bias can be minimised by keeping the number of parameters to a minimum and increasing the quantity of data points in the training set.

In fact, one must also be careful of the latter as older training points can be subject to a prior regime such as a regulatory environment and thus may not be relevant to your current strategy. One method to help mitigate this bias is to perform a sensitivity analysis. This means software the parameters incrementally and plotting a "surface" of performance. Sound, fundamental reasoning for parameter choices should, with all other factors considered, lead to a smoother parameter surface.

If you have a very jumpy performance surface, it often means that a parameter is not reflecting a phenomena and is an artefact of the test data. There is a vast literature on multi-dimensional optimisation algorithms and it is a highly active area of research. I won't dwell on it here, but keep it in the back of your mind when you find a strategy with a fantastic backtest! Look-ahead bias is introduced into a backtesting system when future data is accidentally included at a point backtesting the simulation where that data would not have actually been available.

Look-ahead bias errors can be incredibly subtle. Here are three examples backtesting how look-ahead bias can be introduced:. As with optimisation bias, one must be extremely careful to avoid its introduction. It is often the main reason why trading strategies underperform their backtests significantly in "live trading". Survivorship bias is a particularly dangerous phenomenon and can lead to significantly inflated performance for certain strategy types.

It occurs when strategies are tested on datasets that do not include the full universe of prior assets that may have been strategies at a particular point in time, but only consider those that have "survived" to the current time.

As an example, consider testing a strategy on a random selection of equities before and after the market crash. Some technology stocks went bankrupt, while others managed to stay afloat and even prospered. If we had restricted this strategy only to stocks which made it through the market drawdown period, we would be introducing a backtesting bias because they have already demonstrated their success to us.

In fact, this is just another specific case of software bias, as future information is being incorporated into past analysis. We will now consider certain psychological phenomena that can influence your trading performance. This particular phenomena is not often discussed in the context of quantitative trading.

However, it is discussed extensively in regard to more discretionary trading methods. It has various names, but I've decided to call it "psychological tolerance bias" because it captures the essence of the problem.

When creating backtests over a period of 5 years or more, it is easy to look at an upwardly trending equity curve, calculate the compounded annual return, Sharpe ratio and even drawdown characteristics and be satisfied with the results. This would not be atypical for a momentum strategy. It is straightforward to convince oneself that it is easy to tolerate strategies periods of losses because the overall picture is rosy.

However, in practice, it is far harder! These periods of drawdown are psychologically difficult to backtesting. I have observed first hand what an extended drawdown can be like, in an institutional setting, and it is not pleasant - even if the backtests suggest such periods will occur. The reason I have termed it a "bias" is that often a strategy which would otherwise be successful is stopped from trading during times of extended drawdown and thus will lead trading significant underperformance compared to a backtest.

Thus, even though the strategy is algorithmic in nature, psychological factors can still have a heavy influence on profitability.

The takeaway is to ensure that if you see drawdowns of a certain percentage and duration in the backtests, then you should expect them to occur in live trading environments, and will need to persevere in order to reach profitability once more. The software landscape for strategy backtesting is vast. As quant traders we are interested in the balance of being able to "own" our trading technology stack versus the speed and reliability of our development methodology.

Here are the key considerations for software choice:. Now that we have listed the criteria with which we need to choose our software infrastructure, I want to run through some of the more popular packages and how they compare:.

I am only going to include software that is available to most retail practitioners and software developers, as this is the readership of the site. While other software is available such as the more institutional grade tools, I feel these are too expensive to be effectively used in a retail setting strategies I personally have no experience with them.

Different strategies will require different software packages. My personal preference is for Python as it provides the right degree of customisation, speed of development, testing capability and execution speed for my needs and strategies. In the next few articles on backtesting we will take a look at some particular issues surrounding the implementation of an algorithmic trading backtesting system, as well as how to incorporate the effects of trading exchanges. We will discuss strategy performance measurement and finally conclude with an example strategy.

Read the next article in the series: Successful Backtesting of Algorithmic Trading Strategies - Part II. Learn about QuantStart Read our Books Browse the Articles List Explore the Reading List Backtest with QSTrader Query the Support Knowledge Base. Successful Backtesting of Algorithmic Trading Strategies - Part I. By Michael Halls-Moore on April 26th, This article continues the series on quantitative trading, which started with the Beginner's Guide and Strategy Identification.

What are key reasons for backtesting an algorithmic strategy? Filtration - If you recall from the article on Strategy Identificationour goal at the initial research stage was to set up a strategy pipeline and then filter out any strategy that did not meet certain criteria.

Backtesting provides us with another filtration mechanism, as we can eliminate strategies that do not meet our performance needs. Modelling - Backtesting allows us to safely! Optimisation - Although strategy optimisation is fraught with biases, backtesting allows us to increase the performance of a strategy by modifying the quantity or values of the parameters associated with that strategy and recalculating its performance. Verification - Our strategies are often sourced externally, via our strategy pipeline.

Backtesting a strategy ensures that it has not been incorrectly implemented. Although we will rarely have access to the signals generated by external strategies, we will often have access to the performance metrics such as the Sharpe Ratio and Drawdown characteristics. Thus we can compare them with our own implementation. Biases Affecting Strategy Backtests There are many biases that can affect the performance of a backtested strategy.

Optimisation Software This is probably the most insidious of all backtest biases. Look-Ahead Bias Look-ahead bias is introduced into a backtesting system when future data is accidentally software at a point in the simulation where that data would not have actually been available.

Here are three examples of how look-ahead bias can be introduced: Parameter Calculation - Another common example of look-ahead bias occurs when calculating optimal strategy parameters, such as backtesting linear regressions between two time series. If the whole data set including future data is used to calculate the regression strategies, and thus retroactively applied to a trading strategy for optimisation purposes, then future data is being incorporated and a look-ahead bias exists.

Survivorship Bias Survivorship bias is a particularly dangerous phenomenon and can lead to significantly inflated performance for certain strategy types. There are two main ways to mitigate survivorship bias in your strategy backtests: Survivorship Bias Free Datasets - In the case of equity data it is possible to purchase datasets that include delisted entities, although they are not cheap and only tend to be utilised by institutional firms. In particular, Yahoo Finance data is NOT survivorship bias free, and this is software used by many retail algo traders.

One can also trade on asset classes that are not prone to survivorship bias, such as certain commodities and their future derivatives. Use More Recent Data - In the case of equities, utilising a more recent data set mitigates the possibility that the stock selection chosen is weighted to "survivors", simply as there is less likelihood of overall stock delisting in shorter time periods. One can also start building a personal survivorship-bias free dataset by collecting data from current point onward.

After years, you will have a solid survivorship-bias free set of equities data with which to backtest further strategies.

Psychological Tolerance Bias This particular phenomena is not often discussed in the context of quantitative trading. Software Packages for Backtesting The software landscape for strategy backtesting is vast. Here are the key considerations for software choice: Programming Skill - The choice of environment will in a large part come down to your ability to program backtesting.

This is due to the downside risk of having external bugs trading idiosyncrasies that you are unable to fix in vendor software, which would otherwise be easily remedied if you had more control over your "tech stack".

You also want an environment that strikes the right balance between productivity, library availability and speed of execution.

I make my own personal recommendation below. I am not a fan of this approach as reducing transaction costs are often strategies big component of getting a higher Sharpe ratio.

If you're tied into a particular broker and Tradestation "forces" you to do thisthen you will have a harder time transitioning to new software or a new broker if the need arises. Interactive Brokers provide an API which is robust, albeit with a slightly obtuse interface. Customisation - An environment like MATLAB software Python gives you a great deal of flexibility when creating algo strategies as they provide fantastic libraries for nearly any mathematical operation imaginable, but also allow extensive customisation where necessary.

Strategy Complexity trading Certain software just isn't cut out for heavy number crunching or mathematical complexity. Excel is one such piece of software. While it is good for simpler strategies, it trading really cope with numerous assets or more complicated algorithms, at speed.

Bias Minimisation - Does a particular piece of software or data strategies itself more to trading biases? You need to make sure that if you want to create all the functionality yourself, backtesting you don't introduce bugs which can lead to biases.

Speed of Development - One shouldn't have to spend months and months implementing a backtest engine. Prototyping should only take a trading weeks. Make sure that your software is not hindering your progress to any great extent, just to grab a few extra percentage points of execution speed. However, you will be verging on Linux kernel optimisation and FPGA usage for these domains, which is outside the scope of this article! Cost - Many of the software environments that you can program algorithmic trading strategies with are completely free and open source.

In fact, many hedge funds make use of open source software for their entire algo trading stacks. In addition, Excel and MATLAB are both relatively cheap and there are even free alternatives to each. Now that we have listed the criteria with which we need to choose our software infrastructure, I want to run through some of the more popular packages and how they compare: Backtesting Software Comparison MS Excel Description: WYSIWYG what-you-see-is-what-you-get spreadsheet software.

Extremely widespread in the financial industry. Data and algorithm are tightly coupled. Yes, Excel can be tied into most brokerages. VBA macros allow more advanced functionality at the expense of hiding implementation. More advanced statistical tools are harder to implement as are strategies with many hundreds of assets.

Look-ahead bias is easy to detect via cell-highlighting functionality assuming no VBA. Quick to implement basic strategies. Slow execution speed - suitable only for lower-frequency strategies. Cheap or free depending upon license. Programming environment originally designed for computational mathematics, physics and engineering. Very well suited to vectorised operations and those involving numerical linear algebra.

Provides a wide array of plugins for quant trading. In widespread use in trading hedge funds. No native execution capability, MATLAB requires a separate execution system.

Huge array of community plugins for nearly software areas of computational mathematics. Many advanced statistical methods already available and well-tested. Harder to detect look-ahead bias, requires extensive testing. Short scripts can create sophisticated backtests easily. Poor for traditional iterated loops.

High-level language designed for speed of development. Wide array of libraries for nearly any programmatic task imaginable. Gaining wider acceptance in backtesting fund and investment bank community. Python plugins software for larger brokers, such as Interactive Brokers. Hence backtest and execution system can all be part of the same "tech stack". Python has a very healthy development community and is a mature language. Many plugins exist for the main algorithms, but not quite as big a quant community as exists for MATLAB.

Same bias minimisation problems exist as for any high level language. Need to be backtesting careful about testing. Pythons main advantage is software speed, with robust in built in testing capabilities.

Environment designed for advanced statistical methods and time series analysis. Wide array of specific statistical, econometric and native graphing toolsets. R possesses plugins to some brokers, in particular Interactive Brokers. Thus an end-to-end system can written entirely in R. Mostly useful if performing econometric, statistical or machine-learning strategies due to available plugins. Thus testing must be carried out. R is rapid for writing strategies based on statistical methods.

Mature, high-level language designed for speed of execution. Wide array of quantitative finance and numerical libraries. Harder to debug and often takes longer to implement than Python or MATLAB.

Extremely prevalent in both the buy- and sell-side. Thus many plugins exist. Look-ahead bias can be tricky to eliminate, but no harder than other high-level language. Good debugging tools, but one must be careful when dealing with underlying memory. More lines-of-code LOC often trading to greater likelihood of bugs. This is the main reason to utilise it. CJavaScala Different strategies will require different software packages.

RubyErlangHaskell. CJavaScala.

How to Backtest Strategies on MT4 (Backtest Strategy Tool)