The Ultimate Guide To Successful Algorithmic Trading

Photo From UnSplash

Pretty much everyone who has programmed a computer to do
anything beyond outputting “Hello World” has dreamed of having a computer algorithm (algo) working tirelessly to extract money from the financial markets, be it in stocks, bitcoin, soybeans or anything else traded on an exchange.  “Programming genius, market slayer” is a phrase we’d all like to be associated with. That is what a good algorithmic trader is.

But how realistic is creating and deploying a computerized algo bot, or an army of bots, to make money for you? And, assuming it can be done, how do you actually go about doing it?  This guide walks you through the steps to becoming successful at algo trading.  But be warned – it is much more involved and much more difficult than you might think.

The Basics – What Are We Actually Talking About?

Before we get too far, there is some terminology involved in trading that will help you understand algo trading. There are 3 primary modes of trading. The first is discretionary trading, where a trader makes buy/sell decisions based on any number of factors, some of which can be programmed, and others – such as intuition and hunches – which cannot. Many discretionary traders stare at charts or price ladders on a computer screen for hours at a time, buying and selling as they go along.

The second type of trading is algo trading. In years past, it was called mechanical, systematic, black box or rule based trading. Now most people refer to it as algorithmic or algo trading, but the idea has not changed. The core philosophy is that all the rules for buying and selling (the “trading system” or “trading strategy”) are 100% defined, and strictly followed. This makes algo trading ideal for a computer to execute, and even run automated in real time – without human intervention. One huge benefit of this style of trading is the rules can be historically tested, known as a "backtest." By running a backtest, you can gain confidence in a trading algo before putting money behind it. If the rules were not profitable in the past, they likely won’t be in the future!

The third type of trading combines discretionary and algo trading. This is known as a hybrid or gray box approach. For example, maybe the entries are based on a trader’s intuition, with only the exit rules computerized.

For the discussion below, we will concentrate on the second approach – pure algo trading - 100% computerized rules for buying and selling any instrument. We will look to algo trade on an exchange, which is just a physical or virtual setting where buyers and sellers can execute trades.

Why Should You Listen To Me?

Now that we have basic terms down, you might be wondering why you should listen to me. First, I have been algo trading for over 25 years, and most importantly, not always successfully. Over the years, I have learned and overcome the pitfalls in trading system design that plague many traders. This took years of hard work and tuition (losses) paid to the market. But eventually, I was able to make a go of algo trading, and finished in 1st or 2nd place 3 years in a row in a worldwide, real money futures trading contest, earning over 100% in each of those years.

I was also able to achieve the goal that tantalizes so many part-time hobby traders - making the leap to full time trading, which I still do today. Along the way, I wrote 3 best selling algo trading books, and I share my experiences around the world through workshops, classes and conferences.

So, along with my early trading failures, I have had verified trading success. That is important, since many trading educators have never even traded successfully! The process I lay out later in this article is truly from someone who has “been there, and done that.”

Software Requirements

When personal computers first came on the scene, the software choices for programming trading systems were minuscule. In today’s world, however, the opposite is true. There are so many choices it is hard to decide what to use. A great article on Hackernoon describes some excellent choices: https://hackernoon.com/9-great-tools-for-algo-trading-e0938a6856cd.

Another route you can go is to purchase a retail trading platform, such as Tradestation, Multicharts, or NinjaTrader. For many traders, these platforms work perfectly well, and do everything a trader needs to do.

Bitcoin Chart, from Tradestation Trading Platform

Of course, some programmers will want to program their own backtesting and execution platform – that is what I did 20 some years ago, before I realized it was better in the long run to just use an established platform (I have used Tradestation for over 15 years. Full disclosure: I have a rebate program with Tradestation for attendees of my workshop). With all the excellent computer trading tools and packages out there, I do not recommend this total “do it yourself” approach.

Skills Every Algo Trader Needs

To be a successful algo trader, you must have a few essential skills. First, you should be able to trade, or at least know the basics of trading. Do you know what a stop order is? Or limit order? Do you know the margin requirements for the market you want to trade? Is the exchange where you are trading regulated? Questions like this are important. For example, it is critical you realize the risk inherent in unregulated exchanges.

Do you know specifics of the instrument you want to trade? For example, if you trade live cattle futures, do you know how to avoid having 40,000 pounds of live cattle delivered to your front yard? I doubt it has ever happened to a trader, but it is certainly possible. The more you know about trading in general, the easier the algo trading process will be.

A second skill is being good at math. You should have a good understanding of financial calculations, basic statistics and computing trading performance metrics. A related skill is being good with Excel or other data manipulation software such as Matlab. You will be using such software a lot to supplement your trading strategy analysis, so the better off you are at math, the better you will be at algo trading.

The third important skill is to know how to run your chosen trading platform. This seems like a basic skill, but I always tell traders that they should keep learning their platform until they can fool it – i.e., they can create trading systems that exploit weaknesses in the platform’s backtest engine. By being skilled enough to trick the software, you can avoid many rookie and intermediate level mistakes.

Being able to follow an established scientific approach to trading system development is a third skill every good algo trader has. To create solid trading systems, you have to have a sound process for designing, developing and testing your algo strategies. It is not as simple as just programming and trading. If you do not have the skills or ability to follow a set process, algo trading might not be for you.

The final skill you need to have algo trading success is arguably the most important - programming ability. Remember a while back when I discussed trading software? Well, a key part of knowing which piece of software to use is knowing your programming abilities. Different platforms require different programming abilities, with some platforms requiring C++ type programming skills, while others might only require drag and drop visual programming skills. The key is to be proficient in whatever programming language is required.

Code For Simple Algo Trading System

Successful algo traders program hundreds or even thousands of trading systems over the course of a year. That is because most trading systems are worthless – they lose money in the long run. Can you imagine paying someone to program worthless strategies for you? I sure can’t! So, programming ability is well worth your time if you want to be a successful algo trader.

What Not To Do in Algo Trading

Before I discuss a solid, proven process to developing profitable algo trading systems, it is worth pointing out some of the things NOT to do. Almost every new algo trader falls into these pitfalls, but with a little forewarning, you can easily avoid them. Speaking from personal experience, steering around these traps will save you a lot of money.

First, since many algo traders have programming, science and math backgrounds, they believe that their models need to be complicated. After all, financial markets are complex beasts, and more trading rules and variables should be better able to model that behavior. WRONG! More rules and variables are not better at all. Yes, complicated models will fit historic data better, but financial markets are noisy. Many times, having a lot of rules just models the noise better, not the actual underlying market signal. Most professional algo traders have simple models, since those tend to work the best going forward on unseen data.

Once a trading system model is complete, the second pitfall becomes an issue: optimizing. Just because you have variables (such as moving average lengths, or overbought/oversold thresholds) that could be optimized does not mean they should be optimized. And just because your computer can run a million backtest iterations an hour does not mean you should. Optimizing is great for creating awesome backtests, but remember most of the market data is just noise. A trading strategy optimized for a noisy historical price signal does not translate well to future performance.

A third pitfall is related to the first two pitfalls: building a great backtest. When you are developing an algo system, the only feedback you get on how good it may be is via the historical backtest. So naturally most traders attempt to make the backtest as perfect as possible. An experienced algo trader, however, remembers that the backtest does not matter nearly as much as real time performance. Yes, a backtest should be profitable, but when you find yourself trying to improve the backtest performance, you are in danger of falling into this trap.

Improperly Built Trading Systems Usually Perform Poorly in Real Time

A fourth and final algo trading pitfall is the “too good to be true” trap. Be wary of any historical result that just looks too good to be true. Chances are it won’t perform nearly as well going forward, it if performs at all. Almost every algo trader I know has developed at least one “Holy Grail” trading system, one with historical performance that would astound any investor or trader. But almost without exception, those great strategies fall apart in real time. Maybe it was due to a programming error, over-optimization or tricking the strategy backtest engine, but having a healthy dose a skepticism at the outset keeps you away from strategies like this.

A Proven Process For Developing Algo Trading Systems

Once you avoid the common pitfalls in algo trading, it is time to develop strategies in a controlled, repeatable process. I call my process a Strategy Factory, where trading ideas come in as raw material, “machines” turn ideas into fully tested strategies, and what leaves the factory is either a tradable strategy or a discarded scrap strategy. The steps I use to create a strategy are given below.

The process starts with goals and objectives. Like driving a car to a destination, you have to know where you want to end up before you begin. Identify the market you want to trade, and also the annual return and drawdown you desire. You can have more goals than that, so that is really the bare minimum. Having solid goals and objectives will help you know when you should be satisfied with the trading algo you created, and will help you avoid many of the pitfalls described earlier.

Next, you need an idea to build a strategy with. This does not mean you need to develop a whole economic theory for your strategy, but it also means that randomly generating ideas (such as: buy if the close of 53 bars ago is greater than the close of 22 bars ago) probably will not work. The best ideas have an explanation behind them. For example, “price moving up tends to keep moving up” might be a good idea to code and turn into a strategy. The nice thing is ideas are everywhere, and you can simply modify the ideas you find, tailoring them to fit your desires. Final note: always be on the lookout for trading ideas. You will need to test a lot of them to find a good one.

The next step is to historically test your strategy. I usually run this as two separate steps. First, I run a small scale test over a few years of data, to see if my strategy has any merit. Most strategies fail this step, so it saves me the time and aggravation of a full scale test. I also modify the strategy at this point, if I need to. I can do this without fear of overfitting or curvefitting the strategy to the historical data, since I am only using a few years of data.

Once I have a successful initial test, I then do a more in-depth test. I use a process called walkforward testing, which is superior to a traditional optimized backtest. You could also do out of sample testing at this point. The key is not to test too much during this step. The more testing you do, the more likely your model is going to be curve or overfitted.

After I have a successful walkforward test, I run some random Monte Carlo simulations with my model, to establish its return to drawdown characteristics. You want to have a trading system that provides an acceptable return to drawdown ratio – otherwise why trade it? The flip side, though, is that if the return/drawdown is too good, it generally indicates a trading strategy that has been overfit (discussed earlier as a “too good to be true” trading system).

With historical backtesting completed, I now watch the trading strategy live. Does it fall apart in real time? Many poorly built strategies do. It is important that you verify that the trading system still performs well in the real time market. That makes this step very important, even though it is extremely difficult to do. After all, who wants to spend months watching a trading system they just created, rather than actually trading it? But patience is key, and trust me when I say doing this step will save you money in the long run.

The final hurdle before turning the strategy on is to examine and compare it to your existing portfolio. At this point, you want to ensure that your strategies have low correlation with each other. Excel or other data analysis software is ideal for this task. Trading 5 bitcoin strategies simultaneously is pointless if they are highly correlated. The idea behind trading multiple strategies is to reduce risk through diversification, not to concentrate or magnify it.

Of course, at the end of development, if the strategy has passed all the tests, it is time to turn it on and trade with real money. Usually, this can be automated on your computer or virtual private server, which frees you up to develop the next strategy. At the same time, though, you need to put checks in place to monitor the live strategies. This is critical, but thankfully it is not a cumbersome chore. Knowing when to turn off a misbehaving algo strategy is an important part of live trading.

Where Do You Go From Here?

If you have made it this far, you certainly now have the basics to get started in algo trading. But what’s next?

The first step is to decide if algo trading is really something you want to jump into. Assuming you have the programming skills, you also need the desire. Don’t do it because you have dollar signs in your eyes. Do it for the challenge of cracking the market’s code. Don’t try to force yourself to algo trade if it does not feel appropriate. Good trading means not forcing things – your trading should fit your personality, skills and abilities.

Next, if you have not already, select a trading platform, learn to program strategies with it, and start developing some simple trading algos. Examine sample algos, and try to modify them. Hands on experience with programming trading systems is key, so start as soon as you can. Become as proficient as you can with the trading software and programming of strategies.

There are a few right ways to develop an algo trading system, and many more wrong ways. I’ve already shared some of the good ways, and also some of the bad ways. You might want to take some time, do some research, and search out experts in algo trading who share their methods. Just watch out, as most educators are charlatans who only trade on a simulator. Ask for student references, look for independent verification of trading results, etc. Be skeptical – your algo career depends on doing things correctly, and learning from the correct teacher.

Of course, you could learn to algo trade the way I did:

1. Program an algo, test minimally

2. Trade it live with real money

3. Lose money when incorrectly built algo falls apart

4. Tell yourself it will be different next time

5. Start over at step 1.

On second thought don’t do it the way I did – it is way too expensive!

The next step, once you have a trading system you feel good about, is to take the plunge and trade on a small scale with real money. Trading with real money changes things. I know a lot of simulated trading millionaires, but very few real money trading millionaires.

While it is good to start trading with real money, don’t trade too early in your algo career, especially if you have limited trading capital. Many traders fall into this trap, and blow out their accounts before they really understand what is going on. The markets will always be here, but you can’t participate unless you have trading capital.

Risk Of Ruin Becomes Extreme For Small Accounts (Results for a Sample Algo)

The final step, once you have developed some trading systems and commenced live trading, is to review your performance and improve. Be honest with yourself. If trading is not going well, ask yourself what you can do to improve. It might be changing your development process, or your position sizing approach, or even just trading different markets. The key is you should constantly be looking to get better. After all, there are tons of new algo traders trying to beat you.

A Final Thought

To sum up, keep in mind that algo trading is hard. Being a great programmer is only part of the puzzle. But with the right skills, desire and development process, becoming successful at developing algo trading systems is definitely possible.