Algorithmic trading is a data-driven application. It is the ability to derive intelligence from data — above all — that gives the data science approach entailed in systematic trading an edge over intuitive approaches associated with discretionary trading.
However, it would be wrong — simplistic — to assume that trading intelligence emerges from data science alone.
Trading intelligence emerges from harnessing the interactions of multiple complex concepts, and thus, a holistic approach is essential.
Grasping the concepts involved in trading — and their interactions — should be the first and foremost requirement when starting a trading automation project. The second is a software infrastructure that contributes to harnessing those concepts.
This article condenses three years’ worth of field experience of the author and his partners. It is oriented to developers, data scientists, and traders considering entering the field of algorithmic trading.
In this article, I break down what is required for trading intelligence to emerge, with insight distilled from thousands of hours spent trying many different approaches. I unpack lessons learned in the process of developing the group’s trading automation infrastructure and shed light on tools open sourced to the community along the way.
Disambiguation
Let’s start by agreeing on the meaning of three terms that are often used loosely in the literature, which tends to favor confusion.
A trading system is a framework for handling the low-level logic that serves to structure the processes and methods used to implement and deploy trading strategies.
A trading strategy is the description of a set of actions or events run in stages. Events are triggered upon the validation of precise conditions describing specific market situations. Trading strategies are designed to achieve a specific goal within the broader plan of a trading system via taking and managing positions.
A trading bot is a computer program that, based on available data sets, applies the trading logic defined in a trading strategy while running trading
sessions. Those sessions may be testing sessions that may involve simulations, or they may be live trading sessions that involve placing and managing real orders at the exchange.
sessions. Those sessions may be testing sessions that may involve simulations, or they may be live trading sessions that involve placing and managing real orders at the exchange.
The Bird’s Eye View
In general terms, a trading strategy features a set of instructions whose goal is to make trading decisions based on market data analysis. The quality of the trading logic is, therefore, closely correlated with data analysis capabilities.
Let’s focus on human-programmed trading logic first and leave AI implementations for future articles.
Trading ideas emerge from data analysis and data visualization.
To come up with the rules for when to sell or buy an asset, traders need to formulate hypotheses based on trading ideas. One of the most popular realms of trading ideas derives from what is known as technical analysis.
While markets may be thought by some to behave in a stochastic manner, most traders believe that the psychology of humans making trading decisions affects the behavior of markets. Because humans tend to be driven (or at least affected) by emotions such as fear or greed, they tend to react in a somewhat predictable manner when exposed to environmental stimuli. In a way, traders believe that this leads to patterns emerging within the otherwise chaotic movements of the price (of assets).
There certainly are other forces that affect the price, and that may result in the emergence of patterns at different time scales, like micro and macroeconomic principles affecting the particular market, supply and demand, the interactions with other markets, global politics, asset fundamentals, and so on. It is the multiple and unconstrained sets of variables affecting markets that make them so complex and impossible to figure out.
Because algorithmic trading is a data-driven application, and because human-programmed trading logic derives mostly from the observation of patterns, we can conclude that data visualization is a crucial component too. There is a reason why discretionary traders spend countless hours staring at market charts: to make trading decisions based on what they see. As an algorithmic trader, you too spend time on the charts — a small fraction of what a traditional trader would — in your case, to come up with trading ideas that you later implement as rules in your trading strategies.
Trading strategies result from the implementation, testing, and tuning of trading ideas.
Once a trading idea is described as a set of rules, you need to test your strategy and see how it performs both over historical data — what we call backtesting — and over a live data feed — or what we call paper trading.
The testing of trading strategies is a crucial aspect of systematic trading, as the precision and reliability of tests directly affect how tests map with reality. For your tests to have any value predicting live trading performance, they need to take into account how live trading works and all the variables that affect a live trading session — for example, exchange fees and slippage, that is, the difference between the rate at which orders are placed and the rate at which they are actually filled by the exchanges’ order matching engine.
Live trading performance emerges from the optimal deployment of automated trading strategies.
The deployment of automated strategies in live trading sessions is comparable to a mission-critical financial application that handles real money. The trading bot must be robust and must go through substantial testing and QA to be trusted.
It also needs to be flexible and feature functionality to keep track of strategies’ activities, such as keeping the accounts, providing the capacity to administer, move, or cancel orders, and so on.
We will touch on other considerations later on. For the time being, it is worth noting that algorithmic trading requires a fair bit of infrastructure.
Harnessing the Key Concepts
As transpires from the Bird’s Eye View section, there is substantial software infrastructure required to reliably design, test, and deploy trading strategies. Let’s briefly review the most important features of such an infrastructure, which we will call the platform from now on.
Exchange connectivity
Crypto-exchanges come in and go out of the public’s favor all the time. When we first started working on trading bots, Poloniex was at the top of the food chain. A few months later, by the time our infrastructure was in a
usable alpha stage, Poloniex was nowhere to be seen in the top-20 list of exchanges.
usable alpha stage, Poloniex was nowhere to be seen in the top-20 list of exchanges.
The reason why trading at popular exchanges is important is that only popular exchanges can offer proper liquidity, that is, sufficient volume of bids (buy orders) and asks (sell orders) in the order book to guarantee that no single order affects price significantly.
You don’t want to commit to any particular exchange. At the very least, your platform should be able to connect with more than one of the popular exchanges. This is particularly important if you wish to exploit the inefficiencies of exchanges, for instance, determining if certain exchanges lead major price movements in relation to other exchanges.
Exchanges expose APIs, which may evolve and change over time. Maintaining different connectors is a long-term challenge.
Your platform should be able to handle all sorts of connectivity issues and guarantee the integrity of data extracted from exchanges — and of transactions — at all times.
Connectivity through the internet is tricky for mission-critical applications. There are tons of issues that arise regularly that may break, delay, or corrupt your interactions with the exchange, at the application level, at the network level, at the exchange’s API and internal systems levels, and everywhere in between.
Data mining
You need reliable market data both to test the implementation of the trading idea and to make trading decisions. This means that you need to handle both a historic data set and a live data feed from exchanges.
Exchanges serve raw trades or candles data in small chunks. You want to have access to the complete market history so that you may apply statistical studies and use long-term indicators such as moving averages.
A single hole in the data set may skew your data analysis in unpredictable manners, so the processes for extracting data from exchanges and building your data sets must be bulletproof.
Raw market data is the starting point for numerous transformations that you need to make in search of patterns, correlations, and trading opportunities.
Your platform should provide typical technical analysis studies, but should also provide the tools to build your own.
That is what may differentiate the intelligence you may be able to distill
from the data from what is readily available to common traders.
from the data from what is readily available to common traders.
Data visualization
Humans are visual creatures. While our brains are enclosed in black boxes,
they have evolved sensors to perceive the environment and send valuable
data for the central nervous system to interpret. Vision is our primary sensor.
they have evolved sensors to perceive the environment and send valuable
data for the central nervous system to interpret. Vision is our primary sensor.
Algorithms are good at crunching numbers, but as a human, you greatly benefit from being able to visualize data. It is you that need to find the inspiration in data to come up with trading ideas.
Visualizing data is key for developing trading ideas. Your platform would benefit from an integrated charting system to make the most out of the data.
The ideal case scenario would be counting with features that allow plotting all sorts of data over typical market charts, where you may compare data sets to find visual clues that algorithms would miss.
Strategy testing
You should be able to seamlessly backtest your algorithms over the historic data sets. You also want to be able to test the strategy forwards with live data, both without placing actual orders at the exchange — called paper trading — and placing real orders with a small fraction of your capital — called forward testing.
Neither backtest performance nor forward test performance are accurate
predictors of future performance, but if your strategies don’t do well in tests, it would be unreasonable to trust them with your capital to trade live.
predictors of future performance, but if your strategies don’t do well in tests, it would be unreasonable to trust them with your capital to trade live.
The trading bot should be able to simulate live trading as accurately as
possible when backtesting and paper trading. It needs to simulate exchange events such as the filling of orders, covering the nuances of order tracking and administration described next in this article under the Order Management section. The more that simulations resemble reality, the more likely they are to predict live trading performance.
possible when backtesting and paper trading. It needs to simulate exchange events such as the filling of orders, covering the nuances of order tracking and administration described next in this article under the Order Management section. The more that simulations resemble reality, the more likely they are to predict live trading performance.
That is the basic testing functionality required before you deploy your strategies to trade live. However, there are more advanced platform features that may have a huge impact on the performance of your strategies. Some of them are useful at an earlier stage, during the strategy design and tuning phase.
Such as, for instance, the ability to analyze strategy behavior not just on a consolidated backtest performance report, but on a per-trade basis graphical simulation, directly over the charts.
In the image above, three consecutive positions are simulated depicting entry and exit events, as well as dynamic take profit and stop loss targets. These sorts of features make the granular analysis of trading rules possible and take fine-tuning and optimization to the next level.
Similarly, counting with a robust strategy debugging solution is highly desirable. It is of the utmost importance to have access to the runtime information processed by the trading bot at all times to easily detect the potential shortcomings of rules definitions.
Order management
The placing of orders may seem like a simple problem at a glance. However, the trading bot should provide enough intelligence to track and
administer orders too.
administer orders too.
Orders may get filled at different rates, may get incrementally filled during a period of time, or may not get filled at all. Every time an order is partially filled, the exchange charges fees on the portion filled, which need to be accounted for. The rate at which orders fill may not be the rate your algorithm expected for multiple reasons, including slippage, rounding of decimal places, etc. All of these occurrences must be handled by the trading bot so that the trading session is synchronized with what happens at the exchange.
Moreover, the trading bot should do at least some basic bookkeeping to keep track of balances so that trading sessions may properly use the allocated capital.
Scalable deployment
One of the main advantages of automated trading over manual trading is the ability to work with virtually unlimited numbers of strategies. In a developed stage, your operation may consist of multiple strategies running on multiple markets across multiple exchanges. For that ability to materialize, you need the software infrastructure to coordinate the use of resources.
The effective use of resources makes up trading intelligence too.
The image above shows a deployment of multiple coordinated tasks across multiple machines
Your platform should be able to coordinate the deployment of data mining and trading tasks across different machines so that the operation may scale as you advance in the production of strategies.
The coordination needs to be effective so that strategies running on the same market and exchange may share the same source of data.
Infrastructure scalability
A trading automation platform designed to succeed at trading should be able to cater to different approaches to finding and exploiting trading opportunities. The key concepts described earlier make up a solid piece of infrastructure suitable for technical analysis, statistical analysis, sentiment analysis, and virtually any approach based on data mining.
Where Do I Start?
There are many ways in which you may approach the prospect of getting into algorithmic crypto trading. The first question to settle relates to goals: What is your goal?
If all you wish to do is put your skills to the test, give it a shot, and see how it goes, then you probably don’t need to concern yourself with anything you have read in this article.
Take this article seriously if your goal is to succeed at algorithmic trading. Success means achieving a positive and reliable trading performance. In laypersons’ terms, success entails making money — consistently.
Fortunately, to harness the key concepts that foster the emergence of trading intelligence, you don’t need to build the whole infrastructure yourself. In fact, it would be reasonable and even desirable to avoid that.
Coding, tuning, and stabilizing such a vast infrastructure may take years’ worth of work of a senior developer; such is the level of complexity entailed. Instead, you may use and contribute to open source tools or entire platforms.
Your focus should be on producing the data mining that leads to the emergence of trading ideas and on implementing those ideas into trading strategies. Not only is that the fun part of algorithmic trading but also what determines if your efforts result in positive and reliable performance or not, all other things taken for granted.
If you wish to focus on developing the trading intelligence instead of the infrastructure, then the next step is surveying existing open source algorithmic trading platforms, looking for the one that may best suit your needs.
Feel free to take a look at our GitHub repository and see if what we’ve developed is of use to you. We’ve released all of our work under the Apache License 2.0.
Coming from the crypto universe, I wouldn’t even consider trusting my strategies to a corporate entity that may change its terms of service, commercial policies, business model, or even deny access at any point in time. And those are barely the mild concerns. The worst-case scenario is a corporation turned evil that uses the knowledge about your strategies to front-run them, or even sell them without you knowing. Open source software solves all of those concerns.