Bull And Bear Regime Trading - How To Algo Trade With Trends Without Getting Run Over By Them!

My wife Amy and I have different philosophies when it comes to parties. If the host says the party is from 6:00 PM to 11:00 PM, I like to be there at 6, and leave before 11. Amy, on the other hand, has no problem arriving at 7, and staying until midnight. I worry I will miss some early fun, and she worries she’ll miss some late night fun. Quite a pair we are! I thought about this recently in relation to algo trading, and market timing. Everyone wants to hop on a bull trend right at the start, and ride it until it ends. When you look at a chart after the fact, it looks SO EASY. Just buy at start, and sell at the end. Of course, you do the opposite for a bear trend. Piece of cake — LOL!!! (whom I learned a lot from, and whom I usually agree with) breaks down markets into 3 phases: Market Wizard Dr. Van Tharp Bull Market Bear Market Flat Market And within each market type, you have 2 main characteristics: Volatile Market Non-Volatile Market So, when you take all the combinations, you have 6 unique market types. Makes sense. The theory behind all these market types is this: Just find out what type of market you are in, and then simply trade a system that is best suited for that market. Sounds easy, doesn’t it? Sadly, the reality (from what I’ve found) is not that simple. Referring back to my party story, it would be similar to getting a party invitation, but without a start or end time, or even a date. Arrive too early, and you’ll be bored waiting for others. Arrive too late, and maybe most of the fun has already occurred before you got there! Worst case, you miss the party completely — not a good thing. Same thing on the back end. Stay too long, or leave too early, and chances are you’ll miss something entertaining. And that is where the bull/bear regime trading gets tricky. Successful market regime trading depends on having a solid regime identification plan/technique — in near real time. Most traders can’t do that, and that is where I disagree somewhat with Dr. Tharp that regime trading is fairly easy. My experience is that market regime timing, and turning on and off systems to match the current market, is very hard. Switching systems on and off requires a great switching mechanism, in addition to solid base strategies. But maybe we can use the general idea of market regime, only with a single strategy, not a bunch of market regime specific strategies. Then all we need is a good base strategy, and a good regime detection approach. With that in mind, let’s see if a market regime project yields any useful results in this four part article. Section 1- Bull, Bear and Flat Market Regime Trading Section 2 — Bull and Bear, With Enhanced Flat Market Regime Trading Section 3 — Volatile and Non-Volatile Regime Trading Section 4 — Bull, Bear, Flat, Volatile and Non-Volatile Regimes, All 6 Combinations Section 1 — Bull, Bear and Flat Market Regime Trading The goal of this study is not to create a finalized, ready to trade strategy, but rather to do some research, and find out if taking only long trades in a bull market, and only short trades in a bear market, is something to consider. If this approach proves worthwhile, then you could try it on existing strategies, new strategies, etc. In other words, you will have a LOT more testing to do! But that is a good thing — don;t take my word for all this — test and verify it yourself — that is what good algo trader do. Most traders, myself included, typically develop strategies that work in all markets. We accept that sometimes our strategy will underperform in certain market periods, but overall it should be worthwhile to trade. So why not create multiple strategies for each market? A bullish strategy for bull markets, a bearish one for bearish markets, and a “stay on the sidelines” strategy for flat markets? I personally look at it like this. Let’s say I develop one strategy with 2 optimized variables, over a 10 year period. “Mr. Market Timer” develops three strategies over that same period of time, each with 2 optimized variables. So, his chances of curvefitting or overfitting his models is a lot (3 times?) higher than mine — after all, he is optimizing 3 strategies over the same time period I am optimizing just one. I always think curvefitting is an evil best avoided, so minimizing variables and strategies is a very good idea. In the figure above, there is no doubt that the blue curve better fits the data than the red curve. The blue curve simply “fits” better. This is akin to a great looking backtest, compared to the red curve, a so-so looking backtest. But what about the next few points of data, ones that are not shown on the curve? Which curve will be better at “predicting” those points. Before I started developing trading strategies, I’d say the blue curve was better. But after years and seeing (and sometimes creating) curvefitted algo trading systems, I’d now opt for the simpler red curve. Simple models work better than more complicated models, at least in trading. So, as I come into this study I am not a big personal fan of bull/bear regime trading — it adds complexity to trading models, and may cause more harm than good. Maybe I will change my mind as this study unfolds. I’ll keep an open mind regarding bull/bear regime trading, since after 30 years of trading, I’ve come to realize that the market does not care about my personal preferences. That is why I use my to evaluate every potential trading strategy I have. When the analysis is done properly, the numbers don’t lie. Strategy Factory® approach Core Strategies For This Study I am going to use 4 core strategies for this project, and I’m also going to reverse the logic on each, which would give me 8 total strategies. These are just “test” strategies, not necessarily good or bad. If the current close is the highest close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #1 is a simple breakout strategy. If the current close is the highest close of the last X bars, it sell short at the open of the next bar with a market order. Vice versa for long trades. Strategy #2 is a reverse simple breakout strategy. If the RSi is less than 30 (oversold), it buys at the open of the next bar with a market order. Short trades are signaled for RSI values above 70. Strategy #3 is an RSI strategy. If the RSi is less than 30 (oversold), it sells short at the open of the next bar with a market order. Long trades are signaled for RSI values above 70. Sample code for Tradestation is shown below. Strategy #4 is a reverse RSI strategy. If the current close is greater than the close X bars ago, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #5 is a simple momentum strategy. If the current close is less than the close X bars ago, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #6 is a reverse simple momentum strategy. If the current close crosses above the average close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #7 is a traditional moving average strategy. If the current close crosses below the average close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Note that each of these core strategies is a pretty common approach to trading. I chose them because they were simple and fairly well known. Each of these strategies has a single parameter than can be optimized — the lookback length X. I will use 10, 30 or 50 bars for this optimization. Strategy #8 is a reverse traditional moving average strategy. Bull/Bear/Flat Filters For This Study There were a ton of different bull/bear filters I could have tried here. I chose 6 (one is “always trading”). If you have some cool bull/bear regime markers, just let me know and maybe I’ll test them in a follow up article. Allow long and short trades with no restrictions. Filter #1 — “Always” Trade Filter. Only allow long trades when the current close is above the average close over the last Y bars. Vice versa for short trades. This is a bull or bear filter. Tradestation code is shown below as an example. Filter #2 — Moving Average Filter. Only allow long trades when the current close is above close Y bars ago. Vice versa for short trades. This is a bull or bear filter. Filter #3 — Momentum Filter. Over the last Y bars, did the highest high occur more recently than the lowest low? If so, take only long trades. Vice versa for short trades. This is a bull or bear filter. Filter #4 — High Or Low First Filter. Only allow trades when the Y bar ADX is above 15. This value suggests at least a weak trend is in place. Below 15 for the ADX, just remain flat. This is a flat or not flat filter. Filter #5 — ADX Filter. If the RSI over the last Y bars is below 70, only allow long trades. If the RSI over the last Y bars is above 30, only allow short trades. This is a bull or bear filter. As you can see, most of the filters break down the market action into a bull market or a bear market. In part 2, I’ll take one of these filters and add specific flat regimes to it. Filter #6 — RSI Filter. I will be trying different values of Y for this study. Since the core strategy signals themselves are using short term lengths of 10–50, for the filters I will use longer lengths for Y of 100, 150 and 200. Markets and Bar Sizes For This Study I will be examining 40 different continuous contract futures markets for this study: @AD, @BO, @BP, @C, @CC, @CD, @CL, @CT, @DX, @EC, @EMD, @ES, @FC, @FV, @GC, @HG, @HO , @JY, @KC, @KW, @LC, @LH, @NG, @NK, @NQ, @O, @OJ, @PL, @RB, @RR, @S, @SB, @SF, @SI, @SM, @RTY, @TY, @US, @W, @YM I will also be using 5 bar sizes: Daily, 720 minute, 360 minute, 120 minute and 60 minute bars Other Study Info This will be a 10 year study, from Jan 2009 to Dec 2018. All results shown are for 1 contract, and include appropriate slippage and commission specific to each market. (If you did not realize, each market has its own characteristic slippage, based on its volume, liquidity, contract size, etc.) All together, we then have: 40 markets 5 bar sizes 8 strategies 3 values for length X for each strategy 6 bull/bear filters 3 values for length Y for each filter This will produce 86,400 unique iterations — WOW! You might say — this will take a long time to complete, as it is a big study with many markets and bar sizes. How can you do this in a timely fashion? Well, to help me out, I will be using Multi-Opt, a special software tool created specifically for Tradestation. This software allows for rapid testing and prototyping of strategies — just one of many timesaving tasks it performs. Multi-Opt is going to be my virtual assistant on this project. Otherwise this study would take forever! With Multi-Opt, Tradestation 9.5 and a 2 year old laptop, each part of the study takes about 7 hours to run. Results First, here is an example of what a filter does to the strategy. The top chart contains the strategy with no trend regime filter — it can go long or short at any time, without restrictions. In the bottom chart, there is a trend regime filter based on momentum. When the 100 bar momentum is up (shown with light blue bars), only long trades can be taken. When the momentum is down (shown with red bars), only short trades are permitted. With so many markets, bar sizes, etc, it is clear that just analyzing the results can be overwhelming. So, I am only looking at 2 performance metrics: I obviously could have chosen a ton of various metrics, but one metric for profit and one for risk should tell me what I want to know. Overall Net Profit Maximum Intraday Drawdown I should also point out that different metrics might lead to different conclusions. Bottom line, if you are analyzing the markets in a scientific fashion, you should pick performance criteria you feel is appropriate. You might pick wildly different metrics than I did, and that is OK. With that, let’s get started with analyzing the results. First, let’s take a look at the impact of the Regime Filter: At first glance, it appears like nothing works. No matter what filter is used, on average the strategies all lose money. That is a bit depressing, anddon’t forget — most people lose at trading! But, a second glance reveals some interesting results: Filter #1 is the baseline, which is no filter at all — all trades were taken. It is pretty bad overall. But filters #2 thru #5 show a pretty dramatic improvement in average Net Profit compared to the baseline, and also a significant reduction in maximum drawdown. This shows that the filters appear to work! Filter #6 shows basically no change from the baseline. I am guessing it is not doing much if any filtering at all. Maybe though, the averages are deceiving, and possibly skewed by a few strategies. Let’s take a look at the results for each strategy: Lots of data here, so just focus on the 2 far right columns. Black numbers show that the regime filter is better than the baseline “always trade” case. Red numbers show regime trading is worse. Sometimes the impact is small (for example with Strategy #4), and other times it is dramatic (for example with strategy #8). The results are clear — filters #2 — #5 all improve Net Profit and Max Drawdown, for each of the 8 strategies. This tells us that bull/bear regime filtering is a good idea! Using filters #2, #3 and #4, only take long trends when the long term trend indicator is indicating a bull market. Use filter #5 as a general trend / no trend indicator. What is surprising is that it does not matter much which filter is used — their results are all more or less the same. Maybe these particular filters all catch the same big trends? What about the lookback length of the filter, which in the test was set from 100–200 bars? Does this variable play a role? Again, a lot of data in the table below — just focus on the green highlighted cells. The green highlighted cells are the highest net profit and lowest drawdown for every row. A quick glance shows that in a majority of cases, the lookback length of 200 is the best — those are the columns with the most green cells. This length of 200 bars corresponds to the longest time period to measure the trend. Note that it is not 200 “days” except for daily bars. For a 60 minute chart, use 200 of the 60 minute bars. With so many markets, bar sizes, etc. there are many ways to dissect this data. But for this study I’d like to keep it general. What Conclusions Can We Draw So Far? So, the general conclusions thus far are: Trend Based Bull/Bear Regime Filters On Average Improve Net Profit, And Reduce Drawdown For Filters That Work, There Is Not A Large Degree Of Variation Between Best Filters and Worst Filters The Longer The Regime Filter Lookback Length, The Better The Results So, if I was to incorporate a bull/bear trend filter in my own testing, I would code something like this (using Momentum filter #3): Variables: CanTradeLong(False), CanTradeShort(False); CanTradeLong = False; CanTradeShort = False; If close> close [ ] Then CanTradeLong = True; If close <close [ ] Then CanTradeShort = True; 200 200 CanTradeLong = True and {my long entry rule} then buy next bar at market; CanTradeShort = True and {my short entry rule} then sellshort next bar at market; if if A Small Inspiration For You Just as an example of the possible improvement with a trend regime filter, here is strategy #4 with 120 minute Silver bars, with and without the trend filter #3 incorporated. Quite a difference! After doing this initial study, a few lingering questions still bothered me a bit. Those questions will be discussed below. What is the impact of instead of just bull/bear regimes, adding in a flat regime, to make it bull/bear/flat? Filter #6 (RSI filter) did nothing — why? Section 2 — Expanded Bull, Bear and Flat Market Regime Trading For Part 2, I am going to dig a bit more into regime filtering. Specifically, I am going to see if adding a “flat” period exclusion to an existing filter is of any help. The idea here is that you do not trade new trades during a defined flat period. It could also mean that during a flat period existing trades will be excluded (I will try both). Before I do that, here is a short video giving you highlights of the study so far: The second topic I wanted to examine was the RSI trend filter. In Part 1, I found that this filter — filter #6 — did nothing. This needs to be examined further. So first I will tackle the second topic. In Part 1, I had filter #6 — a simple RSI filter. It did pretty much nothing — why? Here is its definition: Filter #6 — RSI Filter. If the RSI over the last Y bars is below 70, only allow long trades. If the RSI over the last Y bars is above 30, only allow short trades. This is a bull or bear filter. When I added this, I really did not look too much into it, I just used fairly loose overbought and oversold levels of 30 and 70, figuring it would work. Not so fast my friend! Turns out when you calculate RSI over 100–200 bars, the value rarely, if ever, gets above 70 or below 30. Here is an example: So, that explains why this filter is useless. And the bad part is for the Part 2 study, I originally used RSI variations like this for new Filters #5 and #6. And they either do no filtering, or complete filtering (no trades at all!). For me, trying too hard usually leads to curvefitting and overfitting. As a result, I am going to eliminate RSI from trend filtering. I could probably play around with it, and get it to work, but I am always wary of trying to make something work. For me, trying too hard usually leads to curvefitting and overfitting. In Part 1, the moving average Filter (#2), the momentum Filter (#3), the High Or Low First Filter (#4) and the ADX Filter (#5) all did reasonably well, with the ADX filter being the best. For this part of the study, I am going to use the momentum filter #3, which was in the middle of the pack on filter performance. Why not just use the best (ADX #5) filter? Well, remember the goal here is not to find the best filter for trends, but rather to just see how the concept works in general across many markets and bar sizes. I think using an average performing filter can give me more insight into future performance. Of course, I could be wrong on that! With that, let’s look at candidate filters for the Part 2 study. We will have 4 filters, which I’ll identify as “Mome” filters for “momentum:” Although long and short trades with no restrictions. This should equal (or be very close to) Filter #1 in Part 1. Mome Filter #1 — “Always” Trade Filter. Only allow long trades when the current close is above close Y bars ago. Vice versa for short trades. This is a bull or bear filter. This should equal (or be very close to) Filter #3 in Part 1. Mome Filter #2 — Momentum Filter. Only allow long trades when the current close is above close Y bars ago AND if the current close is above the close Y/2 bars ago. Vice versa for short trades. This filter will be flat whenever there is conflicting short and long period momentum. Mome Filter #3 — Momentum Filter. Only allow long trades when the current close is above close Y bars ago AND if the ADX is above 15 (signifying trend mode). Vice versa for short trades. This filter will be flat when the ADX is flat. Mome Filter #4 — ADX Filter. An example of the 4 cases is shown in the figures below. This provides an idea of how selective these filters are. Here are the results for different momentum filters. Momentum 3 and 4 filters (with “flat” periods as opposed to only bull and bear periods) do perform better. Both the net profit and max drawdown improved by about 60%. That is pretty good performance improvement. When the results are examined across all 8 of the core strategies, the filters improve each strategy. Sometimes the difference is small, and sometimes it is very large. Digging a little deeper, the impact of the various filters can best be seen with the average number of trades. Compared to no filter (Mome Filter #1), many of the trades get filtered out with Mome Filter #4. An example of that is shown below: With no trades filtered out, this strategy has 1153 trades. Quite a few! Once trades are filtered out with Mome Filter #4, the number of trades drops to only 18: This is pretty typical. The filters remove many trades, especially for “mome filter #4:” So, while it is nice in general to have more trades, all other things being equal, filtering can reduce the number of bad trades, making the performance better on average. Of course, you have to watch out for “over filtering!” Some of these filters just remove too many trades. Beware of that — a small number of trades — when doing any testing. Statistically inclined traders would likely cringe when the number of trades goes below 50 or so. The smaller the number of trades, the less reliable the results will be. Overall Results I was a bit surprised that adding a “flat” period to the bull/bear regime did not improve results by that much. There was quite a bit of improvement with Mome filter #4, but the very small number of trades gives me pause. I’d be inclined to use Mome filter #3, as it reduces the number of trades but also keeps a majority of them. I should point out that I was “always in” the market, even with the flat filter. Here is an explanation: — can only initiate long trades, simultaneously exiting the short trade Bull market — can only initiate short trades, simultaneously exiting the long trade Bear market — cannot exit or enter trades during the flat period This approach does not keep from being in trades during flat periods; it just restricts trades during flat periods. That is sort of “half flat!” As a result, I decided to run the cases again, adding this code for each strategy: Flat market new This will ensure that there are not open trades during flat periods. I’ll refer to this additional condition as a “bbf” filter — for bull, bear or flat filter. New Results — Bull/Bear/Flat Filter vs. Bull/Bear Filter Here are the results for the bull/bear/flat filter (where all trades are exited in flat zones), versus the bull/bear filter (where trade in progress can remain open in flat zones): NEW FILTER ORIGINAL FILTER Results for filter #2 are pretty much the same, filter #3 is mixed (lower profit, but also lower drawdown) and filter #4 is better (but again, filter #4 does not have many trades). So, nothing definitive here — it seems to be dependent on the filter, instead of being universally good or bad. Conclusion of Section 2 When we look at net profit, both filters #4 are best. But they have a low number of trades, so they are considered suspect. So, that leaves us with this: This data is not conclusive, since the best profit filter is not the lowest drawdown filter. If you desire the best net profit, then “mome filter #3” (double momentum filter) is best, and if low drawdown is the goal, then that same filter with closing all trades during flat periods is probably best. Big Takeaways From Study, To This Point Bull/Bear/Flat Filters Work Better Than Having No Filters With Varying Results Across Filters, It Is Good To Verify Filters For Your Particular Strategy The best way to incorporate what I’ve done is to take my ideas, add them to your strategies and test them out. See what works and doesn’t work. This study isn’t meant to be “use XXXX as a filter” or anything definitive like that. Rather it is to give you ideas and concepts for you to incorporate into your own algo trading. That is where the real benefit is. In Section 3, we will take a short detour, and instead of filtering for bull/bear, we will filter for volatile and non-volatile periods. Section 3 — Volume and Volatility Filters In the first two parts, I looked at trend filters. Now in this section, I am going to focus on filters for volume and volatility. I’ll detail those in a bit, but first, just to recap: We are testing with 5 bar sizes (Daily, 720 minute, 360 minute , 120 minute, 60 minute) We are testing 40 futures markets: @AD, @BO, @BP, @C, @CC, @CD, @CL, @CT, @DX, @EC, @EMD, @ES, @FC, @FV, @GC, @HG, @HO , @JY, @KC, @KW, @LC, @LH, @NG, @NK, @NQ, @O, @OJ, @PL, @RB, @RR, @S, @SB, @SF, @SI, @SM, @RTY, @TY, @US, @W, @YM Finally, we are testing with 8 different strategies: If the current close is the highest close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #1 is a simple breakout strategy. If the current close is the highest close of the last X bars, it will sell short at the open of the next bar with a market order. Vice versa for long trades. Strategy #2 is a reverse simple breakout strategy. If the RSi is less than 30 (oversold), it buys at the open of the next bar with a market order. Short trades are signaled for RSI values above 70. Strategy #3 is an RSI strategy. If the RSi is less than 30 (oversold), it sells short at the open of the next bar with a market order. Long trades are signaled for RSI values above 70. Sample code for Tradestation is shown below. Strategy #4 is a reverse RSI strategy. If the current close is greater than the close X bars ago, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #5 is a simple momentum strategy. s a reverse simple momentum strategy. If the current close is less than the close X bars ago, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #6 i is a traditional moving average strategy. If the current close crosses above the average close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #7 is a reverse traditional moving average strategy. If the current close crosses below the average close of the last X bars, it buys at the open of the next bar with a market order. Vice versa for short trades. Strategy #8 Note that each of these core strategies is a pretty common approach to trading. I chose them not because they were necessarily good or bad, but because they were simple and fairly well known. Each of these strategies has a single parameter than can be optimized — the lookback length X. I will use 10, 30 or 50 bars for this optimization. The goal with this study is not to say “definitely trade this strategy/filter/market/etc.” but rather to give general guidance as to what might lead to fruitful strategies. Test and verify on your own — that is my mantra! For volume and volatility, there are a few schools of thought. With volume, some traders think that larger than normal volume means something significant is happening in the market. Lots of people are trading, the thinking goes, and therefore something significant price movement is in store. A great time to trade! Of course, there is an opposite view. Lower than average volume could signal indecision and uncertainty. This could happen before an explosive move. So, maybe the best time to trade is with low volume? I don’t know the answer, although I will say that based on my experience, the “trade with low volume setups” tends to work better. But maybe that is just me, I don’t know. So, I’ll test both high and low volume approaches. A similar conundrum exists for volatility, which I’ll measure with the Average True Range. Maybe if the current Average True Range is higher than normal, it is a good time to trade. Conversely, maybe a lower than normal Average True range is a better time. I honestly don’t know, so I’ll test both. I am going to test 9 different filters (I’ll refer to them as “VolVol” filters, for Volatility and Volume): VolVol Filter #1 Always allow trades (no filter baseline) VolVol Filter #2 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars ABOVE VolVol Filter #3 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars BELOW VolVol Filter #4 Allow trades only if current bar volume is average volume over the last 100,150 or 200 bars ABOVE VolVol Filter #5 Allow trades only if current bar volume is average volume over the last 100,150 or 200 bars BELOW Filters 6–9 are combination filters for both volume and volatility: VolVol Filter #6 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars ABOVE ABOVE VolVol Filter #7 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars ABOVE BELOW VolVol Filter #8 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars BELOW ABOVE VolVol Filter #9 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars BELOW BELOW All together, we are running 129,600 unique cases. Results The results are somewhat underwhelming for most of the filters. For filters 2 and 3, the average true range filters, Net Profit is worse, and the Max Drawdown is not much changed. For filters 4 and 5, the volume filters, Net Profit is worse, but the Max Drawdown is somewhat less. For all 4 of those, there is not enough evidence to say “we’ve found the Holy Grail!” :( :( For the combination filters 6–9, filters 7 & 8 both have improved Net Profit and reduced Maximum Drawdown. These cases also have a lot less trades when compared to the baseline case. This suggests that the filters are doing their job, and reducing the number of bad trades. What is interesting about filters 7 and 8 is that they are exact opposites, and both do good. Filter 7 like above average volatility, and lower than normal volume. Filter 8 likes below average volatility, and higher than normal volume. Here are sample results, with filter #7 and filter #8: Of course, there are cases where the filters make things worse, but based on the overall numbers, MOST times using filters #7 or #8 should improve performance. What I’d Be Testing On My Possible Strategies, Based On Section 3 Results I would test strategies with the following filters: VolVol Filter #7 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars ABOVE BELOW VolVol Filter #8 Allow trades only if 1 bar ATR (Average True Range) is average ATR over the last 100,150 or 200 bars AND if current bar volume is average volume over the last 100,150 or 200 bars BELOW ABOVE If you do not want to optimize for the long period length, I recommend 150 bars, as that is right in the middle of the range. Then you can save your optimization for other variables! Section 4 — Putting It All Together As a final evaluation, I’ll test the following side by side: Filter # 1 — No filter Filter #2 — Part 2 — Simple Momentum Bull/Bear Filter Filter #3 — Part 2 — Simple Momentum Bull/Bear Filter, with Part 3 VolVol filter #7 Filter #4 — Part 2 — Simple Momentum Bull/Bear Filter, with Part 3 VolVol filter #8 Filter #5 — Last Section of Part 2 — Simple Momentum Bull/Bear/Flat Filter Filter #6 — Last Section of Part 2 — Simple Momentum Bull/Bear/Flat Filter, with Part 3 VolVol filter #7 Filter #7 — Last Section of Part 2 — Simple Momentum Bull/Bear/Flat Filter, with Part 3 VolVol filter #8 These case were generally the best ones from the previous sections. Results When viewed over all 40 instruments, 5 bar sizes and 8 different strategies, overall the data shows: Bull/Bear/Flat filter is better than plain Bull/Bear filter (for drawdown, not profit — compare filters #2 and #5) VolVol filter #8 is better than VolVol #7, when combined with Bull/Bear Filter Filter #7 (Bull/Bear/Flat filter with Volvol #8 filter is best overall) This filter reduces the number of trades by quite a bit, and consequently filters out a lot of bad trades (a very good thing!) This finding holds up even when you look at individual strategies, where filter #7 is typically one of the best for Net Profit improvement and max drawdown reduction: But, a legitimate question to ask at this point is: “Sure, Net Profit improves, and Max Drawdown is lower, but how do you know that this is not just due to filtering? In other words, maybe the filters are just getting rid of a lot of trades — good and bad — and not necessarily just the bad ones!” That is the problem with big datasets and lots of performance numbers. You can easily draw different conclusions based on which data you look at! So, let’s look at a few different numbers, one familiar, one not so familiar. is a great way to see if the filters are filtering out primarily bad trades, or if they are just filtering out all trades: Average Net Profit This shows that Filters #5 and #7 are the best. Filter #5 is the simple momentum Bull/Bear/Flat filter, and Filter #7 is just Filter #5 the volume and volatility filter added on. So, using Average Trade suggests that Filter #5 is actually best. Another way to look at the data is to step back and look at the big picture. What are we really trying to do here with these filters? Really, we are trying to eliminate bad outcomes, while still keeping the good outcomes. One way to see this is by categorizing results for each iteration relative to Net Profit. For example, having an iteration with overall Net Profit above $25,000 for the test duration could be considered a “good” outcome. Conversely, a Net Profit less than -$25,000 (a big loser!) could be considered a bad outcome. So, let’s take a look at this uncommon metric: When you look at the number of cases with more than $25K Net Profit, Filter #2 is the clear winner: But when you look at the removing the bad outcomes, filter #7 is best: But both measures have drawbacks. For example, while filter #7 removes a lot of bad outcomes, it also removes a lot of good outcomes — only 22 “very profitable” cases remain! So maybe a ratio metric is in order — the ratio of very profitable cases to very unprofitable cases: The best one is now filter #2, since it has so many good outcomes relative to bad outcomes. This tells us then that the filter to choose really depends on what is most important to you: Maximizing Net Profit Minimizing Max Drawdown Maximizing Average Profit Per Trade Maximizing Good results / Bad Results What is best for me might not be best for you! In any event, here is a sample of the progress we’ve made in this study. First, we start out with a simple strategy (strategy #5 for this example). It does not look too great… CanTradeLong=True; CanTradeShort=True; CanTradeLong = True and close> close [inputvar2] then buy next bar at market; CanTradeShort = True and close close [InputVar4] Then CanTradeLong = True; If close close [inputvar2] then buy next bar at market; CanTradeShort = True and close <close [inputvar2] then sellshort next bar at market; if if Note in the chart above that there are only 31 trades in the 10 year period. Watch out for that — a small number of trades — when doing any testing. The smaller the number of trades, the less reliable the results will be. Next, we add a volume and volatility filter to the scheme. Now things are looking better! CanTradeLong = False; CanTradeShort = False; If close> close [InputVar4] and Volume>averagefc(volume,InputVar4) and AvgTrueRange( ) averagefc(volume,InputVar4) and AvgTrueRange( ) close [inputvar2] then buy next bar at market; CanTradeShort = True and close close [InputVar4] and close> close [InputVar4/ ] and Volume>averagefc(volume,InputVar4) and AvgTrueRange( ) averagefc(volume,InputVar4) and AvgTrueRange( ) close [inputvar2] then buy next bar at market; CanTradeShort = True and close <close [inputvar2] then sellshort next bar at market; // momentum with flat period and volvol filter 8 2 1 2 1 if if The best part of this transformation is that it is not just for specific cases. This improvement seems to hold up over many strategies, markets and timeframes. Now realize this is just an example — you MUST test your particular situation. Maybe it will help, but there is a chance it will not help. What I’d Be Testing, Based On This Study — Option 1 CanTradeLong = False; CanTradeShort = False; If close> close [InputVar4] and close> close [InputVar4/ ] and Volume>averagefc(volume,InputVar4) and AvgTrueRange( ) averagefc(volume,InputVar4) and AvgTrueRange( )<AvgTrueRange(InputVar4) Then CanTradeShort = True; CanTradeLong = True and YOUR LONG ENTRY then buy next bar at market; CanTradeShort = True and YOUR SHORT ENTRY then sellshort next bar at market; // momentum with flat period and volvol filter 8 //Set InputVar4 to between 100 and 200 (not much difference from 100 to 200) 2 1 2 1 if if What I’d Be Testing, Based On This Study — Option 2 CanTradeLong = False; CanTradeShort = False; If close> close [InputVar4] Then CanTradeLong = True; If close <close [InputVar4] Then CanTradeShort = True; CanTradeLong = True and YOUR LONG ENTRY then buy next bar at market; CanTradeShort = True and YOUR SHORT ENTRY then sellshort next bar at market; // momentum buy/bear filter only //Set InputVar4 to between 100 and 200 (not much difference from 100 to 200) if if Well, after a few hundred thousand tests, I am at the end of the Regime Testing journey. Thanks for reading this far! I think I have found some very interesting things, which I plan to use in my future strategy building (and maybe you should too): Taking only long trades in a bull trend (and short trades in a bear trend) is better than allowing all trades anytime Having defined flat periods can be a good thing Adding filters for higher than average volume, and lower than average volatility (Average True Range) seem to be best Results can vary, that is why incorporating regime filters should only be done as a part of a solid strategy development process (in other words, if you over-optimize these filters, you will be in WORSE shape!) The key with algo trading is to fully test any strategy before going live, and at the same time, avoid over-optimizing. It is a fine line to walk, and it is hard to do, but if algo strategy development was easy everyone would be doing it! Previously published at https://kjtradingsystems.com/bull-bear-regime-trading.html