Last week’s markets were a whirlwind making history for not one, but two crashes now called and On each of these days, market-wide curbs, also called trading circuit breakers, kicked in to prevent panic selling and rapid market decline when the markets dropped more than 7% after the opening bell. Black Monday (2020) Black Thursday (2020). You can read more about these market-wide circuit breakers from NYSE here . Whether the coronavirus is a or , we at PEAK6 hope you and yours are staying healthy and positive during this crisis. We also hope you can join us in acting now to . As each of us are impacted by quarantines, school shutdowns, work from home, and other social distancing, remember that our adaptability is part of our genius. Be who you ought to be and have compassion for those who are in tough spots both now and in the coming weeks. A Quick Note: black swan gray rhino flatten the curve Seeing as these events happen so infrequently, I thought I’d capture some detail around how our systems at PEAK6 experienced them. I won’t go into system volume or resiliency again as I talked about those in . Indebted to the Tech However, the graphs below show how our feed systems recorded these events with a nice big gap during the 15-minute breaks. Flag on the Play, False Start... When a market-wide circuit breaker is triggered, there’s a bit of a scramble. Similar to a false start in football with some confusion and players going back to starting positions, market participants quickly run through their order books acknowledging cancels, busts, and any other system anomalies before the markets reopen after the 15-minute pause. Unlike football, however, when the markets come back, there is actually a bit of hesitancy that’s different from a normal open. Not only do we see wide markets with large spreads on most names, but there’s almost a trickle-in effect until market activity picks back up. You can see this in our market viewing tools below. This first view shows what we’d normally expect as markets are moving and bid/asks change. However, this next view shows the market stagger with trades coming in after the circuit breaker. Both of the days we saw market curbs, we also saw markets stabilize after the break without further drops or need to trigger additional curbs later in the day. Meaning they did their job. Cool! Breaking the Cycle The circuit breaker pattern is widely used in software, just as in exchanges, to reduce harm and allow systems to stabilize. Michael Nygard’s popularized the approach back in 2007. Quick aside, Nygard also has a great talk on real-world software architecture called which you should definitely check out. Release It! Architecture without an End State , What’s neat about circuit breakers is they acknowledge that there are times when further interaction with a failing system only increases the chance of overall system failure. In this way, circuit breakers insulate a system while also allowing the system’s users a chance to fail-fast or accommodate for the failure. As a quick example, consider a simple 3-tier architecture with a UI routing traffic to three instances of a backend service. If service A3 starts experiencing issues at the data layer, it could use a circuit breaker and open the circuit, disabling the connection from the UI. This lets the UI know that the service is unhealthy while also removing stress from DB A3. If DB A3 stabilizes, then service A3 can either half-open (partially allow use while checking for errors) or close the circuit to re-enable operation. For a full rundown on the pattern, check out Martin Fowler’s write up . here Breaking Some Code There are many implementations of the circuit breaker pattern across languages, libraries, and frameworks. Netflix’s library codified a Java version of the approach back in 2012 and libraries like and are also great examples. In the code below from the goresilience library, a function called will be called wrapped within a circuit breaker. As this is code, the circuit will fluctuate from open to closed based on the provided config as e succeeds or fails every other minute. Hystrix resilience4j goresilience errorOnOddMinute example rrorOnOddMinute { minute := time.Now().Minute() minute% != { fmt.Errorf( , minute) } } { runner := circuitbreaker.New(circuitbreaker.Config{ }) { time.Sleep( * time.Millisecond) err := runner.Run(context.TODO(), errorOnOddMinute) err != { err == errors.ErrCircuitOpen { fmt.Println( ) } { fmt.Printf( , err) } } { fmt.Printf( ) } } } func errorOnOddMinute (ctx context.Context) error if 2 0 return "error because %d minute is odd" return nil func main () //ErrorPercentThresholdToOpen: 50, //MinimumRequestToOpen: 20, //SuccessfulRequiredOnHalfOpen: 1, //WaitDurationInOpenState: 5 * time.Second, //MetricsSlidingWindowBucketQuantity: 10, //MetricsBucketDuration: 1 * time.Second, for 75 if nil if "[!] circuit open" else "[-] execution error: %s\n" else "[+] good\n" Breaking Your Own Circuits One last thought on circuit breaking hits a little closer to home. The social distancing many of us are following through on because of the coronavirus is also a kind of circuit breaker. Instead of DB A3, we’re thinking about how our local hospitals and at-risk communities might be insulated from an overwhelming amount of new cases. We open the circuit between healthy, susceptible individuals and newly infected individuals without symptoms by distancing ourselves. If done effectively, it will relieve stress on our entire hospital system so it remains stable and can endure the load. I hope this sheds some light on circuit breaking as a pattern, how it shows up in the wild, and how you might use it in your own systems. If you’re interested in patterns like this and writing resilient code, check out our . We’re always looking for passionate, curious engineers who like working on hard problems! open positions
