Why big data and algorithms won’t improve business strategy

… for most companies.

Over the years, I’ve heard a lot of people talk about algorithmic business and how big data will improve business strategy. For most, I strongly suspect it won’t. I originally wrote this post in February 2015 and what I’ve observed since then makes me feel it is worth repeating. To explain the problem, I’m going to have say certain things that many will find uncomfortable.

Chess and business

I want you to imagine you live in a world where everyone plays Chess against everyone else. How good you are at Chess really matters, it determines your status and your wealth. Everyone plays everyone through a large computer grid. Ranks of Chess players are created and those that win are celebrated. Competition is rife.

The oddest part of this however is that no-one in this world has actually seen a Chessboard. When you play Chess in this world, you do so through a control panel and it looks like this …

Each player takes it in turn to press a piece. Each player is aware of what piece the other player pressed. White starts first.

Then Black counters, then White, then Black and so the game continues building up the sequence of presses.

The game continues until at some point, usually after a long sequence of presses, the system announces that someone has won or it is a draw.

Unbeknownst to either player, there is actually a board and when they press a piece then a piece of that type is randomly selected. That piece is then moved on the board to a randomly selected position from all the legal moves that exist. White’s opening move might be to select ‘Pawn’, one of White’s eight pawns would then be moved one or two spaces forward. But all of this is hidden from the players, they don’t know any of this exists, they have no concept there is a board … all they see is the control panel and the sequence of presses.

However, we have the sequence! Within that sequence people will find ‘favourable’ patterns e.g. the best players seem to press the Queen as often as possible! People will come up with their own favourite combinations and try to learn the combinations of other winners!

“When your opponent moves the Knight, you should counter with Queen, Queen, Rook” etc.

People in this world would write books on secret knowledge such as “The art of the Bishop” or “The ten most popular sequences of Successful Winners”. As you aggregate more and more data from different games in ever larger “big data” systems and AI is applied then the more patterns will be proclaimed and the more secrets of success announced in popular publications. You can already imagine that article in Harvard Business Review collecting sequences from over 100,000 games and announcing the killer combination of Queen, Queen, Pawn, Knight.

Remember this isn’t because people are daft, they simply have no concept of the board in this world. Some through experience might even gain a vague notion that there is a landscape, a board on which we are playing. Let us suppose one of those people — by luck, by accident or by whatever means — twigs that a landscape does exist and somehow finds a way to interact with it. Imagine one day, you play that individual but they can see something quite remarkable … the board. You start off planning to use your favourite opening combination of Pawn, Pawn, Knight. You make your first move.

They counter with Pawn, you make your next move, they counter and you’ve lost.

“What the fiddlesticks happened there!” you scream. Obviously, this was just luck! But just in case this is a new secret sequence, you scribble down their presses. “Pawn, Queen — wow what a simple sequence!”

But every time you play this player, you lose and you lose fast. You keep recording their sequences and even try to use their sequence against them. They beat you in two presses — Pawn (b) and Queen (b) (known as Fool’s Mate) or they beat you in four presses — Pawn (b), Queen(b), Bishop(b), Queen(b) (known as Scholar’s mate) but you can’t replicate their success even though you copy their sequences, it never works and you keep on losing.

People will be amazed by the incredible skill of this player but also confused that they can’t replicate the success. They will start to seek all different forms of answers to explain what is happening. Maybe it’s not just the sequence but the way that the player presses the button? Maybe it’s timing? Maybe it’s what they had for lunch? Maybe it’s their attitude, a cultural thing? They seem to be happy (because they’re winning). Is happiness the secret to winning? All sorts of correlations and speculations will be jumped upon.

However there is no luck to this or special way of pressing buttons. The player controlling Black simply has far greater situational awareness. The problem for the player controlling White is they have no real understanding of the context of the game they are playing. White’s control panel is just a shadow of the landscape and the sequence of presses lacks any positional information or consistent understanding of movement on the board. When faced with a player who does understand the environment then no amount of large scale data analysis on combinations of sequences of presses through the control panel or application of artificial intelligence or algorithms that is going to help you.

I need to emphasise this point of understanding the landscape and the importance of situational awareness and so we’ll now turn our attention to another story — Themistocles.

Themistocles SWOT

The battle of Thermopylae (the tale of the three hundred) and the clash between the Greeks and the mighty army of Xerxes has echoed throughout history as a lesson in the force multiplier effect of landscape. Themistocles devised a strategy whereby the Athenian navy would block the straits of Artemisium forcing the Persians along the coastal road into the narrow pass of Thermopylae where a smaller force could be used to hold back the vastly greater Persian numbers. I’ve provided a map of this below.

Now, Themistocles had options. He could defend around Athens or defend around Thebes but he chose to block the straits and exploit the landscape. Each of Themistocles’ options represents a different WHERE on the map. In the same way that our enlightened Chess player has many WHEREs to move pieces on the board. By understanding the landscape you can make a choice based upon WHY one move is better than another. This is a key point to understand.

People often talk about the importance of WHY in strategy but WHY is a relative statement i.e. WHY here over there. To answer the question of WHY you need to first understand WHERE and that requires situational awareness.

Now imagine you’re in that Greek army. Themistocles had turned up on the eve of battle, he had enthused the troops with purpose and a goal — “We need to defend the Greek states against the horde of Xerxes”. Let us assume the Greek army (a collection of state armies) had varying but overall decent principles e.g. working as a team, being skilled in what you do, collaborating with each other and were well trained. But now imagine, that no-one on the Greek side had a concept of maps or even an understanding of maps.

Themistocles steps forward and tells the troops to have “no fear, we are the greatest warriors in the world” as he announces his great vision in the form of a SWOT (strengths, weaknesses, opportunities and threats) diagram.

How confident would you feel? I know I’d be running for the hills. Now in terms of combat, I’d hope you’d agree that a strategy based upon an understanding of the landscape is going to be vastly superior than a strategy that is derived from a SWOT. Equally from the point of view of communication and learning then a map beats a SWOT hands down.

The question you need to ask yourself however is — what do we most commonly use in business? SWOT or MAP?

On business and situational awareness

The problem that exists in most businesses is that they have little or no situational awareness. Most executives are even unaware that you can map the business environment and increase situational awareness. They are usually like the Chess Players using the Control Panel and oblivious to the existence of the board, relying on magic secrets of success from other players — “I hear Amazon just did this, we should do the same”

I say oblivious because these people aren’t daft, they just don’t know that a landscape exists. But how can we be sure this is happening? Language and behaviour are often our biggest clues. If you take our two Chess players (White with little or no situational awareness, Black with high levels) then you often see a marked difference in behaviour and language.

Those with high levels of situational awareness often talk about positioning and movement. Their strategy is derived from WHERE and so they can clearly articulate WHY they are making a specific choice. They tend to use visual representation (the board, the map) to articulate encounters both present and past. They learn from those visualisations. They learn to anticipate others moves and prepare counters. Those around them can clearly articulate the strategy from the map. Those with situational awareness talk about the importance of WHERE — “we need to drive them into the pass”, “we need to own this part of the market”. They describe how strategy is derived from understanding the context, the environment and exploiting it to your favour. The strategy always adapt to that context.

Those with low levels of situational awareness often talk about the HOW, WHAT and WHEN of action (e.g. the combination of presses). They tend to focus on execution as the key. They poorly articulate the WHY having little or no understanding of potential WHEREs. They have little or no concept of positioning and they often look to ‘copy’ the moves of others — “Company X gained these benefits from implementing a digital first strategy, we should do the same”. Any strategy they have is often gut feel, story telling, alchemy, copying, magic numbers and bereft of any means of visualising the playing field. Those around them often exhibit confusion when asked to describe the strategy with any precision beyond vague platitudes. Those who lack situational awareness often talk about the importance of WHY? They describe how strategy should be derived from your vision (i.e. general aspirations). They often look at you agog when you ask them “Can you show me a map of your competitive environment?”

Back in 2012, I interviewed 160 different high tech companies in Silicon Valley looking at their level of strategic play (specifically their situational awareness) versus their use of open (whether source, hardware, process, APIs or data) as a way of manipulating their environment. This was part of a study for the Leading Edge Forum and we considered that these companies were the most competitive of the competitive.

What the study showed was there was significant difference in the levels of strategic play and situational awareness between companies. When you add in a tendency towards action and performance measure by market cap changes over the last seven years then those with high levels of strategic play had performed vastly better than those with low levels.

Leading Edge Forum 2012 study on situational awareness versus action

Now this was Silicon Valley and a select few. I’ve conducted various interviews since then and I’ve come to conclusion that less than 1% of companies have any mechanism of visualising their environment and continuously learning from it. The vast majority suffer from little to no situational awareness but then they are competing in a world against others who also lack situational awareness.

This is probably why the majority of strategy documents contain a tyranny of action and why most companies seem to duplicate strategy memes and rely on backward causality. We should be digital first, cloud first, we need a strategy for big data, social media, cloud, insight from data, IoT … yada yada. It’s also undoubtably why companies get disrupted by predictable changes and are unable to overcome their inertia to the change.

At the same time, I’ve seen a select group of companies and parts of different Governments use mapping to remarkable effect and discovered others who have equivalent mental models and use this to devastate opponents. Now, I don’t believe we’re anywhere near the pinnacle of situational awareness in business — far better methods, techniques and models will be discovered than those that exist today.

Which brings me back to the title. In the game of Chess above, yes you can use large scale data analytics, AI and algorithms to discover new patterns in the sequences of presses and certainly this will help you against equally blind competitors. Such techniques will also help you in business improve your supply chain or understand user behaviour or marketing or loyalty programmes or operational performance or any number of areas in which we have some understanding of the environment.

But this won’t help you in strategy against the player with better situational awareness. Most business strategy itself operates in a near vacuum of situational awareness. For the vast majority then I’ve yet to see any real evidence to suggest that big data is going to improve this. There are a few and rare exceptions but in general, the key is first to understand the landscape and that a landscape exists.

Alas for most, it will become hunting for that magic sequence … Pawn, Pawn … what? I’ve lost again?

If you’re going — “Well, how do you map a business?” — it’s fairly easy. To help you on your way, I’m writing a book, all creative commons Attribution-ShareAlike 4.0 International on just how to do that with a technique that I’ve been using for over a decade. Please feel free to help yourself.

Originally published at blog.gardeviance.org on January 8, 2017.