Let’s talk about interpretability, transparency, explainability, and the trust headache Explainable (XAI) is getting a lot of attention these days and if you’re like most people, you’re drawn to it because of the conversation around AI and . If so, bad news: it can’t deliver the protection you’re hoping for. Instead, it provides a good source of incomplete . AI trust inspiration Before we get caught up in the trust hype, let’s examine the wisdom of a sentence I’ve been hearing a lot lately: “in order to trust AI, we need it to be able to explain how it made its decisions.” Complexity is the reason for all of it Some tasks are so complicated you instructions. cannot automate them by giving explicit AI is all about automating the ineffable, but don’t expect the ineffable to be easy to wrap your head around. The point of is that by explaining with , you can dodge the headache of figuring out the instructions yourself. That becomes the . AI examples instead AI algorithm’s job How could we possibly trust what we don’t understand? Now that we’re automating things where we couldn’t have handcrafted that model ( ) — because it’s way too complicated — are we seriously expecting to read it and fully grasp how it works? A recipe with a million boring items is something that a computer can remember easily, but it will overwhelm your limited human memory capacity. recipe/instructions So if we can’t just read that complicated tangle and figure out how the recipe is doing its decision-making, why should we it? trust Imagine choosing between two spaceships. Spaceship 1 comes with exact equations explaining how it works, but has never been flown. How Spaceship 2 flies is a mystery, but it has undergone extensive testing, with years of successful flights like the one you’re going on. Which spaceship would you choose? This is a philosophical question, so I can’t answer it for you. I know I have a personal preference — maybe that’s the in me — but I would choose careful testing as a better basis for . statistician trust Carefully testing your system, making sure that it works as it is supposed to — that’s what keeps you safe. Testing as a better basis for trust If we want a student to learn calculus, we want them to generalize beyond the textbook examples rather than overfit to them (“ ” translates approximately to “ ” in plain English). How do we check they’re competent? overfitting memorizing Please don’t take a scalpel and go poking around in your student’s cranial wet stuff to figure out they’re doing the calculus. That’s the equivalent of interpreting the model and I’m glad you don’t do that. You have no idea how the human brain implements calculus (since neither you nor neuroscience can describe the electrochemical signaling in there), but that’s okay — it’s not the best basis for trust anyway. how Craft exams to catch memorization and make sure they reflect the conditions where your student must perform. What you should do instead is design an exam for your student carefully and if the student — human or machine — passes your exam, then you know they’re qualified. That’s pretty much what testing means in AI too. The exam should be crafted to catch overfitting (using brand new data is the best way to foil those pesky memorizers) and relevant to the environment where the student must perform. Experts in take rigorous testing seriously and you should too. applied AI statistical Let’s talk about explainable AI Am I saying that interpretability, transparency, and explainability aren’t important? That’s not it at all. They have their place… in . analytics In many interpretability debates, you’ll notice the participants are talking past one another and misunderstanding the different areas of . They’re fundamentally interested in different classes of application. applied data science If the application involves — in other words, it’s advanced with AI — then of course you need interpretability. How are you going to use a black box for inspiration? With great difficulty, that’s how. generating inspiration analytics If you’re looking for inspiration with advanced , that’s a different goal from building a safe and reliable automated system for decisions at scale, where performance matters most. If your project truly demands both, you can blend two objectives for a price: your result will likely be worse at each of the objectives than if you’d stuck with only one. Don’t pay for things you don’t need. analytics It all boils down to how you the algorithms. There’s nothing to argue about if you frame the discussion in terms of the project goals. apply Often, the people arguing are . Their job is building general-purpose tools which don’t have a business project (or project goals) yet, so like salespeople peddling potato peelers, they are motivated to extoll the virtues of their wares to anyone who will listen, whether or not their captive audience is in danger of ever preparing food. “You this potato peeler” simply isn’t true for everyone — it really depends on your project. Same goes for interpretability and XAI. researchers need Fascination with mechanism If you’re fascinated with how something works (mechanism) for its own sake, that’s the instinct that got trained into you by your STEM classes. It helps you build a new student, a new brain, a new spaceship, a new . research microwave Much of the confusion comes from . Arguments that are appropriate for researchers (build better spaceships) make little sense for those who apply AI (solve problems using existing spaceships). not knowing which AI business you’re in In , a love of mechanism is a great instinct for ; seeing how something works can reveal potential threats and opportunities. If you’re mining data for inspiration, chuck all black boxes out with yesterday’s trash! applied data science analysts Unfortunately, that same instinct will lead you astray if your goal is performance. The mechanism inside the , or , of 1739. Canard Digérateur Digesting Duck Popular nonsense concerning humans Many people demand mechanism as a prerequisite for trust. They have a knee-jerk reaction to : AI “If I don’t know how it’s doing it, I can’t trust it to make decisions.” If you refuse to trust decision-making to something whose process you don’t understand, then you should fire all your human workers, because no one knows how the brain (with its hundred billion neurons!) makes decisions. Holding AI to superhuman standards If you require an interpretation of how a person came to a decision at the level, you should only be satisfied with an answer in terms of electrical signals and neurotransmitters moving from brain cell to brain cell. Do any of your friends describe the reason they ordered coffee instead of tea in terms of chemicals and synapses? Of course not. model Instead people do something else: examine the information and their choices, and then sing you a pretty song that tries to make sense of everything in hindsight. That’s XAI essentially, though human explanations aren’t always right. Behavioral economists get a kick out of planting decisions in their unsuspecting victims — er, experimental participants — and then enjoy listening to incorrect stories all about “why” the people made the decisions (which the experimenter actually made for them). Whenever humans make up a convenient, oversimplified story that fits the inputs and outputs in hindsight, good news! You always have access to the same level of explainability for any model; it can even be model-agnostic. Just look at your input and output data, then tell a fun story. That’s what is all about. analytics When it is simpler than the truth, your explanation is technically a lie. It might inspire you, but it’s closer to a safety blanket than a safety net. Adding a layer of is a good idea wherever you can afford it (remember, don’t take what you see ). The wiser efforts in XAI focus on with inputs and outputs. Sure, they make it sound like something brand new is being invented when it’s just good old fashioned “look at your data and check it makes sense” but I can’t fault the sensibility. The only issue I take with it is that it’s sold to you as a basis for . XAI is many good things, but the way it’s invoked in trust discussions is an audacious con. The explanation will always be horribly oversimplified, so it won’t be true. Oh dear. analytics too seriously analytics trust Explainability provides a cartoon sketch of a but it doesn’t provide the of decision-making. It’s not safe to take a cartoon sketch as more than inspiration, and you’d do well to remember that trust based on XAI is like trust based on a few pieces out of a giant puzzle. why, how Analytics without testing is a one-way ticket to a false sense of security. But let’s get back to the kind of AI where the is interpretable, where we’re not focused on what’s happening in the data. In other words, we want to understand the equivalent of the functioning of ~100,000,000,000 little cells in your skull to explain how you chose that coffee. model really Why can’t we have both? In a perfect world, you’d like perfect performance and perfect interpretability, but usually real life forces you to choose. You know what we call a task where you have both? can Simple, easy, and probably already solved without AI. So let’s talk about those hairy, complicated, and incomprehensible tasks. The ones your brain has evolved to do without telling you how it does them. Or the ones where the signal is a faint needle spread across a huge haystack of . The ones that force you to resort to AI in the first place. For those, you have to choose between: feature combinations you it but it well. Interpretability: do understand doesn’t work you it but it well. Performance: don’t understand does work Remember, explainability can be model-agnostic, so it’s almost a separate analytics project you can tack on if you have the time and energy. True model interpretability, however, will cost you performance if the task requires a complex recipe. When performance matters most, don’t limit your solutions to what the simple human mind can wrap itself around. You want to automate the ineffable, ? The whole point is that some tasks really are so complicated that a good solution will be a rat’s nest of complexity, so you won’t understand the model. At best you’ll be able to with . remember oversimplify analytics Don’t limit your solutions to what the simple human mind can wrap itself around. To really succeed at these tasks with high performance, you have to let go. If you wanted your senses to be controlled by a simple number of brain cells working understandably, they wouldn’t be very effective senses. A fruitfly’s simple brain is easier to understand than yours, but I bet you wouldn’t willingly trade places. You like your performance better. Requiring all models to be interpretable is like demanding that no helper’s brain is more complicated than a fruitfly’s. There’s only so much a fly can help you with. Instead, opt for the trust that comes from making sure that you’re able to verify that your system does, in fact, work. It’s nice to have both, of course, and some tasks are simple enough that they can be solved in a way that delivers all your wishes, but if you can’t have both, isn’t it’s best to go straight for what’s most relevant? That’s performance on well-crafted tests. Hubris and the danger of preferring mechanism Some people still prefer mechanism as their ideal basis for trust. They’d choose the untested spaceship whose physics they’ve read. Lest hubris get the better of us, it’s worth appreciating that mechanism is a step away from performance. People who prefer information about how something works may be trusting too much in their own human ability to leap from complex mechanism to expected performance. Let’s hope they’re clever enough to avoid a nasty splat. Not everything in life is simple In a nutshell, simple solutions didn’t work for tasks that need complicated solutions, so AI comes to the rescue with… complicated solutions. Wishing complicated things were simple does not make them so. If a law says your inherently complicated thing has to be simple, it’s a polite way of saying you can’t have the thing at all. (Which is sometimes for the best.) Why can make solutions that are more complicated than the code you handcraft? A computer can remember (save to disk) a billion examples perfectly in a way that you can’t, so it doesn’t fuzz out the nuances in the same way you do. It doesn’t get bored by writing out a million lines of instructions. Computer memory is nothing new, but with modern computing power we can exploit it at scale. You may have had a few millennia of simplistic recipes that fit in human memory, but now it’s time to turn a new leaf. AI algorithms Today, humanity’s playbook has expanded to automating tasks with complicated recipes. Some of them are so complicated you can’t explain them in three sentences over a beer. We’d best get used to it.

Different

Mind

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Explainable AI won’t deliver. Here’s why.

Too Long; Didn't Read

Companies Mentioned

Cassie Kozyrkov

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Explainable AI won’t deliver. Here’s why.

Too Long; Didn't Read

Companies Mentioned

Cassie Kozyrkov

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES