A beginner’s guide to Computer Vision in Retail

Anyone with a wet finger in the air will by now have heard of the “retail apocalypse” sweeping through the developed world’s malls. “People aren’t spending in stores anymore”, your quarter-informed uncle complains, before moaning that youths are too busy Instagramming their avocado brunches to burn crosses on people’s lawns. Indeed, the old retailing models aren’t working as well as they used to. The fact that they were terrible models to start with probably had something to do with it.

The most forward-thinking retailers are however adapting, and are not only surviving, but thriving. This is in part due to new technologies, which have enabled a wide range of innovations. Least of these technologies is Computer Vision, which can use troves of customer videos to model, analyse and recognise at levels, speeds and a regularity we meat sacks could never even hope to achieve.

Though its use is yet to be democratised, the next few paragraphs give a cursory glance at the areas where the modern store owner might use this technology to stay afloat throughout this Christmas season and the next, proving that artificial intelligence is not only the realm of digital brands, and does indeed have a place in physical retail. But first…

What The Hell is Computer Vision ?

I know, I know, you just got the hang of machine learning, and are in no mood to get into yet another technical lesson, so we’ll make it easy:

Computer Vision is an algorithm that “sees”.

Simple enough, right? More specifically, it is an algorithm which is trained (using videos and images) to process, analyse and identify patterns, and which can then label the world around it accordingly, and make decisions. The tech’s goal is simple : it aims at automating what the human visual system can do.

The reality is immensely complex, and worthy of its own article. Do not hesitate to click HERE for slightly a deeper dive. Onward with the show, now.

Marketing

Though loyalty cards are currently the best way to adapt promotional offers to customers using good ol’ machine learning, this may soon change. The retailing model developed by Amazon Go entails a systematic identification via mobile to enter the store, followed by constant tracking within the store using computer vision. It looks something like this :

From there, it’s just a matter of time before one’s actions are linked to one’s account within a database thanks to facial recognition and tracking. This is fantastic for marketing professionals looking to increase up-selling and complementary purchases :

the key question shifts from “where” and “how many” to “who”, “when”, “how often”, “how long” and “for how many cookies?”.

We’re not yet in a world where you pick up a box of cookies, put it back down in the store, and an email pops up giving you a discount on cookies once you’re home yet, but it hasn’t stopped TwentyBN from mixing computer vision to analyze what shoppers are doing with some very basic conversational notions to create a bot that might push customers to spend more. Check it out in the video below.

The idea of identifying clients thanks to facial recognition would also be particularly attractive to high fashion stores where high spenders could be identified, and their habits, past purchases and shipping address could appear on a screen for all sales associates to see. This is already the case in luxury stores like Louis Vuitton, but in a much more analogue way (ie: 2 gifts and 2 different bills, one paid with card and the other with cash…). Candy retailer Lolli & Pops have already implemented this to identify loyalty members as they walk into the store. By treating them as individuals (and VIPs) the system instills brand loyalty, and converts occasional shoppers into regular customers. Both of which are good for business.

The cameras are already in most stores and are pretty useless as it is — why not make good use of them for once?

Merchandising

There was a time when most store owners would know most of their clients. But those clients wanted cheaper products. And more variety. So the industry concentrated. And expanded. And cities grew. And shoppers became simple numbers, as Traffic and Conversion metrics ruled over Retail-City with an iron fist. This doesn’t cut it anymore, especially for younger generations, cocooned as they are by brands looking for life-long customers (looking at you, Juul). Customers want products that fit their needs, right here and right now.

Ironically enough, technology could help bring more humanity into retail. Thanks to facial recognition, computer vision could potentially identify different demographics if given enough training footage, helping inform strategic decisions about inventory (more cat foods where old ladies live, more condoms where teenagers shop…). This leads to a more efficient supply chain, with a better use of available resources.

Computer vision could also be used to identify what draws peoples’ eyes and what doesn’t, as well as create heat-maps for stores, helping change and adapt layouts depending on the finding.

Think of it as the equivalent of 1000 interns per store writing down what people look like and what they’re looking at each day.

Those interns are arguably cheaper in the U.S. because there are no labor laws there, but for civilized countries, computer vision might very well be the way to go. Check out RetailNext for more info.

Cashierless Stores

As mentioned above, the most advanced example of a cashierless store for now is Amazon Go, which lets you enter its stores via mobile identification, and leave without taking out your wallet. Items are identified via computer vision and the client’s account is automatically billed through the app once the store’s threshold is passed.

The use of mobile is, of course, a fantastic idea: if we throw privacy out the window, the store could track where we went before, where we go after, and link cookies to store behavior. There also is a major implication to these types of stores becoming prevalent: theft prevention becomes much, much easier.

Check out this pretty great non-promotional video to know more.

Yes , this man looks like he has a REALLY annoying voice, but it’s not so bad, promise

Furthermore, it’s likely that the average basket size will increase under this model (basic human psychology), and that we’ll see a rise in both spontaneous purchases and of the average number of store visits per customer. If you don’t have the impression you’re paying, it’s easy to over-do it. More advantages include a smaller workforce and 24/7 opening times, all of which could help salvage stores’ ridiculously thin margins. Some retailers will argue that it will allow them to better reallocate their workforce, but come on,who are they kidding. Given the large number of people relying on retail for jobs, technological joblessness is likely to become a real worry in coming months.

Check out AiFi, Standard Cognition, Zippin and Trigo Vision for more info on cashierless stores.

Stock Visibility

There are a few options to optimize stock visibility using computer vision, but all of them go through a phase of training an algorithm to recognize ranges of products based on their packaging, and knowing how to count how many there are, how many there aren’t and, more importantly, how many there should be.

The latter is important because the till may know how many products there are in stock and how many were sold, but it doesn’t know if theft or counting errors (ie: suppliers skimming off the top) led to fewer physical products in store.

Indeed, cameras could be used to drastically limit theft by identifying when and where they occur. If we can see that an item has been picked up, we can also see that it’s been hidden. We could also potentially check for suspicious behavior from both customers ad employees based on past data (as shown here by startup StopLift). The ROI is also very easy to calculate: stores know very precisely how much they lose to theft and errors. As such, this use case is likely to be one of the first implemented.

Stock visibility is also key to optimizing something many retailers are keen to automate: restocking. It’s indeed repetitive and involves products which tend to be of the same shape, which is ideal for a robot going through aisles (the robotics of it all is another story entirely). Employees could then be better used for higher-value tasks. or simply be fired, as mentioned above.

One of the most original such ideas comes from startup Focal Systems, which aims to equip existing shopping carts with computer tablets that use computer vision to monitor the shelves as the cart moves through the store, and also display digital ads to the shopper (1 stone, 2 birds).

Why use robotics when the customer is the cheapest robot out there?

Conclusion

Much of the above may sound like science fiction. And to most retailers, it is. Though there’s a lot of talk about the blurring between online and offline, most companies first need to concentrate on optimizing physical retail as it is (workforce training, supply chain, pricing…). God knows there’s a lot of work.

Beyond building healthy foundations, none of the above is possible without vast amounts of data. This data exists for retailers, but knowing how to gather it, clean it, and use it is far from easy!

Privacy issues are also likely to abound: tracking every movement in a store and compiling data, which could be mixed with online behavior, is not the most ethical thing one can do.

Nevertheless, and despite all hurdles, there are hopes that the technology will be implemented far and wide, in one form or another, within the next 10 years. Not because it will increase profit (well, that too), but because, perhaps counter-intuitively, it will make retailing a more human experience.

Join a movement

This article was originally made for The Pourquoi Pas, an online magazine providing in-depth analyses of today’s technological challenges. Click here to access it.