Building Google’s Art and Culture Portrait Matcher

When my excited flatmates first showed me the portrait search functionality I was suitably amazed (and somewhat insulted) by the results. For those unfamiliar with it, this feature in Google Art and Culture allows you to find faces in Google’s digitised art collection which are similar to a selfie you submit.

In my case this truly horrendous piece:

Apparently my 17th century doppelgänger

Thanks to a wide array of open source tools and libraries available its possible to create something less fantastical but somewhat functional within an afternoon.

The rough plan

First things first — find some Facial-Recognition-as-a-Service. The brilliant Kairos offers a free and disconcerting API which provides a barrage of information once fed a url of an image:

Generic input image

Returned JSON data (some image info removed for length){"images":[{"faces":[{"attributes":{"age":40,"asian":0.00539,"black":0.00032,"gender":{"femaleConfidence":0.00002"maleConfidence":0.99998,"type":"M"},"glasses":"None","hispanic":0.04204,"lips":"Apart","other":0.01249,"white":0.93976}}]}]}

Kairos also allows labeled images to be enrolled into a database, and when unlabelled images are passed to the API it will return the label of most similar image in the database.

While this is designed to identify people who have previously been enrolled, it can perform the art recognition task pretty respectfully (when its similarity threshold is set low enough).

To find suitable images to enrol I mined the very service I am trying to imitate, scraping the urls of artworks from the Google Arts and Culture website. If I were to invest more than four hours into this I may take the time to label each artwork with a unique id tied to a database of images, artwork names and artists to provide a comprehensive and effortless user experience.

But I’m not.

So the labels for each artwork is simply its url, so the user can be redirected to their matched artwork.

from flask import Flask, redirect

#Some function here, get a suitable image url as labelreturn redirect(label)

Server and hosting

Flask offers a simple option to serve a webpage to a user while using python to handle the image IO and processing. Not wanting to spend too long wrestling with hosting, I simply ran the site locally.

Kairos requires a public image url as its input, so to expose my site beyond the local network Ngrok provides a secure tunnel to my localhost.

The next challenge is to process the user input and API response while continuing to serve the site to the user. To handle the asynchronous events the threading library can be used to seperate the two processes.

Finally it’s time to feed the website with selfies from friends and family, as well as some more generic individuals, the results of which are shown below!

Results from my site — Successfully matches sex & ethnicity