Building Google’s Art and Culture Portrait Matcher

When my excited flatmates first showed me the portrait search functionality I was suitably amazed (and somewhat insulted) by the results. For those unfamiliar with it, this feature in Google Art and Culture allows you to find faces in Google’s digitised art collection which are similar to a selfie you submit. In my case this truly horrendous piece: Apparently my 17th century doppelgänger Thanks to a wide array of open source tools and libraries available its possible to create something less fantastical but somewhat functional within an afternoon. The rough plan First things first — find some Facial-Recognition-as-a-Service. The brilliant offers a free and disconcerting API which provides a barrage of information once fed a url of an image: Kairos Generic input image {"images":[{"faces":[{"attributes":{"age":40,"asian":0.00539,"black":0.00032,"gender":{"femaleConfidence":0.00002"maleConfidence":0.99998,"type":"M"},"glasses":"None","hispanic":0.04204,"lips":"Apart","other":0.01249,"white":0.93976}}]}]} Returned JSON data (some image info removed for length) also allows labeled images to be into a database, and when unlabelled images are passed to the API it will return the label of in the database. Kairos enrolled most similar image While this is designed to identify people who have previously been , it can perform the art recognition task pretty respectfully (when its similarity threshold is set low enough). enrolled To find suitable images to enrol I mined the very service I am trying to imitate, scraping the urls of artworks from the website. If I were to invest more than four hours into this I may take the time to label each artwork with a unique id tied to a database of images, artwork names and artists to provide a comprehensive and effortless user experience. Google Arts and Culture But I’m not. So the labels for each artwork is simply its url, so the user can be redirected to their matched artwork. from flask import Flask, redirect #Some function here, get a suitable image url as labelreturn redirect(label) Server and hosting offers a simple option to serve a webpage to a user while using python to handle the image IO and processing. Not wanting to spend too long wrestling with hosting, I simply ran the site locally. Flask Kairos requires a public image url as its input, so to expose my site beyond the local network provides a secure tunnel to my localhost. Ngrok The next challenge is to process the user input and API response while continuing to serve the site to the user. To handle the asynchronous events the library can be used to seperate the two processes. threading Finally it’s time to feed the website with selfies from friends and family, as well as some more generic individuals, the results of which are shown below! Results from my site — Successfully matches sex & ethnicity