In a recent publication, Apple explained how they used machine learning to directly recognize people in private photos on your iPhones and iPads without having access to your images to train their algorithms. For those of you with Apple products, you can actually research by the person in the Photos app. Indeed, using multiple machine learning-based algorithms that I will cover in this article, running privately on your device, you are able to accurately curate and organize your images and videos on iOS 15. It will recognize the different people and allow you to research in your pictures where the person appears. If you have thousands of photos like I do, you will already have different clusters each representing different people. For example, one such cluster could be all the photos where your friend John is in so that you can name it “John” and then search for images of John in your pictures to have them appear automatically. Watch the video to learn more! Watch the video References: ► Read the Full article: ►Apple, "Recognizing People in Photos Through Private On-Device Machine Learning", (2021), ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/how-apple-photos-recognizes-people/ https://machinelearning.apple.com/research/recognizing-people-photos https://www.louisbouchard.ai/newsletter/ Video Transcript 00:00 In a recent publication, Apple explained how they used machine learning to directly recognize 00:04 people in private photos on your iPhones and iPads without having access to your images 00:09 to train their algorithms. 00:11 I personally do not have an iPhone, so I cannot test it myself, but I am looking for an iPad 00:15 to draw explanations and write math equations and stream it during calls. 00:19 If some of you guys use tablets to do that, please let me know what you think is the best 00:24 to get! 00:25 For those of you with Apple products, you can actually research by the person in the 00:28 Photos app. 00:30 Indeed, using multiple machine learning-based algorithms that I will cover in this video, 00:34 running privately on your device, you are able to accurately curate and organize your 00:39 images and videos on iOS 15. 00:42 It will recognize the different people and allow you to research in your pictures where 00:45 the person appears. 00:47 If you have thousands of photos like I do, you will already have different clusters each 00:50 representing different people. 00:52 For example, one such cluster could be all the photos where your friend John is in so 00:57 that you can name it "John" and then search for images of John in your pictures to have 01:02 them appear automatically. 01:04 It can even recognize photos where the same people frequently appear, even if it doesn't 01:08 know the persons individually or hasn't been directly trained with it, and use it to share 01:13 memories like the "Together" feature shown here. 01:16 This is a super cool built-in application by Apple, and the best is that it even works 01:21 when the face is occluded or sideways, as we will see. 01:24 As I said, it seems to work really well. 01:27 It entirely runs on your device privately, and they are always improving the algorithms, 01:32 but it's even cooler to know how it works, so let dive into it! 01:36 This task of recognizing people in your own picture is extremely challenging because of 01:40 the variability your photos will have. 01:43 Different people, different angles, different scales, different lightings, occlusions because 01:47 your friend was catching a football, or even from other cameras. 01:52 If we would strictly base ourselves on the person's face, this would be pretty incomplete 01:57 as most of our pictures taken on the spot during an event aren't perfect images with 02:02 your friends smiling in front of the camera. 02:04 When you type in John, you'd like to see these events where John won the game by catching 02:09 this ball. 02:10 To attack this, they start by locating the faces and upper bodies of people visible in 02:15 the image using a first detection algorithm. 02:18 This algorithm was trained on many labeled human examples annotated with where the bodies 02:22 and the faces were. 02:24 Meaning that they trained a deep neural network with images sent as inputs, and the outputs 02:28 were only the cropped version of the image with either the bodies or faces of the people. 02:34 This is done by feeding many examples to the network, helping it showing where to focus 02:38 its attention with the correct identified sections. 02:42 This way, it can iteratively learn to find this body part by itself afterward if we show 02:47 it enough examples during training. 02:49 photo: figure 2 cursor + network (U-net, VAE?) with images and box over other image and incorrect 02:50 box until it's better? 02:51 By the way, if you find this interesting, don't forget to subscribe, like the video, 02:53 and share it with your friends or colleagues, it helps a lot! 02:57 Thank you! 02:58 Then, they match the bodies and faces of each individual to have even more data about the 03:03 person in case only one of the two appears in a future image. 03:06 You can see here that both the body and face are sent into a separate model that encodes 03:12 the information, creating embeddings. 03:14 These embeddings are simply the most valuable information about the face and body of the 03:18 person. 03:19 Here, we use another network to encode the information because we want our embeddings 03:23 to be similar for the same person and different for different individuals. 03:27 This is again done with another model that will look like this, inspired by mobilenet, 03:32 which I talked about in my convolutional neural network video. 03:36 It is a lightweight convolutional neural network that can run extremely efficiently, made for 03:41 mobile instead of GPUs. 03:43 If you are not familiar with CNNs, I strongly invite you to watch the video I made explaining 03:48 them simply. 03:49 Basically, it takes the cropped images and compresses the information in a smaller space 03:53 focusing on the most interesting details about the individual. 03:57 This is possible because such a model was trained on a lot of images to do exactly that. 04:02 Then, these embeddings are merged and saved in your phone's gallery unless they have poor 04:07 responses. 04:09 These poor responses may come from unclear faces or upper bodies and would be automatically 04:14 filtered out. 04:15 This is repeated with all your pictures to create clusters out of these embeddings. 04:20 These clusters will be the different people identified. 04:22 It will merge all similar embeddings in small groups where each group is a specific individual. 04:28 So this is the step where all the pictures where John was identified are put into a gallery. 04:33 And what's cool is that this automatically runs during nighttime when your phone charges 04:38 while you sleep and keeps on improving the more pictures you have. 04:42 So once these clusters are created, your new photos containing people are sent to the same 04:46 deep network to create a new embedding per person in the image. 04:50 This new embedding will either join a cluster if they find a match or create a new one based 04:55 on the difference between the embeddings you have in your phone and the new picture's embeddings. 05:00 Here, to find whether it is the same person or not, they focus primarily on the face. 05:04 If it's occluded or sideways, it uses the upper body coupled with what we have from 05:09 the face and takes the time of the photo into account to measure if the clothing could be 05:13 the same or different. 05:15 As you may suspect, the upper body isn't always helpful. 05:18 As they say, "We’ve carefully tuned the set of face and upper body distance thresholds 05:23 to get the most out of the upper body embedding without negatively impacting overall accuracy." 05:29 And this is how Photos regroup your friends within the application without you knowing 05:34 it! 05:36 Another concern was that they want to offer the same experience for all Apple users no 05:40 matter the photographic subject’s skin color, age, or gender. 05:44 It is great that they keep on improving the generalization and working to remove these 05:48 biases from their algorithm the best they can using the broadest datasets possible and 05:53 data augmentation to add variations to the training images. 05:57 If you have an iPhone or iPad, please let me know what you think of this feature in 06:01 the Photos app and how well it works! 06:04 Thank you for watching!
