High-Resolution Photorealistic Image Translation in Real Time

You can apply any design, lighting, or graphics style to your 4K image in real-time using this new machine learning-based approach! If you think this looks interesting, watch the video on this topic and read more about it from the references below 👇 Watch the video References ►Read the full article: https://www.louisbouchard.ai/4k-image-translation-in-real-time/ ►Liang, Jie and Zeng, Hui and Zhang, Lei, (2021), "High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network", https://export.arxiv.org/pdf/2105.09188.pdf ►Code: https://github.com/csjliang/LPTN Video Transcript 00:00 You've all seen these kinds of pictures where a person's face is "toonified" into an anime 00:05 character. 00:06 Many of you must have seen other kinds of image transformations like this, where an 00:10 image is changed to follow the style of a certain artist. 00:13 Here, an even more challenging task could be something like this, where an image is 00:18 transformed into another season or time of the day. 00:21 What you have not seen yet is the time it takes to produce these results and the actual 00:25 resolutions of the produced pictures. 00:28 This new paper is completely transparent towards this as it attacks exactly this problem. 00:33 Indeed, compared to most approaches, they translate high-definition 4K images, and this 00:38 is done in real-time. 00:39 In this work, they showed their results on season translation, night and day translations, 00:45 and photo retouching, which you've been looking at for the last minute. 00:49 This task is also known as 'image-to-image translation', and all the results you see 00:53 here were produced in 4K. 00:55 Of course, this video is not in 4K, and the images were taken from their paper, so it 01:00 might not look that high-quality here. 01:03 Please look at their paper or try their code if you are not convinced! 01:07 These are the most amazing results of this paper. 01:09 Here, you can see their technique below called LPTN, which stands for Laplacian Pyramid Translation 01:16 Network. 01:17 Look at how much less time it took LPTN to produce the image translations where most 01:22 approaches cannot even do it as this amount of definition is just too computationally 01:27 demanding. 01:28 And yes, this is in seconds. 01:30 They could translate 4K images in not even a tenth of a second using a single regular 01:36 GPU. 01:37 It is faster than all these approaches on 480p image translations! 01:41 And yes, it is not eight times faster, but 80 times faster on average! 01:46 But how is that possible? 01:48 How can they be so much more efficient and still produce amazing and high-quality results? 01:53 This is achieved by optimizing the fact that illumination and color manipulation, which 01:58 relates to the style of an image, is contained in the low-frequency component of an image. 02:03 Whereas the content details, which we want to keep when translating an image into another 02:08 style, can be adaptively refined on high-frequency components. 02:13 This is where it becomes interesting. 02:15 These two components can be divided into two tasks that can be performed simultaneously 02:19 by the GPU. 02:21 Indeed, they split the image into low-resolution and high-resolution components, use a network 02:26 to process the information of the low-frequency or the style of the image, 02:30 and render a final image merging this processed style with the refined high-frequency component, 02:37 which is the details of the image but adapted by a smaller sub-network to fit the new style. 02:43 Thus dodging the unavoidable heavy computation consumption when processing the high-resolution 02:48 components in the whole network. 02:50 This has been a long-standing studied field achieved with a popular technique called Laplacian 02:55 Pyramid. 02:57 The main idea of this Laplacian Pyramid method is to decompose the image into high and low-frequency 03:02 segments and reconstruct it afterward. 03:05 First, we produce an average of the initial image, making it blurry and removing high-frequency 03:11 components. 03:12 This is done using a kernel that passes through the whole image to round batches of pixels 03:17 together. 03:18 For example, if they take a 3 by 3 kernel, it would go through the whole image averaging 03:23 3 by 3 patches removing all unique values. 03:26 They are basically blurring the image by softening the edges. 03:30 Then, the difference between this blurry image and the initial image is saved to use at the 03:35 end of the algorithm to re-introduce the details, which are the high-frequency components. 03:41 This is repeated three times with bigger and bigger averaging kernels producing smaller 03:47 and smaller low-frequency versions of the image having less and less high-frequency 03:52 details. 03:53 If you remember, these low-frequency versions of the image contain information about the 03:57 colors in the image and illumination. 03:59 Indeed, they are basically just a blurred low-quality version of our image, which is 04:03 why the model is so much more efficient. 04:06 This is convenient since they are smaller versions of the image, and this is the exact 04:11 information we are trying to change when translating the image into another style. 04:16 Meaning that using these low-frequency versions is much more computationally efficient than 04:21 using the whole image directly, but they are also focused on the information we want to 04:26 change in the image, which is why the results are so great. 04:30 This lower-quality version of the image can be easily translated using an encoder-decoder, 04:35 just like any other image translation technique we previously mentioned, but since it is done 04:40 on a much lower quality image and a much smaller image, 04:43 it is exponentially faster to process. 04:47 The best thing is that the quality of the results only depends on the initially saved 04:52 high-frequency versions of the image sent as input which is not processed throughout 04:57 the whole network. 04:59 This high-frequency information is simply merged at the end of the processing with the 05:03 low-frequency image to improve the details. 05:06 Basically, it is so much faster because the researchers split the image's information 05:11 in two: low-frequency general information and detailed high-frequency information. 05:17 Then, they send only the computational-friendly part of the image, which is exactly what we 05:23 want to transform, the blurry, low-quality general style of the image, or in other words: 05:29 the low-frequency information. 05:30 Then, only fast and straightforward transformations are done on the high-frequency parts of the 05:36 image to resize them and merge them with the blurry newly-stylized image, 05:42 improving the results by adding details on all edges in the picture. 05:46 And voilà! 05:47 You have your results with a fraction of the time and computational power needed. 05:51 This is brilliant, and the code is publicly available if you would like to try it, which 05:56 is always cool! 05:58 As always, the links to the complete article and references are in the description of the 06:02 video. 06:03 Thank you for watching!