Authors:
(1) Hoon Kim, Beeble AI, and contributed equally to this work;
(2) Minje Jang, Beeble AI, and contributed equally to this work;
(3) Wonjun Yoon, Beeble AI, and contributed equally to this work;
(4) Jisoo Lee, Beeble AI, and contributed equally to this work;
(5) Donghyun Na, Beeble AI, and contributed equally to this work;
(6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 3 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction
2. Related Work
3. SwitchLight and 3.1. Preliminaries
3.2. Problem Formulation
3.3. Architecture
3.4. Objectives
4. Multi-Masked Autoencoder Pre-training
5. Data
6. Experiments
7. Conclusion Appendix A. Implementation Details
B. User Study Interface
C. Video Demonstration
D. Additional Qualitative Results & References 3. SwitchLight We introduce SwitchLight, a state-of-the-art framework for human portrait relighting, with its architectural overview presented in Fig. 2. We first provide foundational concepts in Sec. 3.1, and define the problem in Sec. 3.2. This is followed by detailing the architecture in Sec. 3.3, and lastly, we describe the loss functions used in Sec. 3.4. 3.1. Preliminaries In this section, vectors n, v, l, and h are denoted as unit vectors. Specifically, n represents the surface normal, v is the view direction, l is the incident light direction, and h is the half-vector computed from l and v. The dot product is clamped between [0..1], indicated by ⟨·⟩. Image Rendering. The primary goal of image rendering is to create a visual representation that accurately simulates the interactions between light and surfaces. These complex interactions are encapsulated by the rendering equation: A surface intrinsically exhibits both diffuse and specular reflections. The diffuse component uniformly scatters light, ensuring consistent illumination regardless of the viewing angle. In contrast, the specular component is viewing angle-dependent, producing shiny highlights that are crucial for achieving a photorealistic effect. Lambertian Diffuse Reflectance. Lambertian reflectance, a standard model for diffuse reflection, describes a uniform light scatter irrespective of the viewing angle. This ensures a consistent appearance in brightness: Here, σ is the albedo, indicating the intrinsic color and brightness of the surface. Cook-Torrance Specular Reflectance. The Cook-Torrance model, based on microfacet theory, represents surfaces as a myriad of tiny, mirror-like facets. It incorporates a roughness parameter α, which allows precise rendering of surface specular reflectance: In this model, D is the microfacet distribution function, describing the orientation of the microfacets relative to the half-vector h, G is the geometric attenuation factor, accounting for the shadowing and masking of microfacets, and F is the Fresnel term, calculating the reflectance variation depending on the viewing angle, where f0 is the surface Fresnel reflectivity at normal incidence. A lower α value implies a smoother surface with sharper specular highlights, whereas a higher α value indicates a rougher surface, resulting in more diffused reflections. By adjusting α, the Cook-Torrance model can depict a range of specular reflections. Image Formation. Upon the base rendering equation, we include the diffuse and specular components of the BRDF and derive a unified formula: where E(l) denotes the incident environmental lighting. This formula represents the core principle that an image is a product of interplay between the BRDF and lighting. To further clarify this concept, we introduce a rendering function R, succinctly modeling the process of image formation: It is important to note that since the BRDF is a function of surface properties, as detailed in Eqn. 3 and 4, we can now clearly understand that image formation is essentially governed by the interaction of surface attributes and lighting. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Authors: Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 3 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 3 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 3 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 3 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.3. Architecture 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 5. Data 6. Experiments 7. Conclusion Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.2. Problem Formulation 3.3. Architecture 3.3. Architecture 3.4. Objectives 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 4. Multi-Masked Autoencoder Pre-training 5. Data 5. Data 6. Experiments 6. Experiments 7. Conclusion 7. Conclusion Appendix A. Implementation Details B. User Study Interface C. Video Demonstration D. Additional Qualitative Results & References A. Implementation Details A. Implementation Details B. User Study Interface B. User Study Interface C. Video Demonstration C. Video Demonstration D. Additional Qualitative Results & References D. Additional Qualitative Results & References 3. SwitchLight We introduce SwitchLight, a state-of-the-art framework for human portrait relighting, with its architectural overview presented in Fig. 2. We first provide foundational concepts in Sec. 3.1, and define the problem in Sec. 3.2. This is followed by detailing the architecture in Sec. 3.3, and lastly, we describe the loss functions used in Sec. 3.4. 3.1. Preliminaries In this section, vectors n, v, l, and h are denoted as unit vectors. Specifically, n represents the surface normal, v is the view direction, l is the incident light direction, and h is the half-vector computed from l and v. The dot product is clamped between [0..1], indicated by ⟨·⟩. Image Rendering. The primary goal of image rendering is to create a visual representation that accurately simulates the interactions between light and surfaces. These complex interactions are encapsulated by the rendering equation: Image Rendering. A surface intrinsically exhibits both diffuse and specular reflections. The diffuse component uniformly scatters light, ensuring consistent illumination regardless of the viewing angle. In contrast, the specular component is viewing angle-dependent, producing shiny highlights that are crucial for achieving a photorealistic effect. Lambertian Diffuse Reflectance. Lambertian reflectance, a standard model for diffuse reflection, describes a uniform light scatter irrespective of the viewing angle. This ensures a consistent appearance in brightness: Lambertian Diffuse Reflectance. Here, σ is the albedo , indicating the intrinsic color and brightness of the surface. albedo Cook-Torrance Specular Reflectance. The Cook-Torrance model, based on microfacet theory, represents surfaces as a myriad of tiny, mirror-like facets. It incorporates a roughness parameter α, which allows precise rendering of surface specular reflectance: Cook-Torrance Specular Reflectance. In this model, D is the microfacet distribution function, describing the orientation of the microfacets relative to the half-vector h, G is the geometric attenuation factor, accounting for the shadowing and masking of microfacets, and F is the Fresnel term, calculating the reflectance variation depending on the viewing angle, where f0 is the surface Fresnel reflectivity at normal incidence. A lower α value implies a smoother surface with sharper specular highlights, whereas a higher α value indicates a rougher surface, resulting in more diffused reflections. By adjusting α, the Cook-Torrance model can depict a range of specular reflections. Image Formation. Upon the base rendering equation, we include the diffuse and specular components of the BRDF and derive a unified formula: Image Formation. where E(l) denotes the incident environmental lighting. This formula represents the core principle that an image is a product of interplay between the BRDF and lighting. To further clarify this concept, we introduce a rendering function R, succinctly modeling the process of image formation: It is important to note that since the BRDF is a function of surface properties, as detailed in Eqn. 3 and 4, we can now clearly understand that image formation is essentially governed by the interaction of surface attributes and lighting. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

This New Tech Makes Faces in Photos Look Incredibly Realistic

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

12 Key Aspects for Assessing the Power of Text-to-Image Models

Beeble Researchers Use Physics-Based Models to Achieve Realistic Lighting Effects in Images

See the Stunning Details This AI Preserves in Relit Photos

New Framework by Beeble Researchers Promises to Bring Realistic Glow to Digital Portraits Using AI

New AI Relighting Model Outperforms Previous Models

12 Key Aspects for Assessing the Power of Text-to-Image Models

Beeble Researchers Use Physics-Based Models to Achieve Realistic Lighting Effects in Images

See the Stunning Details This AI Preserves in Relit Photos

New Framework by Beeble Researchers Promises to Bring Realistic Glow to Digital Portraits Using AI

New AI Relighting Model Outperforms Previous Models

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps