Authors:
(1) Hoon Kim, Beeble AI, and contributed equally to this work;
(2) Minje Jang, Beeble AI, and contributed equally to this work;
(3) Wonjun Yoon, Beeble AI, and contributed equally to this work;
(4) Jisoo Lee, Beeble AI, and contributed equally to this work;
(5) Donghyun Na, Beeble AI, and contributed equally to this work;
(6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 13 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction
2. Related Work
3. SwitchLight and 3.1. Preliminaries
3.2. Problem Formulation
3.3. Architecture
3.4. Objectives
4. Multi-Masked Autoencoder Pre-training
5. Data
6. Experiments
7. Conclusion Appendix A. Implementation Details
B. User Study Interface
C. Video Demonstration
D. Additional Qualitative Results & References C. Video Demonstration We present a detailed video demonstration of our SwitchLight framework. Initially, we use real-world videos from Pexels [1] to showcase its robust generalizability and practicality. Then, for state-of-the-art comparisons, we utilize the FFHQ dataset [25] to demonstrate its advanced relighting capabilities over previous methods. The presentation includes several key components: De-rendering: This stage demonstrates the extraction of normal, albedo, roughness, and reflectivity attributes from any given image, a process known as inverse rendering.


Neural Relighting: Leveraging these intrinsic properties, our system adeptly relights images to align with a new, specified target lighting environment.


Real-time Physically Based Rendering (PBR): Utilizing the Three.js framework and integrating derived intrinsic properties with the Cook-Torrance reflectance model, we facilitate real-time rendering. This enables achieving 30 fps on a MacBook Pro with an Apple M1 chip (8-core CPU and 8-core GPU) and 16 GB of RAM.


Copy Light: Leveraging SwitchLight’s ability to predict lighting conditions of a given input image, we explore an intriguing application. This process involves two images, a source and a reference. We first extract their intrinsic surface attributes and lighting conditions. Then, by combining the source intrinsic attributes with the reference lighting condition, we generate a new, relit image. In this image, the source foreground remains unchanged, but its lighting is altered to match that of the reference image.


State-of-the-Art Comparisons: We benchmark our framework against leading methods, specifically Total Relight [34] and Lumos [52], to highlight substantial performance improvements over these approaches. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Authors: Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 13 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 13 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 13 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 13 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.3. Architecture 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 5. Data 6. Experiments 7. Conclusion Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.2. Problem Formulation 3.3. Architecture 3.3. Architecture 3.4. Objectives 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 4. Multi-Masked Autoencoder Pre-training 5. Data 5. Data 6. Experiments 6. Experiments 7. Conclusion 7. Conclusion Appendix A. Implementation Details B. User Study Interface C. Video Demonstration D. Additional Qualitative Results & References A. Implementation Details A. Implementation Details B. User Study Interface B. User Study Interface C. Video Demonstration C. Video Demonstration D. Additional Qualitative Results & References D. Additional Qualitative Results & References C. Video Demonstration We present a detailed video demonstration of our SwitchLight framework. Initially, we use real-world videos from Pexels [1] to showcase its robust generalizability and practicality. Then, for state-of-the-art comparisons, we utilize the FFHQ dataset [25] to demonstrate its advanced relighting capabilities over previous methods. The presentation includes several key components: De-rendering: This stage demonstrates the extraction of normal, albedo, roughness, and reflectivity attributes from any given image, a process known as inverse rendering. Neural Relighting: Leveraging these intrinsic properties, our system adeptly relights images to align with a new, specified target lighting environment. Real-time Physically Based Rendering (PBR): Utilizing the Three.js framework and integrating derived intrinsic properties with the Cook-Torrance reflectance model, we facilitate real-time rendering. This enables achieving 30 fps on a MacBook Pro with an Apple M1 chip (8-core CPU and 8-core GPU) and 16 GB of RAM. Copy Light: Leveraging SwitchLight’s ability to predict lighting conditions of a given input image, we explore an intriguing application. This process involves two images, a source and a reference. We first extract their intrinsic surface attributes and lighting conditions. Then, by combining the source intrinsic attributes with the reference lighting condition, we generate a new, relit image. In this image, the source foreground remains unchanged, but its lighting is altered to match that of the reference image. State-of-the-Art Comparisons: We benchmark our framework against leading methods, specifically Total Relight [34] and Lumos [52], to highlight substantial performance improvements over these approaches. De-rendering: This stage demonstrates the extraction of normal, albedo, roughness, and reflectivity attributes from any given image, a process known as inverse rendering. De-rendering: This stage demonstrates the extraction of normal, albedo, roughness, and reflectivity attributes from any given image, a process known as inverse rendering. De-rendering: Neural Relighting: Leveraging these intrinsic properties, our system adeptly relights images to align with a new, specified target lighting environment. Neural Relighting: Leveraging these intrinsic properties, our system adeptly relights images to align with a new, specified target lighting environment. Neural Relighting: Real-time Physically Based Rendering (PBR): Utilizing the Three.js framework and integrating derived intrinsic properties with the Cook-Torrance reflectance model, we facilitate real-time rendering. This enables achieving 30 fps on a MacBook Pro with an Apple M1 chip (8-core CPU and 8-core GPU) and 16 GB of RAM. Real-time Physically Based Rendering (PBR): Utilizing the Three.js framework and integrating derived intrinsic properties with the Cook-Torrance reflectance model, we facilitate real-time rendering. This enables achieving 30 fps on a MacBook Pro with an Apple M1 chip (8-core CPU and 8-core GPU) and 16 GB of RAM. Real-time Physically Based Rendering (PBR): Copy Light: Leveraging SwitchLight’s ability to predict lighting conditions of a given input image, we explore an intriguing application. This process involves two images, a source and a reference. We first extract their intrinsic surface attributes and lighting conditions. Then, by combining the source intrinsic attributes with the reference lighting condition, we generate a new, relit image. In this image, the source foreground remains unchanged, but its lighting is altered to match that of the reference image. Copy Light: Leveraging SwitchLight’s ability to predict lighting conditions of a given input image, we explore an intriguing application. This process involves two images, a source and a reference. We first extract their intrinsic surface attributes and lighting conditions. Then, by combining the source intrinsic attributes with the reference lighting condition, we generate a new, relit image. In this image, the source foreground remains unchanged, but its lighting is altered to match that of the reference image. Copy Light: State-of-the-Art Comparisons: We benchmark our framework against leading methods, specifically Total Relight [34] and Lumos [52], to highlight substantial performance improvements over these approaches. State-of-the-Art Comparisons: We benchmark our framework against leading methods, specifically Total Relight [34] and Lumos [52], to highlight substantial performance improvements over these approaches. State-of-the-Art Comparisons: This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Researchers Develop AI That Can Copy, Adjust, and Perfect Lighting in Any Image

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

12 Key Aspects for Assessing the Power of Text-to-Image Models

Beeble Researchers Use Physics-Based Models to Achieve Realistic Lighting Effects in Images

See the Stunning Details This AI Preserves in Relit Photos

New Framework by Beeble Researchers Promises to Bring Realistic Glow to Digital Portraits Using AI

New AI Relighting Model Outperforms Previous Models

12 Key Aspects for Assessing the Power of Text-to-Image Models

Beeble Researchers Use Physics-Based Models to Achieve Realistic Lighting Effects in Images

See the Stunning Details This AI Preserves in Relit Photos

New Framework by Beeble Researchers Promises to Bring Realistic Glow to Digital Portraits Using AI

New AI Relighting Model Outperforms Previous Models

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps