Neuria

neuria

DIY Fake News Detector: Unmask misinformation with Recurrent Neural Networks

I was reading through my newsreader this morning and wondered what I could do if I could mainline that content right into my brain. What could you do if your noggin had access to all of the Internet's content in just a thought? Would you do it? The thought started with a series of articles on Nueralink and raised even more questions than just becoming NEO. Here are some of the questions that popped into my head. I'm just asking '"for a friend." What would the warranty be on the chip? Do I get a limited warranty on parts and labor for the life of the implant?
Do I need a subscription, and what happens if I stop paying?
Would the free plan include advertising? If I am not on a paid tier, will I get an antacid ad when I try to fall asleep?
Is the installation DIY, or must I book an appointment with a certified technician?
Is there an open-source option?
Can I fall asleep in a meeting and just let it answer for me?
What happens if my neural implant gets hacked?
If I'm unsatisfied with my neural implant, can I return it for a full refund or store credit?
What's the battery life on this thing? Will I need to plug myself in at night?
Can I resell my implant on Ebay? Mercari?
Does it ship with Amazon Prime?
Does it integrate with Zapier?
Can it run Linux?
Would I get spammed in my brain?
Can salespeople contact me through it?
Does it have an alarm clock?
Is there a payment plan, or could I get it on layaway?
Will it help me remember everyone’s name at my high school reunion?
Could it help me forget half of the people from my high school? I’m just curious about the consumer aspects of this. Honestly, I think we would turn into Jello if we could call up any fact, language, directions or ability to know everything. Why leave the house? Some of the articles that got me started. https://www.statnews.com/2024/07/08/neuralink-elon-musk-scientific-ethics-brain-computer-interface/?embedable=true https://augmentedhorizon.beehiiv.com/p/chip-every-brain-internet-rewiring-kids-minds?embedable=true https://www.hindustantimes.com/business/elon-musk-says-second-neuralink-brain-implant-to-give-people-superpowers-in-a-week-101720666555269.html?embedable=true I was reading through my newsreader this morning and wondered what I could do if I could mainline that content right into my brain. What could you do if your noggin had access to all of the Internet's content in just a thought? Would you do it? The thought started with a series of articles on Nueralink and raised even more questions than just becoming NEO. Here are some of the questions that popped into my head. I'm just asking '"for a friend." What would the warranty be on the chip? Do I get a limited warranty on parts and labor for the life of the implant? Do I need a subscription, and what happens if I stop paying? Would the free plan include advertising? If I am not on a paid tier, will I get an antacid ad when I try to fall asleep? Is the installation DIY, or must I book an appointment with a certified technician? Is there an open-source option? Can I fall asleep in a meeting and just let it answer for me? What happens if my neural implant gets hacked? If I'm unsatisfied with my neural implant, can I return it for a full refund or store credit? What's the battery life on this thing? Will I need to plug myself in at night? Can I resell my implant on Ebay? Mercari? Does it ship with Amazon Prime? Does it integrate with Zapier? Can it run Linux? Would I get spammed in my brain? Can salespeople contact me through it? Does it have an alarm clock? Is there a payment plan, or could I get it on layaway? Will it help me remember everyone’s name at my high school reunion? Could it help me forget half of the people from my high school? What would the warranty be on the chip? Do I get a limited warranty on parts and labor for the life of the implant? Do I need a subscription, and what happens if I stop paying? Would the free plan include advertising? If I am not on a paid tier, will I get an antacid ad when I try to fall asleep? Is the installation DIY, or must I book an appointment with a certified technician? Is there an open-source option? Can I fall asleep in a meeting and just let it answer for me? What happens if my neural implant gets hacked? If I'm unsatisfied with my neural implant, can I return it for a full refund or store credit? What's the battery life on this thing? Will I need to plug myself in at night? Can I resell my implant on Ebay? Mercari? Does it ship with Amazon Prime? Does it integrate with Zapier? Can it run Linux? Would I get spammed in my brain? Can salespeople contact me through it? Does it have an alarm clock? Is there a payment plan, or could I get it on layaway? Will it help me remember everyone’s name at my high school reunion? Could it help me forget half of the people from my high school? I’m just curious about the consumer aspects of this. Honestly, I think we would turn into Jello if we could call up any fact, language, directions or ability to know everything. Why leave the house? Some of the articles that got me started. https://www.statnews.com/2024/07/08/neuralink-elon-musk-scientific-ethics-brain-computer-interface/?embedable=true https://www.statnews.com/2024/07/08/neuralink-elon-musk-scientific-ethics-brain-computer-interface/?embedable=true https://augmentedhorizon.beehiiv.com/p/chip-every-brain-internet-rewiring-kids-minds?embedable=true https://augmentedhorizon.beehiiv.com/p/chip-every-brain-internet-rewiring-kids-minds?embedable=true https://www.hindustantimes.com/business/elon-musk-says-second-neuralink-brain-implant-to-give-people-superpowers-in-a-week-101720666555269.html?embedable=true https://www.hindustantimes.com/business/elon-musk-says-second-neuralink-brain-implant-to-give-people-superpowers-in-a-week-101720666555269.html?embedable=true

If You Could Get a Neural Implant to Access ChatGPT, or Any LLM, Would You Let It?

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References 5 LIMITATIONS AND CONCLUSION We have introduced a matte extraction approach using the deep image prior. The algorithm is simple, requiring only a few tens of lines of code modification to an existing U-net. Our approach is training-free and is thus particularly suitable for the diverse, few-ofa-kind subjects in entertainment video production. It also may be of intrinsic theoretical interest in terms of the nature and solution of the matte extraction problem. A further potential use would be to produce ground-truth mattes to be used for DL training. As is the case with many matting algorithms, it assumes coarse guidance in the form of a trimap or similar constraints. This can be created by the artist using readily available semi-automatic tools. Computational cost is the major limitation of the method, in common with classic methods [Levin et al. 2008]. Compute times for the examples shown in the paper are measured in minutes (but not hours) on a single previous generation Nvidia Volta GPU. This restricts the use of our algorithm to high-quality offline applications where extensive non-real-time computation is the norm, primarily movies and videos. On the other hand, the computation can take advantage of support for multiple GPUs provided in deep learning frameworks, and intermediate results can be visualized. Our method can produce temporally consistent matte extractions from video by warm-starting the optimization from the previous frame (see accompanying video), however in our experience this requires that the trimaps have smooth motion from frame-to-frame. A topic for future work is to consider recurrent or other network architectures that might make the trimap choice more forgiving. This paper has focused on introducing the DIP matting algorithm. There was relatively little architecture and parameter exploration, and further improvements may be possible. ACKNOWLEDGMENTS G.G. Heitmann, Peter Hillman, and Kathleen Beeler gave helpful insights and feedback. REFERENCES Adobe 2018. How to use Select Subject in Photoshop for One-Click Selections. https: //www.photoshopessentials.com Yagiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik. 2018. Semantic soft segmentation. ACM Trans. Graph. 37, 4 (2018). Godzilla vs. Cat 2021. https://www.youtube.com/watch?v=nf7GsKFepDg. Mikhail Erofeev, Yury Gitman, Dmitriy Vatolin, Alexey Fedorov, and Jue Wang. 2015. Perceptually Motivated Benchmark for Video Matting. In British Machine Vision Conference. Yossi Gandelsman, Assaf Shocher, and Michal Irani. 2019. "Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. In Comp. Vision and Pattern Recognition Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. 2011. A global sampling method for alpha matting. Comp. Vision and Pattern Recognition Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided Image Filtering. In European Conf. Comp. Vision. G.G. Heitman. 2020. Communication from technical artist, Weta Digital Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Int. Conf. Learning Representations. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Trans. PAMI 30, 2 (2008). Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman. 2020. Real-Time High-Resolution Background Matting. arXiv (2020). Y. A Mishima. 1992. Software Chromakeyer Using Polyhedric Slice. In NICOGRAPH. Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A Perceptually Motivated Online Benchmark for Image Matting. In Comp. Vision and Pattern Recognition. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI (2015). Mike Seymour. 2020. Art of LED Wall Virtual Production. fxguide.com. Yanan Sun, Chi-Keung Tang, and Yu-Wing Tai. 2021. Semantic Image Matting. In Comp. Vision and Pattern Recognition. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep Image Prior. In Comp. Vision and Pattern Recognition. Jue Wang and Michael F. Cohen. 2007. Image and Video Matting: A Survey. Found. Trends Comput. Graph. Vis. 3, 2 (2007). Yong Xu, Baoling Liu, Yuhui Quan, and Hui Ji. 2022. Unsupervised Deep Background Matting Using Deep Matte Prior. IEEE Trans. Circuits Syst. Video Technol. (2022). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References 5 LIMITATIONS AND CONCLUSION We have introduced a matte extraction approach using the deep image prior. The algorithm is simple, requiring only a few tens of lines of code modification to an existing U-net. Our approach is training-free and is thus particularly suitable for the diverse, few-ofa-kind subjects in entertainment video production. It also may be of intrinsic theoretical interest in terms of the nature and solution of the matte extraction problem. A further potential use would be to produce ground-truth mattes to be used for DL training. As is the case with many matting algorithms, it assumes coarse guidance in the form of a trimap or similar constraints. This can be created by the artist using readily available semi-automatic tools. Computational cost is the major limitation of the method, in common with classic methods [Levin et al. 2008]. Compute times for the examples shown in the paper are measured in minutes (but not hours) on a single previous generation Nvidia Volta GPU. This restricts the use of our algorithm to high-quality offline applications where extensive non-real-time computation is the norm, primarily movies and videos. On the other hand, the computation can take advantage of support for multiple GPUs provided in deep learning frameworks, and intermediate results can be visualized. Our method can produce temporally consistent matte extractions from video by warm-starting the optimization from the previous frame (see accompanying video), however in our experience this requires that the trimaps have smooth motion from frame-to-frame. A topic for future work is to consider recurrent or other network architectures that might make the trimap choice more forgiving. This paper has focused on introducing the DIP matting algorithm. There was relatively little architecture and parameter exploration, and further improvements may be possible. ACKNOWLEDGMENTS G.G. Heitmann, Peter Hillman, and Kathleen Beeler gave helpful insights and feedback. REFERENCES Adobe 2018. How to use Select Subject in Photoshop for One-Click Selections. https: //www.photoshopessentials.com Yagiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik. 2018. Semantic soft segmentation. ACM Trans. Graph. 37, 4 (2018). Godzilla vs. Cat 2021. https://www.youtube.com/watch?v=nf7GsKFepDg. Mikhail Erofeev, Yury Gitman, Dmitriy Vatolin, Alexey Fedorov, and Jue Wang. 2015. Perceptually Motivated Benchmark for Video Matting. In British Machine Vision Conference. Yossi Gandelsman, Assaf Shocher, and Michal Irani. 2019. "Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. In Comp. Vision and Pattern Recognition Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. 2011. A global sampling method for alpha matting. Comp. Vision and Pattern Recognition Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided Image Filtering. In European Conf. Comp. Vision. G.G. Heitman. 2020. Communication from technical artist, Weta Digital Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Int. Conf. Learning Representations. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Trans. PAMI 30, 2 (2008). Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman. 2020. Real-Time High-Resolution Background Matting. arXiv (2020). Y. A Mishima. 1992. Software Chromakeyer Using Polyhedric Slice. In NICOGRAPH. Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A Perceptually Motivated Online Benchmark for Image Matting. In Comp. Vision and Pattern Recognition. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI (2015). Mike Seymour. 2020. Art of LED Wall Virtual Production. fxguide.com. Yanan Sun, Chi-Keung Tang, and Yu-Wing Tai. 2021. Semantic Image Matting. In Comp. Vision and Pattern Recognition. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep Image Prior. In Comp. Vision and Pattern Recognition. Jue Wang and Michael F. Cohen. 2007. Image and Video Matting: A Survey. Found. Trends Comput. Graph. Vis. 3, 2 (2007). Yong Xu, Baoling Liu, Yuhui Quan, and Hui Ji. 2022. Unsupervised Deep Background Matting Using Deep Matte Prior. IEEE Trans. Circuits Syst. Video Technol. (2022). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Limitations and Conclusion

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References ABSTRACT Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may appear only a few times in a single movie – if a method of creating ground truth for training exists, just use that method to produce the desired mattes. We introduce a training-free high quality neural matte extraction approach that specifically targets the assumptions of visual effects production. Our approach is based on the deep image prior, which optimizes a deep neural network to fit a single image, thereby providing a deep encoding of the particular image. We make use of the representations in the penultimate layer to interpolate coarse and incomplete "trimap" constraints. Videos processed with this approach are temporally consistent. The algorithm is both very simple and surprisingly effective. CCS CONCEPTS • Computing methodologies → Image processing; • Applied computing → Media arts. KEYWORDS Alpha matting, deep learning, visual effects. ACM Reference Format: Sharif Elcott, J.P. Lewis, Nori Kanazawa, and Christoph Bregler. 2022. TrainingFree Neural Matte Extraction for Visual Effects. In SIGGRAPH Asia 2022 Technical Communications (SA ’22 Technical Communications), December 6–9, 2022, Daegu, Republic of Korea. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3550340.3564230 1 INTRODUCTION Alpha matte extraction refers to the underconstrained inverse problem of finding the unknown translucency or coverage 𝛼 of a foreground object [Wang and Cohen 2007]. Matte extraction is widely used to provide alternate backgrounds for video meetings, as well as to produce visual effects (VFX) for movies, television, and social media. However, it is not always recognized in the research literature that these two applications have significantly different requirements. This paper introduces a neural matte extraction method specifically addressed to the assumptions and requirements of VFX. pecifically addressed to the assumptions and requirements of VFX. Matting for video calls requires real-time performance and assumes a single class of subject matter, front-facing humans. This is a "train once and use many times" situation for which it is possible and advantageous to obtain training data. Video call matting often assumes a fixed camera, and may require a "clean plate" image of the room without the participant. On the contrary, VFX production has the following assumptions and requirements: • Diverse and often rare ("one-off") subject matter. For example the youtube video [Godzilla vs. Cat 2021] (30M views) involves mattes of a cat, ships, and wreckage. A physical prop such as a crashed alien spaceship might only be used in a few seconds in a single movie, never to be seen again. Thus, gathering a ground-truth training dataset for deep learning is often pointless: if there is a method to generate the ground-truth mattes for training, just use this method to produce the desired mattes – no need to train a model! • Visual effects frequently involves moving cameras as well as extreme diversity of moving backgrounds. For example, an actor might be filmed as they run or travel in a vehicle any place on earth, or on set with extraterrestrial props in the near background. Thus, even for the major case of repeated subject matter (humans), gathering a representative training dataset is more challenging than in the case of video calls. • Real-time performance is not required. Instead, the on-set filming time is to be minimized, due to the combined cost of the actors (sometimes with 7-figure salaries) and movie crew. It is often cheaper to employ an artist to work days to "fix it in post" rather than spend a few minutes on-set (with actors and crew waiting) to address an issue. • Clean-plates are not a desirable approach for matting, and are often not possible. With moving cameras, a motion control rig is required to obtain a clean plate. The indoor use of motion control for clean plates is expensive due to the previous principle (minimizing on-set time). Moreover, this approach is generally not feasible outdoors, for reasons including background movement (e.g. plants moved by wind) and changing lighting (moving clouds blocking the sun). Our solution, a matte extraction approach employing the deep image prior [Ulyanov et al. 2018], addresses these requirements. It has the following characteristics and contributions: • It is a deep neural network matte extraction method targeted towards the requirements of VFX rather than video meetings. To our knowledge, this is the first deep neural matte extraction method that requires neither training data nor expensive on-set image capture to support the matting process (i.e. clean plates or greenscreens). • It relies exclusively on the given image or video along with coarse "trimaps" (Fig. 1) that are easily created during post production using existing semi-automatic tools. Professional software such as Nuke or After Effects is typically used, however a familiar example is the "Select Subject" tool in Photoshop [Adobe 2018], followed by dilation/erosion and automated with "Actions". • It does not require a clean plate, thus allowing extractions from outdoor footage. • It does not require greenscreens, yet can produce detailed high quality mattes even when the foreground and background subject have similar colors (green hair in Fig. 1). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References ABSTRACT Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may appear only a few times in a single movie – if a method of creating ground truth for training exists, just use that method to produce the desired mattes. We introduce a training-free high quality neural matte extraction approach that specifically targets the assumptions of visual effects production. Our approach is based on the deep image prior, which optimizes a deep neural network to fit a single image, thereby providing a deep encoding of the particular image. We make use of the representations in the penultimate layer to interpolate coarse and incomplete "trimap" constraints. Videos processed with this approach are temporally consistent. The algorithm is both very simple and surprisingly effective. CCS CONCEPTS • Computing methodologies → Image processing; • Applied computing → Media arts. KEYWORDS Alpha matting, deep learning, visual effects. ACM Reference Format: Sharif Elcott, J.P. Lewis, Nori Kanazawa, and Christoph Bregler. 2022. TrainingFree Neural Matte Extraction for Visual Effects. In SIGGRAPH Asia 2022 Technical Communications (SA ’22 Technical Communications), December 6–9, 2022, Daegu, Republic of Korea. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3550340.3564230 1 INTRODUCTION Alpha matte extraction refers to the underconstrained inverse problem of finding the unknown translucency or coverage 𝛼 of a foreground object [Wang and Cohen 2007]. Matte extraction is widely used to provide alternate backgrounds for video meetings, as well as to produce visual effects (VFX) for movies, television, and social media. However, it is not always recognized in the research literature that these two applications have significantly different requirements. This paper introduces a neural matte extraction method specifically addressed to the assumptions and requirements of VFX. pecifically addressed to the assumptions and requirements of VFX. Matting for video calls requires real-time performance and assumes a single class of subject matter, front-facing humans. This is a "train once and use many times" situation for which it is possible and advantageous to obtain training data. Video call matting often assumes a fixed camera, and may require a "clean plate" image of the room without the participant. On the contrary, VFX production has the following assumptions and requirements: • Diverse and often rare ("one-off") subject matter. For example the youtube video [Godzilla vs. Cat 2021] (30M views) involves mattes of a cat, ships, and wreckage. A physical prop such as a crashed alien spaceship might only be used in a few seconds in a single movie, never to be seen again. Thus, gathering a ground-truth training dataset for deep learning is often pointless: if there is a method to generate the ground-truth mattes for training, just use this method to produce the desired mattes – no need to train a model! • Visual effects frequently involves moving cameras as well as extreme diversity of moving backgrounds. For example, an actor might be filmed as they run or travel in a vehicle any place on earth, or on set with extraterrestrial props in the near background. Thus, even for the major case of repeated subject matter (humans), gathering a representative training dataset is more challenging than in the case of video calls. • Real-time performance is not required. Instead, the on-set filming time is to be minimized, due to the combined cost of the actors (sometimes with 7-figure salaries) and movie crew. It is often cheaper to employ an artist to work days to "fix it in post" rather than spend a few minutes on-set (with actors and crew waiting) to address an issue. • Clean-plates are not a desirable approach for matting, and are often not possible. With moving cameras, a motion control rig is required to obtain a clean plate. The indoor use of motion control for clean plates is expensive due to the previous principle (minimizing on-set time). Moreover, this approach is generally not feasible outdoors, for reasons including background movement (e.g. plants moved by wind) and changing lighting (moving clouds blocking the sun). Our solution, a matte extraction approach employing the deep image prior [Ulyanov et al. 2018], addresses these requirements. It has the following characteristics and contributions: • It is a deep neural network matte extraction method targeted towards the requirements of VFX rather than video meetings. To our knowledge, this is the first deep neural matte extraction method that requires neither training data nor expensive on-set image capture to support the matting process (i.e. clean plates or greenscreens). • It relies exclusively on the given image or video along with coarse "trimaps" (Fig. 1) that are easily created during post production using existing semi-automatic tools. Professional software such as Nuke or After Effects is typically used, however a familiar example is the "Select Subject" tool in Photoshop [Adobe 2018], followed by dilation/erosion and automated with "Actions". • It does not require a clean plate, thus allowing extractions from outdoor footage. • It does not require greenscreens, yet can produce detailed high quality mattes even when the foreground and background subject have similar colors (green hair in Fig. 1). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Abstract and Introduction

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References 3 METHOD Our method starts with a DIP network to reconstruct the target image. A second output head is added and tasked with inpainting the desired alpha in the trimap unknown region, constrained by the values in known regions. The main idea is that the first output head forces the network representation preceding it to "understand the structure of the image," while the second output head makes use of that information in estimating the matte. In addition to the alpha output we add two additional networks which simultaneously reconstruct the foreground and background. Similarly to the first output of the first network, these networks’ outputs are constrained to match the target image but, unlike the first network, they are constrained only in their respective regions of the trimap. Similarly to the alpha output, they extrapolate those constraints to inpaint the unconstrained region (see Fig. 2). The latter three outputs – 𝛼,ˆ 𝐹,ˆ 𝐵ˆ – are coupled via an additional constraint that they together satisfy (1) (see Detailed Loss below). We put 𝐹ˆ and 𝐵ˆ in separate networks (as in [Gandelsman et al. 2019]) rather than as additional outputs heads of the first network (as in [Sun et al. 2021]) in order to allow the inpainting of the foreground to be independent of the background. All three networks share the same generic U-net structure [Ulyanov et al. 2018], except for an additional output head in the first network. Our experiments use Adam [Kingma and Ba 2015] with a learning rate of 0.001. 3.1 Detailed Loss The first term of our loss function is the reconstruction loss between the first network output and the target image: The second loss term constrains 𝛼ˆ, where 𝑇 is the trimap and 𝐶 = 𝐹 ∪ 𝐵 is the constrained region of the trimap. The reconstruction losses for the foreground and background outputs are defined similarly to 𝐿𝐼 , but constrained only in their respective regions of the trimap: The three networks’ outputs are coupled via the alpha-compositing equation as follows: where 𝑈 = 𝐼 − 𝐶 is the unconstrained region of the trimap. Finally, we include an exclusion loss similar to [Sun et al. 2021] to prevent the structure of the foreground from leaking into the background and vice-versa: The total loss is the sum of the above six components: Unlike other DIP-based techniques, our algorithm does not require early stopping since the goal is to exactly fit both the image and the trimap constraints. 3.2 Temporal continuity In our experiments temporal continuity was obtained by warmstarting the optimization for frames other than the first with the final weight values of the previous frame, and stopping with a loss threshold rather than a fixed number of iterations (see video). This simple strategy produces reasonable results even on the relatively difficult case of hair. It also reduces the compute time by roughly an order of magnitude. 3.3 Why does it work? The deep image prior empirically demonstrates that standard convolutional networks can act as a good low-level prior for image reconstruction tasks, but why can it be adapted to invent plausible alpha mattes? Our intuition is as follows: The DIP provides a somewhat deep and hierarchical encoding of the particular image. The features in this encoding will primarily span valid image structures, with noise lying outside this space. In our work these features are re-combined to produce the matte. This works because the alpha matte contains structures that relate to those in the foreground image. In this respect our approach resembles a "deep" version of the guided image filter [He et al. 2010]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References 3 METHOD Our method starts with a DIP network to reconstruct the target image. A second output head is added and tasked with inpainting the desired alpha in the trimap unknown region, constrained by the values in known regions. The main idea is that the first output head forces the network representation preceding it to "understand the structure of the image," while the second output head makes use of that information in estimating the matte. In addition to the alpha output we add two additional networks which simultaneously reconstruct the foreground and background. Similarly to the first output of the first network, these networks’ outputs are constrained to match the target image but, unlike the first network, they are constrained only in their respective regions of the trimap. Similarly to the alpha output, they extrapolate those constraints to inpaint the unconstrained region (see Fig. 2). The latter three outputs – 𝛼,ˆ 𝐹,ˆ 𝐵ˆ – are coupled via an additional constraint that they together satisfy (1) (see Detailed Loss below). We put 𝐹ˆ and 𝐵ˆ in separate networks (as in [Gandelsman et al. 2019]) rather than as additional outputs heads of the first network (as in [Sun et al. 2021]) in order to allow the inpainting of the foreground to be independent of the background. All three networks share the same generic U-net structure [Ulyanov et al. 2018], except for an additional output head in the first network. Our experiments use Adam [Kingma and Ba 2015] with a learning rate of 0.001. 3.1 Detailed Loss The first term of our loss function is the reconstruction loss between the first network output and the target image: The second loss term constrains 𝛼ˆ, where 𝑇 is the trimap and 𝐶 = 𝐹 ∪ 𝐵 is the constrained region of the trimap. The reconstruction losses for the foreground and background outputs are defined similarly to 𝐿𝐼 , but constrained only in their respective regions of the trimap: The three networks’ outputs are coupled via the alpha-compositing equation as follows: where 𝑈 = 𝐼 − 𝐶 is the unconstrained region of the trimap. Finally, we include an exclusion loss similar to [Sun et al. 2021] to prevent the structure of the foreground from leaking into the background and vice-versa: The total loss is the sum of the above six components: Unlike other DIP-based techniques, our algorithm does not require early stopping since the goal is to exactly fit both the image and the trimap constraints. 3.2 Temporal continuity In our experiments temporal continuity was obtained by warmstarting the optimization for frames other than the first with the final weight values of the previous frame, and stopping with a loss threshold rather than a fixed number of iterations (see video). This simple strategy produces reasonable results even on the relatively difficult case of hair. It also reduces the compute time by roughly an order of magnitude. 3.3 Why does it work? The deep image prior empirically demonstrates that standard convolutional networks can act as a good low-level prior for image reconstruction tasks, but why can it be adapted to invent plausible alpha mattes? Our intuition is as follows: The DIP provides a somewhat deep and hierarchical encoding of the particular image. The features in this encoding will primarily span valid image structures, with noise lying outside this space. In our work these features are re-combined to produce the matte. This works because the alpha matte contains structures that relate to those in the foreground image. In this respect our approach resembles a "deep" version of the guided image filter [He et al. 2010]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Method

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References 4 RESULTS AND FAILURE CASES We use images, trimaps, and GT alpha from the dataset [Rhemann et al. 2009] to allow comparison to the ground truth. Fig. 1 shows a moderately challenging case with extensive hair, some of which is similar to the background color. Fig. 3 shows several additional examples. While the ground truth and estimated alpha maps appear identical at first glance, small differences can be seen upon enlargement. These may be due to imperfections in our algorithm, but also may reflect differences in color spaces between our algorithm and the ground-truth estimation process. Fig. 2 shows the extrapolated foreground 𝐹ˆ and background 𝐵ˆ for a particular image. Note that the inpainted colors need to be correct only in the unconstrained region surrounding the object border in order to allow compositing over a new background, as is the case in this figure. The unrealistic extrapolated 𝐹ˆ over the pure background regions is ignored. 4.1 Failure cases While objects with "holes" sometimes yield good mattes (e.g. the plant in Fig. 3), they are also a failure case (Fig. 4). In some cases the trimap can be adjusted to highlight the missing holes, though this would be laborious in cases such as the cup in Fig. 4. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References 4 RESULTS AND FAILURE CASES We use images, trimaps, and GT alpha from the dataset [Rhemann et al. 2009] to allow comparison to the ground truth. Fig. 1 shows a moderately challenging case with extensive hair, some of which is similar to the background color. Fig. 3 shows several additional examples. While the ground truth and estimated alpha maps appear identical at first glance, small differences can be seen upon enlargement. These may be due to imperfections in our algorithm, but also may reflect differences in color spaces between our algorithm and the ground-truth estimation process. Fig. 2 shows the extrapolated foreground 𝐹ˆ and background 𝐵ˆ for a particular image. Note that the inpainted colors need to be correct only in the unconstrained region surrounding the object border in order to allow compositing over a new background, as is the case in this figure. The unrealistic extrapolated 𝐹ˆ over the pure background regions is ignored. 4.1 Failure cases While objects with "holes" sometimes yield good mattes (e.g. the plant in Fig. 3), they are also a failure case (Fig. 4). In some cases the trimap can be adjusted to highlight the missing holes, though this would be laborious in cases such as the cup in Fig. 4. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Results and Failure Cases

Author:
(1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com);
(2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com);
(3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com);
(4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction
Background and Related Work
Method
Results and Failure Cases
Limitations and Conclusion, Acknowledgements and References 2 BACKGROUND AND RELATED WORK The alpha-compositing equation is where 𝑖 ∈ {𝑟, 𝑔, 𝑏}, 𝐼𝑖 are red, green, and blue values at a pixel of the given image (given), 𝐹𝑖 are the foreground object color (unknown), 𝐵𝑖 are the background object color (unknown), and 𝛼 ∈ [0, 1] is the (partially unknown) "alpha matte", representing the translucency or partial coverage of the foreground object at the pixel. Typical images contain large areas that are relatively easily estimated to be solid "foreground" (𝛼 = 1) or "background" (𝛼 = 0), as well as smaller regions where the alpha value is fractional. The matte extraction problem requires finding the these fractional alpha values. The problem is underconstrained because there are only three known values 𝐼𝑟, 𝐼𝑔, 𝐼𝑏 but seven unknown values. Hair is the prototypical challenge for matte extraction due to its irregular coverage and translucency, though fractional alpha also generally appears along all object edges due to the partial coverage of a pixel by the foreground object. It is also necessary to estimate the unknown foreground color in the fractional alpha regions, since without this the foreground cannot be composited over a different background. Many methods assume that approximate demarcations of the solid foreground and background are provided, e.g. in the form of artist-provided "trimaps" [Rhemann et al. 2009] or scribbles [Levin et al. 2008]. While the research literature generally considers the problem of matte extraction from arbitrary natural backgrounds, in industry practice matte extraction from a greenscreen background is far from a solved problem and often requires an artist-curated combination of techniques to obtain the required quality [Erofeev et al. 2015; Heitman 2020]. LED walls remove the need for greenscreens in some cases, and are well suited for fast-paced television production, however they also have drawbacks: computationally expensive physical or character simulations are not possible due to the need for real-time playback in a game engine, bright backgrounds may introduce challenging light spill on the foreground physical objects [Seymour 2020], their cost is prohibitive for smaller studios, and the effects must be finished at the time of the principle shoot thus excluding the creative control and iterative improvement available in traditional post production. Progress on matte extraction algorithms has been greatly facilitated by datasets that include ground truth (GT) mattes [Erofeev et al. 2015; Rhemann et al. 2009]. The GT mattes have been obtained by different means including chroma keying from a greenscreen and photographing a representative toy object in front of a background image on a monitor (Fig. 1 (a)). Failure to closely approximate these GT mattes indicates a poor algorithm, but an exact match may be unobtainable for several reasons: 1) the chroma key itself involves an imperfect algorithm [Erofeev et al. 2015], 2) light from multiple locations can physically scatter though (e.g.) translucent hair to arrive at a single pixel; this cannot be simulated in a purely 2D matte extraction process, 3) the image gamma or color space used in the benchmarks is not always evident and using a different gamma will produce small differences. The underconstrained nature of matte extraction has been approached with a variety of methods [Wang and Cohen 2007]. One classic approach estimates the unknown alpha at a pixel based on similarity to the distributions of known foreground and background colors in solid regions [He et al. 2011; Mishima 1992]. Another prominent principle finds the unknown alpha value by propagating from surrounding known values [Aksoy et al. 2018; Levin et al. 2008]. These approaches often require solving a system involving a generalized Laplacian formed from the pixel affinities in the unknown region, which prevents real-time or interactive use. Deep learning (DL) approaches are used in recent research e.g. [Lin et al. 2020; Sun et al. 2021]. Much of this work focuses on providing alternate backgrounds for video meetings, and training databases consist of mostly forward-facing human heads. A state of the art method [Lin et al. 2020] demonstrates high-quality matte extraction on HD-resolution images at 60fps. Many DL methods adopt a combination of techniques, e.g. separate networks for overall segmentation and for fractional alpha regions, or other hybrid approaches. 2.1 Deep Image Prior The Deep Image Prior (DIP) [Ulyanov et al. 2018] demonstrates that the architecture of an untrained convolutional network provides a surprisingly good prior for tasks such as image inpainting and denoising. The key observation is that while a powerful DNN can fit arbitrary image structures such as noise, it is "easier" to fit natural image structures, as reflected in faster loss curve decay. Most experiments [Ulyanov et al. 2018] optimize the weights of a U-net [Ronneberger et al. 2015] that maps fixed random noise to a single output image. Uncorrupted features of the image are fit earlier in the optimization process, so early stopping results in a corrected (denoised or inpainted) image. The DIP has been successfully applied to other problems in image processing. Unsupervised coarse binary segmentation is demonstrated in [Gandelsman et al. 2019], based on a principle that it is easier (in terms of loss decay) to fit each component of a mixture of images with a separate DIP rather than using a single model. Concurrent with our work, [Xu et al. 2022] formulate a DIP approach to background matting. This problem scenario differs from our work in that it requires a clean plate, and hence is unsuitable for many VFX applications such as those that are filmed outdoors with a moving camera. Our work also uses the DIP. We focus on high-quality estimation of fractional alpha mattes and, in contrast to previous work, undertake the challenging case where no clean plate is available, instead relying only on trimaps that can be easily created during post-production. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Author: Author: (1) Sharif Elcott, equal contribution of Google Japan (Email: selcott@google.com); (2) J.P. Lewis, equal contribution of Google Research USA (Email: jplewis@google.com); (3) Nori Kanazawa, Google Research USA (Email: kanazawa@google.com); (4) Christoph Bregler, Google Research USA (Email: bregler@google.com). Table of Links Abstract and Introduction Background and Related Work Method Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Abstract and Introduction Abstract and Introduction Background and Related Work Background and Related Work Method Method Results and Failure Cases Results and Failure Cases Limitations and Conclusion, Acknowledgements and References Limitations and Conclusion, Acknowledgements and References 2 BACKGROUND AND RELATED WORK The alpha-compositing equation is where 𝑖 ∈ {𝑟, 𝑔, 𝑏}, 𝐼𝑖 are red, green, and blue values at a pixel of the given image (given), 𝐹𝑖 are the foreground object color (unknown), 𝐵𝑖 are the background object color (unknown), and 𝛼 ∈ [0, 1] is the (partially unknown) "alpha matte", representing the translucency or partial coverage of the foreground object at the pixel. Typical images contain large areas that are relatively easily estimated to be solid "foreground" (𝛼 = 1) or "background" (𝛼 = 0), as well as smaller regions where the alpha value is fractional. The matte extraction problem requires finding the these fractional alpha values. The problem is underconstrained because there are only three known values 𝐼𝑟, 𝐼𝑔, 𝐼𝑏 but seven unknown values. Hair is the prototypical challenge for matte extraction due to its irregular coverage and translucency, though fractional alpha also generally appears along all object edges due to the partial coverage of a pixel by the foreground object. It is also necessary to estimate the unknown foreground color in the fractional alpha regions, since without this the foreground cannot be composited over a different background. Many methods assume that approximate demarcations of the solid foreground and background are provided, e.g. in the form of artist-provided "trimaps" [Rhemann et al. 2009] or scribbles [Levin et al. 2008]. While the research literature generally considers the problem of matte extraction from arbitrary natural backgrounds, in industry practice matte extraction from a greenscreen background is far from a solved problem and often requires an artist-curated combination of techniques to obtain the required quality [Erofeev et al. 2015; Heitman 2020]. LED walls remove the need for greenscreens in some cases, and are well suited for fast-paced television production, however they also have drawbacks: computationally expensive physical or character simulations are not possible due to the need for real-time playback in a game engine, bright backgrounds may introduce challenging light spill on the foreground physical objects [Seymour 2020], their cost is prohibitive for smaller studios, and the effects must be finished at the time of the principle shoot thus excluding the creative control and iterative improvement available in traditional post production. Progress on matte extraction algorithms has been greatly facilitated by datasets that include ground truth (GT) mattes [Erofeev et al. 2015; Rhemann et al. 2009]. The GT mattes have been obtained by different means including chroma keying from a greenscreen and photographing a representative toy object in front of a background image on a monitor (Fig. 1 (a)). Failure to closely approximate these GT mattes indicates a poor algorithm, but an exact match may be unobtainable for several reasons: 1) the chroma key itself involves an imperfect algorithm [Erofeev et al. 2015], 2) light from multiple locations can physically scatter though (e.g.) translucent hair to arrive at a single pixel; this cannot be simulated in a purely 2D matte extraction process, 3) the image gamma or color space used in the benchmarks is not always evident and using a different gamma will produce small differences. The underconstrained nature of matte extraction has been approached with a variety of methods [Wang and Cohen 2007]. One classic approach estimates the unknown alpha at a pixel based on similarity to the distributions of known foreground and background colors in solid regions [He et al. 2011; Mishima 1992]. Another prominent principle finds the unknown alpha value by propagating from surrounding known values [Aksoy et al. 2018; Levin et al. 2008]. These approaches often require solving a system involving a generalized Laplacian formed from the pixel affinities in the unknown region, which prevents real-time or interactive use. Deep learning (DL) approaches are used in recent research e.g. [Lin et al. 2020; Sun et al. 2021]. Much of this work focuses on providing alternate backgrounds for video meetings, and training databases consist of mostly forward-facing human heads. A state of the art method [Lin et al. 2020] demonstrates high-quality matte extraction on HD-resolution images at 60fps. Many DL methods adopt a combination of techniques, e.g. separate networks for overall segmentation and for fractional alpha regions, or other hybrid approaches. 2.1 Deep Image Prior The Deep Image Prior (DIP) [Ulyanov et al. 2018] demonstrates that the architecture of an untrained convolutional network provides a surprisingly good prior for tasks such as image inpainting and denoising. The key observation is that while a powerful DNN can fit arbitrary image structures such as noise, it is "easier" to fit natural image structures, as reflected in faster loss curve decay. Most experiments [Ulyanov et al. 2018] optimize the weights of a U-net [Ronneberger et al. 2015] that maps fixed random noise to a single output image. Uncorrupted features of the image are fit earlier in the optimization process, so early stopping results in a corrected (denoised or inpainted) image. The DIP has been successfully applied to other problems in image processing. Unsupervised coarse binary segmentation is demonstrated in [Gandelsman et al. 2019], based on a principle that it is easier (in terms of loss decay) to fit each component of a mixture of images with a separate DIP rather than using a single model. Concurrent with our work, [Xu et al. 2022] formulate a DIP approach to background matting. This problem scenario differs from our work in that it requires a clean plate, and hence is unsuitable for many VFX applications such as those that are filmed outdoors with a moving camera. Our work also uses the DIP. We focus on high-quality estimation of fractional alpha mattes and, in contrast to previous work, undertake the challenging case where no clean plate is available, instead relying only on trimaps that can be easily created during post-production. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Training-Free Neural Matte Extraction for Visual Effects: Background and Related Work

Authors:
(1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com);
(2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com);
(3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Methods Results Discussion Conclusions and References 4 Discussion Deep diffusion surrogates can aid in obtaining the steady-state solution for multiscale modeling. In this paper we looked at 20 sources randomly placed in 2D lattice. It’s still far less complex than the real problem, e.g., simulation of a vascular system. Nevertheless, this is a step forward in that direction. We have shown that increasing the number of sources already pose a number of challenges in different aspects. We showed how the network architecture, the training set structure and size, the loss function, the hyperparameters for training algorithms and defining metrics to evaluate the task-specific performance of the trained network (which may differ from the loss function used in training) are all aspects that affect the final product. In the case of the NN architecture we argued that the encoder-decoder CNN architecture performs well not due to data compression, as believed by many, rather to data transformation akin to Fourier transform. However a more rigorous proof is, both required and desired. That’s not to say that other architecture should not be able to perform well in predicting the steady-state solution. In fact, in prior work [17] we combined an ED-CNN and a CNN for a similar task and found that the CNN improved the performance by reinforcing the sources in the prediction. The results shown in this paper highlight that the largest absolute error occurs at and near the sources. Therefore, we reckon the architecture such as a UNet [27] as a good candidate for this task and we leave it for future work. We considered different loss functions and compared the different performances due to the different loss functions. The wide numeric range for input and output of neural networks makes the analysis very sensitive to choices in the loss function. Our results suggest that the loss function can have a significant effect on the model’s performance. However, we showed that the data set size has a greater effect in the model’s performance and, furthermore, the performance associated to the loss function depends on the data set size. We also showed how a large enough data set reduces the performance fluctuations in the test set. In a real problem the landscape of possibilities is unknown, which implies that the model fluctuations are, at least, unknown. This hints the difficulty in bounding the performance error for any unseen configuration. The naive solution is to increase the training data set. Increasing the training set arbitrarily will lead to an increase in training time. So care must be taken in such approach. One needs to take the best of both worlds, i.e., models trained on sufficiently large data sets to reduce fluctuations but small enough in order be able to train in a fashionable time. A better curated data set where configuration redundancy is kept at a minimum can lead to better performance. Another approach that seem promising is active learning [28] whereby data is ranked by the magnitude of the performance error and data with the largest error is then fed into the training. Defining the right metrics in deep learning is highly challenging. Partly because quantifying the degree of success can be difficult, whereas it is fairly easy agreeing in the ideal success. In other words, quantifying good enough is not straightforward and requires bench-marking different approaches for comparison. A thorough discussion on benchmark suite can be found in [29]. This paper is available on arxiv under CC 4.0 license. Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Authors: Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Discussion Discussion Conclusions and References Conclusions and References 4 Discussion Deep diffusion surrogates can aid in obtaining the steady-state solution for multiscale modeling. In this paper we looked at 20 sources randomly placed in 2D lattice. It’s still far less complex than the real problem, e.g., simulation of a vascular system. Nevertheless, this is a step forward in that direction. We have shown that increasing the number of sources already pose a number of challenges in different aspects. We showed how the network architecture, the training set structure and size, the loss function, the hyperparameters for training algorithms and defining metrics to evaluate the task-specific performance of the trained network (which may differ from the loss function used in training) are all aspects that affect the final product. In the case of the NN architecture we argued that the encoder-decoder CNN architecture performs well not due to data compression, as believed by many, rather to data transformation akin to Fourier transform. However a more rigorous proof is, both required and desired. That’s not to say that other architecture should not be able to perform well in predicting the steady-state solution. In fact, in prior work [17] we combined an ED-CNN and a CNN for a similar task and found that the CNN improved the performance by reinforcing the sources in the prediction. The results shown in this paper highlight that the largest absolute error occurs at and near the sources. Therefore, we reckon the architecture such as a UNet [27] as a good candidate for this task and we leave it for future work. We considered different loss functions and compared the different performances due to the different loss functions. The wide numeric range for input and output of neural networks makes the analysis very sensitive to choices in the loss function. Our results suggest that the loss function can have a significant effect on the model’s performance. However, we showed that the data set size has a greater effect in the model’s performance and, furthermore, the performance associated to the loss function depends on the data set size. We also showed how a large enough data set reduces the performance fluctuations in the test set. In a real problem the landscape of possibilities is unknown, which implies that the model fluctuations are, at least, unknown. This hints the difficulty in bounding the performance error for any unseen configuration. The naive solution is to increase the training data set. Increasing the training set arbitrarily will lead to an increase in training time. So care must be taken in such approach. One needs to take the best of both worlds, i.e., models trained on sufficiently large data sets to reduce fluctuations but small enough in order be able to train in a fashionable time. A better curated data set where configuration redundancy is kept at a minimum can lead to better performance. Another approach that seem promising is active learning [28] whereby data is ranked by the magnitude of the performance error and data with the largest error is then fed into the training. Defining the right metrics in deep learning is highly challenging. Partly because quantifying the degree of success can be difficult, whereas it is fairly easy agreeing in the ideal success. In other words, quantifying good enough is not straightforward and requires bench-marking different approaches for comparison. A thorough discussion on benchmark suite can be found in [29]. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Optimizing Data Set Size and Loss Functions for Enhanced Neural Network Performance

Understanding Factors Affecting Neural Network Performance in Diffusion Prediction

Authors:
(1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com);
(2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com);
(3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Methods Results Discussion Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Authors: Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Discussion Discussion Conclusions and References Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Architecting Neural Networks for Diffusion Prediction: A Study on Encoder-Decoder CNNs

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Results

ডিপ নিউরাল নেটওয়ার্ক লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করতে: ফলাফল

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Ergebnisse

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: resultados

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : résultats

लिम्फोमा घावों का पता लगाने और मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: परिणाम

リンパ腫病変を検出し定量化するディープニューラルネットワーク：結果

림프종 병변을 탐지하고 정량화하기 위한 심층 신경망: 결과

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Resultados

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: результаты

Lenfoma Lezyonlarını Tespit Etmek ve Ölçmek İçin Derin Sinir Ağları: Sonuçlar

Mạng lưới thần kinh sâu để phát hiện và định lượng tổn thương ung thư hạch: Kết quả

深度神经网络检测和量化淋巴瘤病变：结果

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Materials and Methods

ডিপ নিউরাল নেটওয়ার্ক লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করার জন্য: উপাদান এবং পদ্ধতি

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Materialien und Methoden

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: materiales y métodos

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : matériels et méthodes

लिम्फोमा घावों का पता लगाने और मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: सामग्री और विधियाँ

リンパ腫病変を検出し定量化するディープニューラルネットワーク：材料と方法

림프종 병변을 검출하고 정량화하기 위한 심층 신경망: 재료 및 방법

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Materiais e Métodos

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: материалы и методы

Lenfoma Lezyonlarını Tespit Etmek ve Ölçmek İçin Derin Sinir Ağları: Malzemeler ve Yöntemler

Mạng lưới thần kinh sâu để phát hiện và định lượng tổn thương ung thư hạch: Vật liệu và phương pháp

深度神经网络检测和量化淋巴瘤病变：材料和方法

Authors:
(1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com);
(2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA;
(3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada;
(4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea;
(5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada;
(6) Don Wilson, BC Cancer, Vancouver, BC, Canada;
(7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada;
(8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada;
(9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada;
(10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA;
(11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA;
(12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA;
(13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada;
(14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro
Related Work
Materials and Methods
Results
Discussion
Conclusion and References VI. CONCLUSION In this study, we assessed various neural network architectures for automating lymphoma lesion segmentation in PET/CT images across multiple datasets. We examined the reproducibility of lesion measures, revealing differences among networks, highlighting their suitability for specific clinical uses. Additionally, we introduced three lesion detection criteria to assess network performance at a per-lesion level, emphasizing their clinical relevance. Lastly, we discussed challenges related to ground truth consistency and stressed the importance of having well-defined protocol for segmentation. This work provides valuable insights into deep learning’s potentials and limitations in lymphoma lesion segmentation and emphasizes the need for standardized annotation practices to enhance research validity and clinical applications. REFERENCES [1] S. F. Barrington et al. “FDG PET for therapy monitoring in Hodgkin and non-Hodgkin lymphomas”. In: European Journal of Nuclear Medicine and Molecular Imaging 44.1 (Aug. 2017), pp. 97–110. ISSN: 1619- 7089. [2] K. Okuyucu et al. “Prognosis estimation under the light of metabolic tumor parameters on initial FDG-PET/CT in patients with primary extranodal lymphoma”. en. In: Radiol. Oncol. 50.4 (Dec. 2016), pp. 360–369. [3] N. Wu et al. “Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening”. In: IEEE Transactions on Medical Imaging 39.4 (2020), pp. 1184–1194. [4] C. Yuan et al. “Diffuse large B-cell lymphoma segmentation in PET-CT images via hybrid learning for feature fusion”. In: Medical Physics 48.7 (2021), pp. 3665– 3678. [5] H. Hu et al. “Lymphoma Segmentation in PET Images Based on Multi-view and Conv3D Fusion Strategy”. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). 2020, pp. 1197–1200. [6] H. Li et al. “DenseX-Net: An End-to-End Model for Lymphoma Segmentation in Whole-Body PET/CT Images”. In: IEEE Access 8 (2020), pp. 8004–8018. [7] L. Liu et al. “Improved Multi-modal Patch Based Lymphoma Segmentation with Negative Sample Augmentation and Label Guidance on PET/CT Scans”. In: Multiscale Multimodal Medical Imaging. Ed. by X. Li et al. Cham: Springer Nature Switzerland, 2022, pp. 121–129. ISBN: 978-3-031-18814-5. [8] C. S. Constantino et al. “Evaluation of Semiautomatic and Deep Learning–Based Fully Automatic Segmentation Methods on [18F]FDG PET/CT Images from Patients with Lymphoma: Influence on Tumor Characterization”. In: Journal of Digital Imaging 36.4 (Aug. 2023), pp. 1864–1876. ISSN: 1618-727X. [9] A. J. Weisman et al. “Comparison of 11 automated PET segmentation methods in lymphoma”. English. In: Physics in medicine & biology 65.23 (2020), pp. 235019–235019. [10] A. J. Weisman et al. “Convolutional Neural Networks for Automated PET/CT Detection of Diseased Lymph Node Burden in Patients with Lymphoma”. In: Radiology: Artificial Intelligence 2.5 (2020), e200016. [11] C. Jiang et al. “Deep learning–based tumour segmentation and total metabolic tumour volume prediction in the prognosis of diffuse large B-cell lymphoma patients in 3D FDG-PET images”. In: European Radiology 32.7 (July 2022), pp. 4801–4812. ISSN: 1432-1084. [12] P. Blanc-Durand et al. “Fully automatic segmentation of diffuse large B cell lymphoma lesions on 3D FDGPET/CT for total metabolic tumour volume prediction using a convolutional neural network.” In: European Journal of Nuclear Medicine and Molecular Imaging 48.5 (May 2021), pp. 1362–1370. ISSN: 1619-7089. [13] S. Ahamed et al. “A U-Net Convolutional Neural Network with Multiclass Dice Loss for Automated Segmentation of Tumors and Lymph Nodes from Head and Neck Cancer PET/CT Images”. In: Head and Neck Tumor Segmentation and Outcome Prediction. Ed. by V. Andrearczyk et al. Cham: Springer Nature Switzerland, 2023, pp. 94–106. ISBN: 978-3-031-27420-6. [14] S. Gatidis et al. “A whole-body FDG-PET/CT Dataset with manually annotated Tumor Lesions”. In: Scientific Data 9.1 (Oct. 2022), p. 601. ISSN: 2052-4463. [15] M. Pop et al. “Left-Ventricle Quantification Using Residual U-Net”. English. In: vol. 11395. Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges. Switzerland: Springer International Publishing AG, 2019, pp. 371–380. ISBN: 0302-9743. [16] A. Myronenko. “3D MRI Brain Tumor Segmentation Using Autoencoder Regularization”. English. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Cham: Springer International Publishing, 2019, pp. 311–320. ISBN: 0302-9743. [17] F. Isensee et al. “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”. In: Nature Methods 18.2 (Dec. 2020), pp. 203–211. [18] A. Hatamizadeh et al. “Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images”. English. In: (2022). [19] M. J. Cardoso et al. MONAI: An open-source framework for deep learning in healthcare. 2022. arXiv: 2211. 02701 [cs.LG]. [20] S. Ahamed et al. “Towards enhanced lesion segmentation using a 3D neural network trained on multiresolution cropped patches of lymphoma PET images”. In: Journal of Nuclear Medicine 64.supplement 1 (2023), P1360–P1360. ISSN: 0161-5505. [21] J. L. Fleiss. “Measuring nominal scale agreement among many raters”. English. In: Psychological bulletin 76.5 (1971), pp. 378–382. [22] A. K. Jha et al. “Nuclear Medicine and artificial intelligence: Best practices for evaluation (the RELAINCE guidelines)”. en. In: J. Nucl. Med. 63.9 (Sept. 2022), pp. 1288–1299. [23] N. Hasani et al. “Artificial Intelligence in Lymphoma PET Imaging: A Scoping Review (Current Trends and Future Directions)”. In: PET clinics 17.1 (Jan. 1), pp. 145–174. ISSN: 1556-8598. [24] S. K. Warfield et al. “Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation”. en. In: IEEE Trans. Med. Imaging 23.7 (July 2004), pp. 903–921. This paper is available on arxiv under CC 4.0 license. Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Authors: Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro Related Work Materials and Methods Results Discussion Conclusion and References Abstract and Intro Abstract and Intro Related Work Related Work Materials and Methods Materials and Methods Results Results Discussion Discussion Conclusion and References Conclusion and References VI. CONCLUSION In this study, we assessed various neural network architectures for automating lymphoma lesion segmentation in PET/CT images across multiple datasets. We examined the reproducibility of lesion measures, revealing differences among networks, highlighting their suitability for specific clinical uses. Additionally, we introduced three lesion detection criteria to assess network performance at a per-lesion level, emphasizing their clinical relevance. Lastly, we discussed challenges related to ground truth consistency and stressed the importance of having well-defined protocol for segmentation. This work provides valuable insights into deep learning’s potentials and limitations in lymphoma lesion segmentation and emphasizes the need for standardized annotation practices to enhance research validity and clinical applications. REFERENCES [1] S. F. Barrington et al. “FDG PET for therapy monitoring in Hodgkin and non-Hodgkin lymphomas”. In: European Journal of Nuclear Medicine and Molecular Imaging 44.1 (Aug. 2017), pp. 97–110. ISSN: 1619- 7089. [2] K. Okuyucu et al. “Prognosis estimation under the light of metabolic tumor parameters on initial FDG-PET/CT in patients with primary extranodal lymphoma”. en. In: Radiol. Oncol. 50.4 (Dec. 2016), pp. 360–369. [3] N. Wu et al. “Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening”. In: IEEE Transactions on Medical Imaging 39.4 (2020), pp. 1184–1194. [4] C. Yuan et al. “Diffuse large B-cell lymphoma segmentation in PET-CT images via hybrid learning for feature fusion”. In: Medical Physics 48.7 (2021), pp. 3665– 3678. [5] H. Hu et al. “Lymphoma Segmentation in PET Images Based on Multi-view and Conv3D Fusion Strategy”. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). 2020, pp. 1197–1200. [6] H. Li et al. “DenseX-Net: An End-to-End Model for Lymphoma Segmentation in Whole-Body PET/CT Images”. In: IEEE Access 8 (2020), pp. 8004–8018. [7] L. Liu et al. “Improved Multi-modal Patch Based Lymphoma Segmentation with Negative Sample Augmentation and Label Guidance on PET/CT Scans”. In: Multiscale Multimodal Medical Imaging. Ed. by X. Li et al. Cham: Springer Nature Switzerland, 2022, pp. 121–129. ISBN: 978-3-031-18814-5. [8] C. S. Constantino et al. “Evaluation of Semiautomatic and Deep Learning–Based Fully Automatic Segmentation Methods on [18F]FDG PET/CT Images from Patients with Lymphoma: Influence on Tumor Characterization”. In: Journal of Digital Imaging 36.4 (Aug. 2023), pp. 1864–1876. ISSN: 1618-727X. [9] A. J. Weisman et al. “Comparison of 11 automated PET segmentation methods in lymphoma”. English. In: Physics in medicine & biology 65.23 (2020), pp. 235019–235019. [10] A. J. Weisman et al. “Convolutional Neural Networks for Automated PET/CT Detection of Diseased Lymph Node Burden in Patients with Lymphoma”. In: Radiology: Artificial Intelligence 2.5 (2020), e200016. [11] C. Jiang et al. “Deep learning–based tumour segmentation and total metabolic tumour volume prediction in the prognosis of diffuse large B-cell lymphoma patients in 3D FDG-PET images”. In: European Radiology 32.7 (July 2022), pp. 4801–4812. ISSN: 1432-1084. [12] P. Blanc-Durand et al. “Fully automatic segmentation of diffuse large B cell lymphoma lesions on 3D FDGPET/CT for total metabolic tumour volume prediction using a convolutional neural network.” In: European Journal of Nuclear Medicine and Molecular Imaging 48.5 (May 2021), pp. 1362–1370. ISSN: 1619-7089. [13] S. Ahamed et al. “A U-Net Convolutional Neural Network with Multiclass Dice Loss for Automated Segmentation of Tumors and Lymph Nodes from Head and Neck Cancer PET/CT Images”. In: Head and Neck Tumor Segmentation and Outcome Prediction. Ed. by V. Andrearczyk et al. Cham: Springer Nature Switzerland, 2023, pp. 94–106. ISBN: 978-3-031-27420-6. [14] S. Gatidis et al. “A whole-body FDG-PET/CT Dataset with manually annotated Tumor Lesions”. In: Scientific Data 9.1 (Oct. 2022), p. 601. ISSN: 2052-4463. [15] M. Pop et al. “Left-Ventricle Quantification Using Residual U-Net”. English. In: vol. 11395. Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges. Switzerland: Springer International Publishing AG, 2019, pp. 371–380. ISBN: 0302-9743. [16] A. Myronenko. “3D MRI Brain Tumor Segmentation Using Autoencoder Regularization”. English. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Cham: Springer International Publishing, 2019, pp. 311–320. ISBN: 0302-9743. [17] F. Isensee et al. “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”. In: Nature Methods 18.2 (Dec. 2020), pp. 203–211. [18] A. Hatamizadeh et al. “Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images”. English. In: (2022). [19] M. J. Cardoso et al. MONAI: An open-source framework for deep learning in healthcare. 2022. arXiv: 2211. 02701 [cs.LG]. [20] S. Ahamed et al. “Towards enhanced lesion segmentation using a 3D neural network trained on multiresolution cropped patches of lymphoma PET images”. In: Journal of Nuclear Medicine 64.supplement 1 (2023), P1360–P1360. ISSN: 0161-5505. [21] J. L. Fleiss. “Measuring nominal scale agreement among many raters”. English. In: Psychological bulletin 76.5 (1971), pp. 378–382. [22] A. K. Jha et al. “Nuclear Medicine and artificial intelligence: Best practices for evaluation (the RELAINCE guidelines)”. en. In: J. Nucl. Med. 63.9 (Sept. 2022), pp. 1288–1299. [23] N. Hasani et al. “Artificial Intelligence in Lymphoma PET Imaging: A Scoping Review (Current Trends and Future Directions)”. In: PET clinics 17.1 (Jan. 1), pp. 145–174. ISSN: 1556-8598. [24] S. K. Warfield et al. “Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation”. en. In: IEEE Trans. Med. Imaging 23.7 (July 2004), pp. 903–921. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv available on arxiv

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Conclusion and References

ডিপ নিউরাল নেটওয়ার্ক লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করার জন্য: উপসংহার এবং রেফারেন্স

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Schlussfolgerung und Referenzen

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: conclusiones y referencias

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : conclusion et références

लिम्फोमा घावों का पता लगाने और मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: निष्कर्ष और संदर्भ

リンパ腫病変の検出と定量化のためのディープニューラルネットワーク：結論と参考文献

림프종 병변을 검출하고 정량화하기 위한 심층 신경망: 결론 및 참고 문헌

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Conclusão e Referências

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: заключение и ссылки

Lenfoma Lezyonlarını Tespit Etmek ve Ölçmek İçin Derin Sinir Ağları: Sonuç ve Referanslar

Mạng lưới thần kinh sâu để phát hiện và định lượng tổn thương ung thư hạch: Kết luận và tài liệu tham khảo

深度神经网络检测和量化淋巴瘤病变：结论和参考文献

Authors:
(1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com);
(2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA;
(3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada;
(4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea;
(5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada;
(6) Don Wilson, BC Cancer, Vancouver, BC, Canada;
(7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada;
(8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada;
(9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada;
(10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA;
(11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA;
(12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA;
(13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada;
(14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro
Related Work
Materials and Methods
Results
Discussion
Conclusion and References II. RELATED WORK Numerous works have explored the application of deep learning methods for segmenting lymphoma in PET/CT images. Yuan et al. [4] developed a feature fusion technique to utilize the complementary information from multi-modality data. Hu et al. [5] proposed fusing a combination of 3D ResUNet trained on volumetric data and three 2D ResUNet trained on 2D slices from three orthogonal directions for enhancing segmentation performance. Li et al. [6] proposed DenseX-Net trained in an end-to-end fashion integrating supervised and unsupervised methods for lymphoma detection and segmentation. Liu et al. [7] introduced techniques such as patch-based negative sample augmentation and label guidance for training a 3D Residual-UNet for lymphoma segmentation. A major limitation of all these works was that they were developed on relatively smaller-sized datasets (less than 100 images). Moreover, most of these methods did not compare the performance of their proposed methods with other baselines or with the performance of physicians. Constantino et al. [8] compared the performances of 7 semi-automated and 2 deep learning segmentation methods, while Weisman et al. [9] compared 11 automated segmentation techniques, although both these studies were performed on smaller datasets of sizes 65 and 90 respectively. Weisman et al. [10] compared the segmentation performances of automated 3D Deep Medic method with that of physician although even this study included just 90 lymphoma cases. Except for [10], none of these studies reported model generalization on out-ofdistribution dataset (such as on data collected from different centers), limiting their robustness quantification and external validity. Jiang et al. [11] used a relatively larger dataset as compared to the above studies with 297 images to train a 3D UNet. They even performed out-of-distribution testing on 117 images collected from a different center. To the best of our knowledge, the largest lymphoma PET/CT dataset for deep learning-based lesion segmentation ever reported is the work by Blanc-Durand et al. [12] who used 639 images for model development and 94 for external testing; however, this study only used standard segmentation evaluation metrics and assessed their model’s ability for predicting accurate TMTV. Both the studies [11] and [12] are limited by the fact that their datasets exclusively consisted of patients diagnosed with diffuse large B-cell lymphoma (DLBCL), representing only a single subtype of lymphoma. Most of the existing studies on deep learning-based lymphoma segmentation report their performances on generic segmentation metrics such as Dice similarity coefficient (DSC), intersection-over-union (IoU), sensitivity, etc. In the presence of large segmented lesions, very small missed lesions or small false positives do not contribute much to the DSC value. Hence, there is a need to report the volumes of false positives and false negatives. It will also be beneficial to evaluate the detection performances on a per-lesion basis (number of connected components detected vs missed), since automated detection of even a few voxels of all lesions can help physicians quickly locate the regions of interest, even if the DSC is low. Moreover, the difficulty of the segmentation/detection task is often not assessed via inter- or intra-observer agreement analysis. Our study aims to address these limitations. We trained and validated four deep neural networks on lymphoma PET/CT datasets from three cohorts, encompassing two distinct subtypes of lymphoma: DLBCL and primary mediastinal large B-cell lymphoma (PMBCL). (i) We performed both in (images coming from same cohorts as the training/validation set) and out-of-distribution or external (images from a fourth cohort not used for training/validation) testing to evaluate the robustness of our models. (ii) We reported the performance using DSC, volumes of false positives and negatives, and evaluated the performance dependence on six different types of lesion measures. (iii) We also evaluated the ability of our networks to reproduce these ground truth lesion measures and computed networks’ error in predicting them. (iv) We proposed three types of detection criteria for our use-case and evaluate model’s performance on these metrics. (v) Finally, we evaluated the intra- and inter-observer agreement to give a measure of the difficulty of lesion segmentation task on our datasets. This paper is available on arxiv under CC 4.0 license. Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Authors: Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro Related Work Materials and Methods Results Discussion Conclusion and References Abstract and Intro Abstract and Intro Related Work Related Work Materials and Methods Materials and Methods Results Results Discussion Discussion Conclusion and References Conclusion and References II. RELATED WORK Numerous works have explored the application of deep learning methods for segmenting lymphoma in PET/CT images. Yuan et al. [4] developed a feature fusion technique to utilize the complementary information from multi-modality data. Hu et al. [5] proposed fusing a combination of 3D ResUNet trained on volumetric data and three 2D ResUNet trained on 2D slices from three orthogonal directions for enhancing segmentation performance. Li et al. [6] proposed DenseX-Net trained in an end-to-end fashion integrating supervised and unsupervised methods for lymphoma detection and segmentation. Liu et al. [7] introduced techniques such as patch-based negative sample augmentation and label guidance for training a 3D Residual-UNet for lymphoma segmentation. A major limitation of all these works was that they were developed on relatively smaller-sized datasets (less than 100 images). Moreover, most of these methods did not compare the performance of their proposed methods with other baselines or with the performance of physicians. Constantino et al. [8] compared the performances of 7 semi-automated and 2 deep learning segmentation methods, while Weisman et al. [9] compared 11 automated segmentation techniques, although both these studies were performed on smaller datasets of sizes 65 and 90 respectively. Weisman et al. [10] compared the segmentation performances of automated 3D Deep Medic method with that of physician although even this study included just 90 lymphoma cases. Except for [10], none of these studies reported model generalization on out-ofdistribution dataset (such as on data collected from different centers), limiting their robustness quantification and external validity. Jiang et al. [11] used a relatively larger dataset as compared to the above studies with 297 images to train a 3D UNet. They even performed out-of-distribution testing on 117 images collected from a different center. To the best of our knowledge, the largest lymphoma PET/CT dataset for deep learning-based lesion segmentation ever reported is the work by Blanc-Durand et al. [12] who used 639 images for model development and 94 for external testing; however, this study only used standard segmentation evaluation metrics and assessed their model’s ability for predicting accurate TMTV. Both the studies [11] and [12] are limited by the fact that their datasets exclusively consisted of patients diagnosed with diffuse large B-cell lymphoma (DLBCL), representing only a single subtype of lymphoma. Most of the existing studies on deep learning-based lymphoma segmentation report their performances on generic segmentation metrics such as Dice similarity coefficient (DSC), intersection-over-union (IoU), sensitivity, etc. In the presence of large segmented lesions, very small missed lesions or small false positives do not contribute much to the DSC value. Hence, there is a need to report the volumes of false positives and false negatives. It will also be beneficial to evaluate the detection performances on a per-lesion basis (number of connected components detected vs missed), since automated detection of even a few voxels of all lesions can help physicians quickly locate the regions of interest, even if the DSC is low. Moreover, the difficulty of the segmentation/detection task is often not assessed via inter- or intra-observer agreement analysis. Our study aims to address these limitations. We trained and validated four deep neural networks on lymphoma PET/CT datasets from three cohorts, encompassing two distinct subtypes of lymphoma: DLBCL and primary mediastinal large B-cell lymphoma (PMBCL). (i) We performed both in (images coming from same cohorts as the training/validation set) and out-of-distribution or external (images from a fourth cohort not used for training/validation) testing to evaluate the robustness of our models. (ii) We reported the performance using DSC, volumes of false positives and negatives, and evaluated the performance dependence on six different types of lesion measures. (iii) We also evaluated the ability of our networks to reproduce these ground truth lesion measures and computed networks’ error in predicting them. (iv) We proposed three types of detection criteria for our use-case and evaluate model’s performance on these metrics. (v) Finally, we evaluated the intra- and inter-observer agreement to give a measure of the difficulty of lesion segmentation task on our datasets. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Related Work

ডিপ নিউরাল নেটওয়ার্ক লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করতে: সম্পর্কিত কাজ

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Verwandte Arbeiten

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: trabajo relacionado

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : travaux connexes

लिम्फोमा घावों का पता लगाने और उनकी मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: संबंधित कार्य

リンパ腫病変の検出と定量化のためのディープニューラルネットワーク: 関連研究

림프종 병변을 탐지하고 정량화하기 위한 심층 신경망: 관련 연구

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Trabalho Relacionado

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: соответствующая работа

Lenfoma Lezyonlarını Tespit ve Ölçmek İçin Derin Sinir Ağları: İlgili Çalışma

Mạng lưới thần kinh sâu để phát hiện và định lượng tổn thương ung thư hạch: Công việc liên quan

深度神经网络检测和量化淋巴瘤病变：相关工作

Authors:
(1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com);
(2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA;
(3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada;
(4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea;
(5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada;
(6) Don Wilson, BC Cancer, Vancouver, BC, Canada;
(7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada;
(8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada;
(9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada;
(10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA;
(11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA;
(12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA;
(13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada;
(14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro
Related Work
Materials and Methods
Results
Discussion
Conclusion and References V. DISCUSSION In this work, we trained and evaluated four distinct neural network architectures to automate the segmentation of lymphoma lesions from PET/CT datasets sourced from three different cohorts. To assess models performance, we conducted comprehensive evaluations on internal test set originating from these three cohorts and showed that SegResNet and UNet outperformed DynUNet and SwinUNETR on the DSC (mean and median) and median FPV metrics, while SwinUNETR had the best median FNV. In addition to internal evaluations, we extended our analysis to encompass an external outof-distribution testing phase on a sizable public lymphoma PET/CT dataset. On this external test set as well, SegResNet emerged as the top performer in terms of DSC and FPV metrics, underscoring its robustness and effectiveness, while UNet displayed the best performance on FNV. It is important to highlight that SegResNet and UNet were trained on patches of larger sizes, specifically (224, 224, 224) and (192, 192, 192) respectively, while DynUNet and SwinUNETR were trained using relatively smaller patches, namely (160, 160, 160) and (128, 128, 128) respectively. Utilizing larger patch sizes during training allows the neural networks to capture a more extensive contextual understanding of the data, thereby enhancing its performance in segmentation tasks [17]. This observation aligns with our results, where the superior performance of SegResNet and UNet can be attributed to their exposure to larger patch sizes during training. Moreover, larger batch sizes enable robust training by accurately estimating the gradients [17], but with our chosen training patch sizes, we could not train SegResNet, DynUNet and SwinUNETR with nb > 1 due to memory limitations (although we could accommodate nb = 8 for UNet). Hence, for a fair comparison between networks, all networks were trained with nb = 1. It is worth noting that our inability to train DynUNet and SwinUNETR on larger patch and mini-batch sizes was primarily due to computational resource limitations. However, this limitation presents an avenue for future research, where training these models with larger patches and batch sizes could potentially yield further improvements in segmentation accuracy. We assessed the reproducibility of lesions measures and found that on the internal test set, TMTV and TLG were reproducible across all networks, while Dmax was not reproducible by any network. SUVmean was reproducible by all networks except UNet, SUVmax by only SegResNet and number of lesions by only UNet and SegResNet. On the external test set, reproducibility was more limited, with only SUVmean being reproducible by both SegResNet and SwinUNETR, number of lesions by SegResNet, and TLG by DynUNet (Fig. 3 and 4). Furthermore, we quantified the networks’ error in estimating the value of lesion measures using MAPE and found that MAPE generally decreases as a function of lesion measure values (for all lesion measures) on the combined internal and external test set (Fig. 5). The networks generally made significant errors in the accurate prediction when the ground truth lesion measures were very small. We also showed that, in general, on a set of images with larger patient level lesion SUVmean, SUVmean, TMTV, and TLG, a network is able to predict a higher median DSC, although for very high values of these lesion measures, the performance generally plateaus. On the other hand, the DSC performance is not much affected by the number of lesions, while for a set of images with higher Dmax, the performance generally decreases for all networks (Fig. 7). As much of PET/CT data is privately owned by healthcare institutions, it poses significant challenges for researchers in accessing diverse datasets for training and testing deep learning models. In such a scenario, to improve the interpretability of models, it is crucial for researchers to investigate how the performance of their models depend on dataset characteristics. By studying how model performance correlates with the image/lesion characteristics, researchers can gain insights into the strengths and limitations of their models [13]. Alongside the evaluation of segmentation performance, we also introduced three distinct detection criteria, denoted as Criterion 1, 2, and 3. These criteria served a specific purpose: to evaluate the networks’ performance on a per-lesion basis. This stands in contrast to the segmentation performance assessment, which primarily focuses on the voxel-level accuracy of the networks. The rationale behind introducing these detection criteria lies in the need to assess how well the networks identify and detect lesions within the images, as opposed to merely evaluating their ability to delineate lesion boundaries at the voxel level. The ability to detect the presence of lesions (Criterion 1) is crucial, as it directly influences whether a potential health concern is identified or missed. Detecting even a single voxel of a lesion could trigger further investigation or treatment planning. Lesion count and accurate localization (Criterion 2) are important for treatment planning and monitoring disease progression. Knowing not only that a lesion exists but also how many there are and where they are located can significantly impact therapeutic decisions. Criterion 3 which focused on segmenting lesions based on lesion metabolic characteristics (SUVmax), adds an additional layer of clinical relevance. Using these detection metrics, we assessed the sensitivities and FP detections for all networks and showed that depending on the detection criteria, a network can have very high sensitivity even when the DSC performance was low. Given these different detection criteria, a trained model can be chosen based on specific clinical use cases. For example, some use cases might involve being able to detect all lesions without being overly cautious about segmenting exact lesion boundary, while some other use cases might be looking for more robust boundary delineations. Furthermore, we assessed the intra-observer variability of a physician in segmenting both “easy” and “hard” cases, noting challenges in consistent segmentation of cases from the “hard” subset. In lymphoma lesion segmentation, cases can vary in difficulty due to factors like size, shape, and location of lesions, or image quality. By identifying which cases are consistently difficult for even an experienced physician to segment, we gained insights into the complexities and nuances of the segmentation task. Finally, we also assessed the interobserver agreement between three physicians. Although, we inferred that there was substantial level of agreement between the three physicians, the assessment was performed only on 9 cases, resulting in low statistical power. To improve the consistency of ground truth in medical image segmentation, a well-defined protocol is essential. This protocol should engage multiple expert physicians independently in delineating regions of interest (ROIs) or lesions within PET/CT images. Instead of a single physician segmenting a cohort independently, multiple annotators should segment the same images without knowledge of each other’s work. Discrepancies or disagreements among physicians can be resolved through structured approaches such as facilitated discussions, clinical information reviews, or image clarification. This robust ground truth process enhances inter-observer agreement accuracy and strengthens the validity of research findings and clinical applications relying on these annotations. This paper is available on arxiv under CC 4.0 license. Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Authors: Authors: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro Related Work Materials and Methods Results Discussion Conclusion and References Abstract and Intro Abstract and Intro Related Work Related Work Materials and Methods Materials and Methods Results Results Discussion Discussion Conclusion and References Conclusion and References V. DISCUSSION In this work, we trained and evaluated four distinct neural network architectures to automate the segmentation of lymphoma lesions from PET/CT datasets sourced from three different cohorts. To assess models performance, we conducted comprehensive evaluations on internal test set originating from these three cohorts and showed that SegResNet and UNet outperformed DynUNet and SwinUNETR on the DSC (mean and median) and median FPV metrics, while SwinUNETR had the best median FNV. In addition to internal evaluations, we extended our analysis to encompass an external outof-distribution testing phase on a sizable public lymphoma PET/CT dataset. On this external test set as well, SegResNet emerged as the top performer in terms of DSC and FPV metrics, underscoring its robustness and effectiveness, while UNet displayed the best performance on FNV. It is important to highlight that SegResNet and UNet were trained on patches of larger sizes, specifically (224, 224, 224) and (192, 192, 192) respectively, while DynUNet and SwinUNETR were trained using relatively smaller patches, namely (160, 160, 160) and (128, 128, 128) respectively. Utilizing larger patch sizes during training allows the neural networks to capture a more extensive contextual understanding of the data, thereby enhancing its performance in segmentation tasks [17]. This observation aligns with our results, where the superior performance of SegResNet and UNet can be attributed to their exposure to larger patch sizes during training. Moreover, larger batch sizes enable robust training by accurately estimating the gradients [17], but with our chosen training patch sizes, we could not train SegResNet, DynUNet and SwinUNETR with nb > 1 due to memory limitations (although we could accommodate nb = 8 for UNet). Hence, for a fair comparison between networks, all networks were trained with nb = 1. It is worth noting that our inability to train DynUNet and SwinUNETR on larger patch and mini-batch sizes was primarily due to computational resource limitations. However, this limitation presents an avenue for future research, where training these models with larger patches and batch sizes could potentially yield further improvements in segmentation accuracy. We assessed the reproducibility of lesions measures and found that on the internal test set, TMTV and TLG were reproducible across all networks, while Dmax was not reproducible by any network. SUVmean was reproducible by all networks except UNet, SUVmax by only SegResNet and number of lesions by only UNet and SegResNet. On the external test set, reproducibility was more limited, with only SUVmean being reproducible by both SegResNet and SwinUNETR, number of lesions by SegResNet, and TLG by DynUNet (Fig. 3 and 4). Furthermore, we quantified the networks’ error in estimating the value of lesion measures using MAPE and found that MAPE generally decreases as a function of lesion measure values (for all lesion measures) on the combined internal and external test set (Fig. 5). The networks generally made significant errors in the accurate prediction when the ground truth lesion measures were very small. We also showed that, in general, on a set of images with larger patient level lesion SUVmean, SUVmean, TMTV, and TLG, a network is able to predict a higher median DSC, although for very high values of these lesion measures, the performance generally plateaus. On the other hand, the DSC performance is not much affected by the number of lesions, while for a set of images with higher Dmax, the performance generally decreases for all networks (Fig. 7). As much of PET/CT data is privately owned by healthcare institutions, it poses significant challenges for researchers in accessing diverse datasets for training and testing deep learning models. In such a scenario, to improve the interpretability of models, it is crucial for researchers to investigate how the performance of their models depend on dataset characteristics. By studying how model performance correlates with the image/lesion characteristics, researchers can gain insights into the strengths and limitations of their models [13]. Alongside the evaluation of segmentation performance, we also introduced three distinct detection criteria, denoted as Criterion 1, 2, and 3. These criteria served a specific purpose: to evaluate the networks’ performance on a per-lesion basis. This stands in contrast to the segmentation performance assessment, which primarily focuses on the voxel-level accuracy of the networks. The rationale behind introducing these detection criteria lies in the need to assess how well the networks identify and detect lesions within the images, as opposed to merely evaluating their ability to delineate lesion boundaries at the voxel level. The ability to detect the presence of lesions (Criterion 1) is crucial, as it directly influences whether a potential health concern is identified or missed. Detecting even a single voxel of a lesion could trigger further investigation or treatment planning. Lesion count and accurate localization (Criterion 2) are important for treatment planning and monitoring disease progression. Knowing not only that a lesion exists but also how many there are and where they are located can significantly impact therapeutic decisions. Criterion 3 which focused on segmenting lesions based on lesion metabolic characteristics (SUVmax), adds an additional layer of clinical relevance. Using these detection metrics, we assessed the sensitivities and FP detections for all networks and showed that depending on the detection criteria, a network can have very high sensitivity even when the DSC performance was low. Given these different detection criteria, a trained model can be chosen based on specific clinical use cases. For example, some use cases might involve being able to detect all lesions without being overly cautious about segmenting exact lesion boundary, while some other use cases might be looking for more robust boundary delineations. Furthermore, we assessed the intra-observer variability of a physician in segmenting both “easy” and “hard” cases, noting challenges in consistent segmentation of cases from the “hard” subset. In lymphoma lesion segmentation, cases can vary in difficulty due to factors like size, shape, and location of lesions, or image quality. By identifying which cases are consistently difficult for even an experienced physician to segment, we gained insights into the complexities and nuances of the segmentation task. Finally, we also assessed the interobserver agreement between three physicians. Although, we inferred that there was substantial level of agreement between the three physicians, the assessment was performed only on 9 cases, resulting in low statistical power. To improve the consistency of ground truth in medical image segmentation, a well-defined protocol is essential. This protocol should engage multiple expert physicians independently in delineating regions of interest (ROIs) or lesions within PET/CT images. Instead of a single physician segmenting a cohort independently, multiple annotators should segment the same images without knowledge of each other’s work. Discrepancies or disagreements among physicians can be resolved through structured approaches such as facilitated discussions, clinical information reviews, or image clarification. This robust ground truth process enhances inter-observer agreement accuracy and strengthens the validity of research findings and clinical applications relying on these annotations. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv available on arxiv

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Discussion

লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করার জন্য গভীর নিউরাল নেটওয়ার্ক: আলোচনা

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Diskussion

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: discusión

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : discussion

लिम्फोमा घावों का पता लगाने और मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: चर्चा

リンパ腫病変を検出し定量化するディープニューラルネットワーク：議論

림프종 병변을 탐지하고 정량화하기 위한 심층 신경망: 토론

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Discussão

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: обсуждение

Lenfoma Lezyonlarını Tespit Etmek ve Ölçmek İçin Derin Sinir Ağları: Tartışma

Mạng lưới thần kinh sâu để phát hiện và định lượng tổn thương ung thư hạch: Thảo luận

深度神经网络检测和量化淋巴瘤病变：讨论

Author:
(1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com);
(2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA;
(3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada;
(4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea;
(5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada;
(6) Don Wilson, BC Cancer, Vancouver, BC, Canada;
(7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada;
(8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada;
(9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada;
(10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA;
(11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA;
(12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA;
(13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada;
(14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro
Related Work
Materials and Methods
Results
Discussion
Conclusion and References Abstract This study performs comprehensive evaluation of four neural network architectures (UNet, SegResNet, DynUNet, and SwinUNETR) for lymphoma lesion segmentation from PET/CT images. These networks were trained, validated and tested on a diverse, multi-institutional dataset of 611 cases. Internal testing (88 cases; total metabolic tumor volume (TMTV) range [0.52, 2300] ml) showed SegResNet as the top performer with a median Dice similarity coefficient (DSC) of 0.76 and median false positive volume (FPV) of 4.55 ml; all networks had a median false negative volume (FNV) of 0 ml. On the unseen external test set (145 cases with TMTV range: [0.10, 2480] ml), SegResNet achieved the best median DSC of 0.68 and FPV of 21.46 ml, while UNet had the best FNV of 0.41 ml. We assessed reproducibility of six lesion measures, calculated their prediction errors, and examined DSC performance in relation to these lesion measures, offering insights into segmentation accuracy and clinical relevance. Additionally, we introduced three lesion detection criteria, addressing the clinical need for identifying lesions, counting them, and segmenting based on metabolic characteristics. We also performed expert intra-observer variability analysis revealing the challenges in segmenting “easy” vs. “hard” cases, to assist in the development of more resilient segmentation algorithms. Finally, we performed inter-observer agreement assessment underscoring the importance of a standardized ground truth segmentation protocol involving multiple expert annotators. Code is available at: https://github.com/microsoft/lymphoma-segmentationdnn. Index Terms— Positron emission tomography, computed tomography, deep learning, segmentation, detection, lesion measures, intra-observer variability, inter-observer variability I. INTRODUCTION F LUORODEOXYGLUCOSE (18F-FDG) PET/CT imaging is the standard of care for lymphoma patients, providing accurate diagnoses, staging, and therapy response evaluation. However, traditional qualitative assessments, like Deauville scores [1], can introduce variability due to observer’s subjectivity in image interpretation. Using quantitative PET analysis that incorporates lesion measures such as mean lesion standardized uptake value (SUVmean), total metabolic tumor volume (TMTV), and total lesion glycolysis (TLG) offers a promising path to more reliable prognostic decisions, enhancing our ability to predict patient outcomes in lymphoma with greater precision and confidence [2]. Quantitative assessment in PET/CT imaging often relies on manual lesion segmentation, which is time-consuming and prone to intra- and inter-observer variabilities. Traditional thresholding-based automated techniques can miss low-uptake disease and produce false positives in regions of physiological high uptake of radiotracers. Therefore, deep learning offers promise for automating lesion segmentation, reducing variability, increasing patient throughput, and potentially aiding in the detection of challenging lesions [3]. Although promising, deep learning methods face challenges of their own. Convolutional neural networks (CNNs) require large, well-annotated datasets that can be difficult to obtain. Models trained on small datasets may not be generalizable. Moreover, lymphoma lesions vary significantly in size, shape, and metabolic activity, making training deep networks accurately challenging in the absence of well-defined priors. Deep learning aims to reduce observer variability, but inconsistent manual annotations used for training can lead to error perpetuation. Understanding these challenges is crucial towards harnessing the full potential of these methods in PET/CT quantitative analysis. This paper is available on arxiv under CC 4.0 license. Author: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Author: Author: (1) Shadab Ahamed, University of British Columbia, Vancouver, BC, Canada, BC Cancer Research Institute, Vancouver, BC, Canada. He was also a Mitacs Accelerate Fellow (May 2022 - April 2023) with Microsoft AI for Good Lab, Redmond, WA, USA (e-mail: shadabahamed1996@gmail.com); (2) Yixi Xu, Microsoft AI for Good Lab, Redmond, WA, USA; (3) Claire Gowdy, BC Children’s Hospital, Vancouver, BC, Canada; (4) Joo H. O, St. Mary’s Hospital, Seoul, Republic of Korea; (5) Ingrid Bloise, BC Cancer, Vancouver, BC, Canada; (6) Don Wilson, BC Cancer, Vancouver, BC, Canada; (7) Patrick Martineau, BC Cancer, Vancouver, BC, Canada; (8) Franc¸ois Benard, BC Cancer, Vancouver, BC, Canada; (9) Fereshteh Yousefirizi, BC Cancer Research Institute, Vancouver, BC, Canada; (10) Rahul Dodhia, Microsoft AI for Good Lab, Redmond, WA, USA; (11) Juan M. Lavista, Microsoft AI for Good Lab, Redmond, WA, USA; (12) William B. Weeks, Microsoft AI for Good Lab, Redmond, WA, USA; (13) Carlos F. Uribe, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada; (14) Arman Rahmim, BC Cancer Research Institute, Vancouver, BC, Canada, and University of British Columbia, Vancouver, BC, Canada. Table of Links Abstract and Intro Related Work Materials and Methods Results Discussion Conclusion and References Abstract and Intro Abstract and Intro Related Work Related Work Materials and Methods Materials and Methods Results Results Discussion Discussion Conclusion and References Conclusion and References Abstract Abstract This study performs comprehensive evaluation of four neural network architectures (UNet, SegResNet, DynUNet, and SwinUNETR) for lymphoma lesion segmentation from PET/CT images. These networks were trained, validated and tested on a diverse, multi-institutional dataset of 611 cases. Internal testing (88 cases; total metabolic tumor volume (TMTV) range [0.52, 2300] ml) showed SegResNet as the top performer with a median Dice similarity coefficient (DSC) of 0.76 and median false positive volume (FPV) of 4.55 ml; all networks had a median false negative volume (FNV) of 0 ml. On the unseen external test set (145 cases with TMTV range: [0.10, 2480] ml), SegResNet achieved the best median DSC of 0.68 and FPV of 21.46 ml, while UNet had the best FNV of 0.41 ml. We assessed reproducibility of six lesion measures, calculated their prediction errors, and examined DSC performance in relation to these lesion measures, offering insights into segmentation accuracy and clinical relevance. Additionally, we introduced three lesion detection criteria, addressing the clinical need for identifying lesions, counting them, and segmenting based on metabolic characteristics. We also performed expert intra-observer variability analysis revealing the challenges in segmenting “easy” vs. “hard” cases, to assist in the development of more resilient segmentation algorithms. Finally, we performed inter-observer agreement assessment underscoring the importance of a standardized ground truth segmentation protocol involving multiple expert annotators. Code is available at: https://github.com/microsoft/lymphoma-segmentationdnn . https://github.com/microsoft/lymphoma-segmentationdnn Index Terms — Positron emission tomography, computed tomography, deep learning, segmentation, detection, lesion measures, intra-observer variability, inter-observer variability Index Terms I. INTRODUCTION F LUORODEOXYGLUCOSE (18F-FDG) PET/CT imaging is the standard of care for lymphoma patients, providing accurate diagnoses, staging, and therapy response evaluation. However, traditional qualitative assessments, like Deauville scores [1], can introduce variability due to observer’s subjectivity in image interpretation. Using quantitative PET analysis that incorporates lesion measures such as mean lesion standardized uptake value (SUVmean), total metabolic tumor volume (TMTV), and total lesion glycolysis (TLG) offers a promising path to more reliable prognostic decisions, enhancing our ability to predict patient outcomes in lymphoma with greater precision and confidence [2]. Quantitative assessment in PET/CT imaging often relies on manual lesion segmentation, which is time-consuming and prone to intra- and inter-observer variabilities. Traditional thresholding-based automated techniques can miss low-uptake disease and produce false positives in regions of physiological high uptake of radiotracers. Therefore, deep learning offers promise for automating lesion segmentation, reducing variability, increasing patient throughput, and potentially aiding in the detection of challenging lesions [3]. Although promising, deep learning methods face challenges of their own. Convolutional neural networks (CNNs) require large, well-annotated datasets that can be difficult to obtain. Models trained on small datasets may not be generalizable. Moreover, lymphoma lesions vary significantly in size, shape, and metabolic activity, making training deep networks accurately challenging in the absence of well-defined priors. Deep learning aims to reduce observer variability, but inconsistent manual annotations used for training can lead to error perpetuation. Understanding these challenges is crucial towards harnessing the full potential of these methods in PET/CT quantitative analysis. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Neural Networks to Detect and Quantify Lymphoma Lesions: Abstract and Intro

লিম্ফোমা ক্ষত সনাক্ত এবং পরিমাপ করার জন্য গভীর নিউরাল নেটওয়ার্ক: বিমূর্ত এবং ভূমিকা

Tiefe neuronale Netzwerke zur Erkennung und Quantifizierung von Lymphomläsionen: Zusammenfassung und Einführung

Redes neuronales profundas para detectar y cuantificar lesiones de linfoma: resumen e introducción

Réseaux de neurones profonds pour détecter et quantifier les lésions de lymphome : résumé et introduction

लिम्फोमा घावों का पता लगाने और मात्रा निर्धारित करने के लिए डीप न्यूरल नेटवर्क: सार और परिचय

リンパ腫病変を検出し定量化するディープニューラルネットワーク：概要と概要

림프종 병변을 탐지하고 정량화하기 위한 심층 신경망: 개요 및 소개

Redes Neurais Profundas para Detectar e Quantificar Lesões de Linfoma: Resumo e Introdução

Глубокие нейронные сети для обнаружения и количественной оценки поражений лимфомой: краткое содержание и введение

Lenfoma Lezyonlarını Tespit Etmek ve Ölçmek İçin Derin Sinir Ağları: Özet ve Giriş

Mạng lưới thần kinh sâu để phát hiện và định lượng các tổn thương ung thư hạch: Tóm tắt và giới thiệu

深度神经网络检测和量化淋巴瘤病变：摘要和简介

One of the most common question asked during deep learning knowledge interviews is - “Why can’t we use a quadratic cost function to train a Neural Network?”. In this article, we’ll discuss the answer to this question. There will be a lot of Math involved but nothing crazy! I will keep things simple and precise. Let’s start with contemplating the general architecture of a Neural network The General Architecture of a Neural Network We have a series of inputs forming an “input layer”, a series of neurons in the “hidden layer” and one neuron forming an “output layer” for a binary classification problem. For this question, we will assume that we are dealing with a binary classifier, so we have just one output value out of the network. Now, look at the following figure where we’ve highlighted the input layer in green, the output neuron in red, and one neuron of the hidden layer in orange. From all greens to the orange, we see that all inputs are connected to the orange neuron. In other words, the “activation” of the orange neuron happens using the “aggregation” of all the green neurons in the input layer. This process is replicated over all neurons over all layers until we reach the final red output neuron. What if we replace the orange neuron with the red neuron, i.e., we remove the hidden layer and connect the red neuron with the green neurons directly? We will get the following: For this question, we will assume the aforementioned ‘simplistic architecture’ and the result can be generalized to the complete architecture as well. Now let’s introduce some Math step-by-step. What we see above is the basic “weight update” equation for a Neural network. I have removed the extra hyperparameters such as the learning factor and the sub-sampling (min-batch), etc. w_k is the vector of weights and the weights are the ‘parameters’ of our Neural network model. w_k comprises individual weights gathered in a column vector. These weights are associated with the inputs to the model (that is the green neurons in our architecture). We have a cost function C_i where i = 1 to n are the number of data instances in our sample. The cost function C is the “error” between the actual output y and the output from the neural network (red neuron). Each data instance will produce a predicted output as against an actual output, therefore, there will be a cost or error for every data instance. The objective of the model is to minimize this cost function on an average over the entire dataset. And as we know, the minimization step involves taking a derivative with respect to the model parameters (weights). We do this using the partial derivative of the vector w_k. All this means is that the cost C will be an expression/aggregation of weights w_1 to w_q and we will differentiate to each weight w and collect that in a vector. This is called the negative “gradient vector”. It is used to update the weight vector from the k-th iteration to the (k+1)th iteration. The methodology is Stochastic Gradient descent but we will leave that out for this article. In a nutshell, the neural network learns by an update to the weights via the negative gradient vector averaged over all the samples and calculated for w_k. This helps us move to the minimization of the cost function and helps the network to learn and improve its accuracy. It is obvious that if the updates to the weights are not happening, because the negative gradient is moving towards zero, the learning has stopped. This doesn’t necessarily imply that we have reached the minimum! Because our cost function is highly complicated and we need to find a minimum in a multi-dimensional space. Therefore, there could be many local minima where the gradient is zero and the network stops learning. Anyway, we don’t have to worry about that for this problem. Let’s look at the following expression: This expression defines z_i as a weighted sum of the inputs x_ji. Note that these inputs are the green neurons in our architecture. As we have no hidden layer, we combine the inputs x_ji and the weights w_j and add a bias term to get z_i which is what is represented by the connecting arrows from the green neurons to the red neuron in our architecture. Since we have q inputs, we have x_j and w_j where j = 1 to q But, we don’t pass z_i to the red neuron. We apply an “activation function” to it. This activation function can be different for different neural networks. For the purposes of this problem, we assume the activation function is “Sigmoid” or “logistic”. I will assume here that the reader is aware of this function and move on further. Next comes our main problem statement -  How do we define the Cost function (C)? It is well known that for binary classification, the cost function is “Cross entropy” but the question here is why can’t it be “Quadratic”. Let’s define the expressions of both the cost functions: Quadratic cost function –   Cross Entropy cost function – While the quadratic cost function is straightforward (think least squares minimization between the actual output y_i and the predicted output a_i), we can offer some explanation for the cross-entropy cost function. This is akin to negative log-likelihood in our regression models. Note that there is a negative sign outside the brackets, which is used to keep the cost positive (because a_i will be between 0 and 1 - an output of sigmoid, therefore, the term inside the brackets will be always negative). Also note that when a_i gets really close to y_i, the cost gets really close to zero. This is because, when y_i = 1 and a_i ~ 1, ln(a_i) will be approximately 0. Similarly, when y_i= 0 and a_i  ~ 0, ln(1-a_i) will be approximately 0. Thus, this function keeps the cost positive and minimal when the model is predicting well. However, the same can be said about the quadratic cost function as well. But, we don’t use it. Why? Here comes the explanation We go back to the basic weight update equation we saw earlier and input the quadratic cost function to it. We get the following Now to keep things simple, we will consider only one data point, that is i=1 and n=1. And we differentiate partially for each weight w_j. We get the following: Recall that since i = 1, we have Substituting the value of z, we get That is our gradient vector which is responsible for updating the weights of the network will have a derivative of the sigmoid function when we use a quadratic cost function. Now let’s look at the behavior of the derivative of the sigmoid function: From the above plot, it is clear that the derivative, representing the slope of the sigmoid function, approaches 0 as soon as the input z becomes large! What does this mean? This means that the gradient vector will be zero when the activation input z is large. Therefore, the network will stop learning as the weights won’t get updated. Recall that this does not mean we have reached a minimum. This means we are stuck at an undesirable point and in the function space which could be far from the minimum value. This is known as “learning slow down”. However, this does NOT occur with a cross-entropy cost function. We perform the same substitution using the cross-entropy cost function and get the following: It is interesting to note that the term: occurs in the gradient for quadratic cost as well. However, there is a trick which we will use to simplify it. The gradient of the sigmoid function can be expressed as follows We substitute that into our original expression and get the following: That is our gradient vector which is responsible for updating the weights of the network does not have a derivative of the sigmoid function when we use a cross entropy cost function. Hence, there is no slowdown in learning with this cost function. We juxtapose the gradients again for a better look: This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well! One of the most common question asked during deep learning knowledge interviews is - “ Why can’t we use a quadratic cost function to train a Neural Network? ”. In this article, we’ll discuss the answer to this question. There will be a lot of Math involved but nothing crazy! I will keep things simple and precise. Why can’t we use a quadratic cost function to train a Neural Network? Let’s start with contemplating the general architecture of a Neural network Neural network The General Architecture of a Neural Network We have a series of inputs forming an “input layer”, a series of neurons in the “hidden layer” and one neuron forming an “output layer” for a binary classification problem. For this question, we will assume that we are dealing with a binary classifier, so we have just one output value out of the network. Now, look at the following figure where we’ve highlighted the input layer in green, the output neuron in red, and one neuron of the hidden layer in orange. From all greens to the orange, we see that all inputs are connected to the orange neuron. In other words, the “activation” of the orange neuron happens using the “aggregation” of all the green neurons in the input layer. This process is replicated over all neurons over all layers until we reach the final red output neuron. What if we replace the orange neuron with the red neuron, i.e., we remove the hidden layer and connect the red neuron with the green neurons directly? We will get the following: We will get the following: For this question, we will assume the aforementioned ‘simplistic architecture’ and the result can be generalized to the complete architecture as well. Now let’s introduce some Math step-by-step. Now let’s introduce some Math step-by-step. What we see above is the basic “weight update” equation for a Neural network. I have removed the extra hyperparameters such as the learning factor and the sub-sampling (min-batch), etc. w_k is the vector of weights and the weights are the ‘parameters’ of our Neural network model. w_k comprises individual weights gathered in a column vector. These weights are associated with the inputs to the model (that is the green neurons in our architecture). We have a cost function C_i where i = 1 to n are the number of data instances in our sample. The cost function C is the “error” between the actual output y and the output from the neural network (red neuron). Each data instance will produce a predicted output as against an actual output, therefore, there will be a cost or error for every data instance. The objective of the model is to minimize this cost function on an average over the entire dataset. And as we know, the minimization step involves taking a derivative with respect to the model parameters (weights). We do this using the partial derivative of the vector w_k. All this means is that the cost C will be an expression/aggregation of weights w_1 to w_q and we will differentiate to each weight w and collect that in a vector. This is called the negative “gradient vector”. It is used to update the weight vector from the k-th iteration to the (k+1)th iteration. The methodology is Stochastic Gradient descent but we will leave that out for this article. In a nutshell, the neural network learns by an update to the weights via the negative gradient vector averaged over all the samples and calculated for w_k. This helps us move to the minimization of the cost function and helps the network to learn and improve its accuracy. It is obvious that if the updates to the weights are not happening, because the negative gradient is moving towards zero, the learning has stopped. This doesn’t necessarily imply that we have reached the minimum! Because our cost function is highly complicated and we need to find a minimum in a multi-dimensional space. Therefore, there could be many local minima where the gradient is zero and the network stops learning. Anyway, we don’t have to worry about that for this problem. Let’s look at the following expression: Let’s look at the following expression: This expression defines z_i as a weighted sum of the inputs x_ji. Note that these inputs are the green neurons in our architecture. As we have no hidden layer, we combine the inputs x_ji and the weights w_j and add a bias term to get z_i which is what is represented by the connecting arrows from the green neurons to the red neuron in our architecture. Since we have q inputs, we have x_j and w_j where j = 1 to q But, we don’t pass z_i to the red neuron. We apply an “activation function” to it. This activation function can be different for different neural networks. For the purposes of this problem, we assume the activation function is “Sigmoid” or “logistic”. I will assume here that the reader is aware of this function and move on further. Next comes our main problem statement -  How do we define the Cost function (C)? It is well known that for binary classification, the cost function is “Cross entropy” but the question here is why can’t it be “Quadratic”. Let’s define the expressions of both the cost functions: Let’s define the expressions of both the cost functions: Quadratic cost function – Cross Entropy cost function – Quadratic cost function – Cross Entropy cost function – While the quadratic cost function is straightforward (think least squares minimization between the actual output y_i and the predicted output a_i), we can offer some explanation for the cross-entropy cost function. This is akin to negative log-likelihood in our regression models. Note that there is a negative sign outside the brackets, which is used to keep the cost positive (because a_i will be between 0 and 1 - an output of sigmoid, therefore, the term inside the brackets will be always negative). Also note that when a_i gets really close to y_i, the cost gets really close to zero. This is because, when y_i = 1 and a_i ~ 1, ln(a_i) will be approximately 0. Similarly, when y_i= 0 and a_i  ~ 0, ln(1-a_i) will be approximately 0. Thus, this function keeps the cost positive and minimal when the model is predicting well. However, the same can be said about the quadratic cost function as well. But, we don’t use it. Why? Here comes the explanation We go back to the basic weight update equation we saw earlier and input the quadratic cost function to it. We get the following Now to keep things simple, we will consider only one data point, that is i=1 and n=1. And we differentiate partially for each weight w_j. We get the following: Recall that since i = 1, we have Substituting the value of z, we get That is our gradient vector which is responsible for updating the weights of the network will have a derivative of the sigmoid function when we use a quadratic cost function. Now let’s look at the behavior of the derivative of the sigmoid function: Now let’s look at the behavior of the derivative of the sigmoid function: From the above plot, it is clear that the derivative, representing the slope of the sigmoid function, approaches 0 as soon as the input z becomes large! What does this mean? This means that the gradient vector will be zero when the activation input z is large. Therefore, the network will stop learning as the weights won’t get updated. Recall that this does not mean we have reached a minimum. This means we are stuck at an undesirable point and in the function space which could be far from the minimum value. This is known as “learning slow down”. However, this does NOT occur with a cross-entropy cost function. We perform the same substitution using the cross-entropy cost function and get the following: We perform the same substitution using the cross-entropy cost function and get the following: It is interesting to note that the term: occurs in the gradient for quadratic cost as well. However, there is a trick which we will use to simplify it. The gradient of the sigmoid function can be expressed as follows We substitute that into our original expression and get the following: That is our gradient vector which is responsible for updating the weights of the network does not have a derivative of the sigmoid function when we use a cross entropy cost function. Hence, there is no slowdown in learning with this cost function. not We juxtapose the gradients again for a better look: We juxtapose the gradients again for a better look: This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. This answers our original question - we don’t use the quadratic cost function because it leads to a slowdown of learning. Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well! Note that the above analysis was done only on the output layer (of one neuron), however, it is can be generalized for a general neural network as well!

Why Quadratic Cost Functions Are Ineffective in Neural Network Training

কেন চতুর্মুখী খরচ ফাংশন নিউরাল নেটওয়ার্ক প্রশিক্ষণ অকার্যকর হয়

Warum quadratische Kostenfunktionen beim Training neuronaler Netze unwirksam sind

Por qué las funciones de costo cuadrático son ineficaces en el entrenamiento de redes neuronales

Pourquoi les fonctions de coût quadratiques sont inefficaces dans la formation sur les réseaux neuronaux

न्यूरल नेटवर्क प्रशिक्षण में द्विघात लागत फ़ंक्शन अप्रभावी क्यों हैं

ニューラルネットワークのトレーニングで二次コスト関数が効果がない理由

신경망 훈련에서 2차 비용 함수가 비효율적인 이유

Por que as funções de custo quadráticas são ineficazes no treinamento de redes neurais

Почему квадратичные функции стоимости неэффективны при обучении нейронных сетей

İkinci Dereceden Maliyet Fonksiyonları Sinir Ağı Eğitiminde Neden Etkisizdir?

Tại sao hàm chi phí bậc hai không hiệu quả trong đào tạo mạng lưới thần kinh

为什么二次成本函数在神经网络训练中无效

Authors:
(1) Yuxin Meng;
(2) Feng Gao;
(3) Eric Rigall;
(4) Ran Dong;
(5) Junyu Dong;
(6) Qian Du. Table of Links Abstract and Intro
Background
Proposed Method
Experimental Results and Analysis
Conclusions and Future Work
References V. CONCLUSIONS AND FUTURE WORK In this paper, we present a SST prediction approach based on physical knowledge correction, which utilizes historical observed data to refine and adjust the physical component in the numerical model data. Specifically, a prior network was employed to extract physical knowledge from the observed data. Subsequently, we generated physics-enhanced SST by applying the pretrained prior network over numerical model data. Finally, the generated data were used to train the ConvLSTM network for SST prediction. Additionally, the physical knowledge-based enhanced data were leveraged to train the ConvLSTM network, which further improved the prediction performance. The proposed method achieved the best performance compared to six state-of-the-art methods. Although the physical part of the numerical model data has been corrected by our proposed method, the prediction performance could be further improved if an interpretable model is employed. In the future, we plan to extract more pertinent knowledge from the deep networks, and then design interpretable models more suitable for practical applications. This paper is available on arxiv under CC 4.0 license. Authors: (1) Yuxin Meng; (2) Feng Gao; (3) Eric Rigall; (4) Ran Dong; (5) Junyu Dong; (6) Qian Du. Authors: Authors: (1) Yuxin Meng; (2) Feng Gao; (3) Eric Rigall; (4) Ran Dong; (5) Junyu Dong; (6) Qian Du. Table of Links Abstract and Intro Background Proposed Method Experimental Results and Analysis Conclusions and Future Work References Abstract and Intro Abstract and Intro Background Background Proposed Method Proposed Method Experimental Results and Analysis Experimental Results and Analysis Conclusions and Future Work Conclusions and Future Work References References V. CONCLUSIONS AND FUTURE WORK In this paper, we present a SST prediction approach based on physical knowledge correction, which utilizes historical observed data to refine and adjust the physical component in the numerical model data. Specifically, a prior network was employed to extract physical knowledge from the observed data. Subsequently, we generated physics-enhanced SST by applying the pretrained prior network over numerical model data. Finally, the generated data were used to train the ConvLSTM network for SST prediction. Additionally, the physical knowledge-based enhanced data were leveraged to train the ConvLSTM network, which further improved the prediction performance. The proposed method achieved the best performance compared to six state-of-the-art methods. Although the physical part of the numerical model data has been corrected by our proposed method, the prediction performance could be further improved if an interpretable model is employed. In the future, we plan to extract more pertinent knowledge from the deep networks, and then design interpretable models more suitable for practical applications. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Deep Neural Network for Sea Surface Temperature Prediction: Conclusions & Future Work

সমুদ্র পৃষ্ঠের তাপমাত্রার পূর্বাভাসের জন্য গভীর নিউরাল নেটওয়ার্ক: উপসংহার এবং ভবিষ্যতের কাজ

Tiefes neuronales Netzwerk zur Vorhersage der Meeresoberflächentemperatur: Schlussfolgerungen und zukünftige Arbeiten

Red neuronal profunda para la predicción de la temperatura de la superficie del mar: conclusiones y trabajo futuro

Réseau neuronal profond pour la prévision de la température de la surface de la mer : conclusions et travaux futurs

समुद्री सतह के तापमान की भविष्यवाणी के लिए डीप न्यूरल नेटवर्क: निष्कर्ष और भविष्य का कार्य

海面温度予測のためのディープニューラルネットワーク：結論と今後の課題

해수면 온도 예측을 위한 심층 신경망: 결론 및 향후 연구

Rede Neural Profunda para Previsão da Temperatura da Superfície do Mar: Conclusões e Trabalhos Futuros

Глубокая нейронная сеть для прогнозирования температуры поверхности моря: выводы и будущая работа

Deniz Yüzeyi Sıcaklığı Tahmini için Derin Sinir Ağı: Sonuçlar ve Gelecekteki Çalışmalar

Mạng lưới thần kinh sâu để dự đoán nhiệt độ bề mặt biển: Kết luận và công việc trong tương lai

用于海面温度预测的深度神经网络：结论与未来工作

Stories, News & Mentions about NEURIA

Neuria’s Stories on HackerNoon