Gauss Labs

gausslabs

Table of Links Abstract and 1. Introduction 2. Methods 2.1. Modeling Panspermia and Terraformation 2.2. Identifying the Presence of Terraformed Planets and 2.3. Software and Availability 3. Results 3.1. Panspermia can increase the correlation between planets’ compositions and positions 3.2. Likely terraformed planets can be identified from clustering 4. Summary and Discussion 5. Acknowledgements and References APPENDIX A. Appendix 4. SUMMARY AND DISCUSSION We showed how a positive Mantel correlation coefficient with a low p-value can be used to identify when a region of space contains life, under the postulates that life can spread between planetary systems and terraform planets. Going beyond (or below) a population-scale biosignature, we demonstrated how to select, from a population of planets, clusters which have high-likelihood to contain terraformed planets. More importantly, these selected clusters have a low-likelihood to contain non-terraformed planets. This is increasingly important as the field of astrobiology reconciles with unavoidable false positives for more traditional biosignatures. We assumed that the background non-terraformed distribution of planetary compositions is uncorrelated and drawn from a flat random distribution of compositions (a reasonable prior given the high sensitivity of planet formation to initial conditions of the planetary disk; (Tasker et al. 2020; Pacetti et al. 2022)). In the event that underlying abiotic planets are correlated in position and composition space (perhaps from some undiscovered astrophysical phenomena), then the value of the Mantel correlation coefficient considered to be a statistically significant biosignature would have to increase. It’s possible that if the underlying coefficient of non-terraformed planets were too high, the spread of life would actually increase the Mantel p-value, even early in the simulation. If planetary compositions are actually clustered around a few characteristic planet types, we could represent the initial non-terraformed planets as being drawn from a distribution punctuated with narrow Gaussians in composition space, and the results might be different yet. Our results underscore the importance of better understanding the baseline abiotic diversity of planetary populations, including better constraining which features of planet formation best predict planetary characteristics over time. Our model demonstrates that such information could be used to aid life detection in the absence of making progress on a theory of living systems. Our approach here also provides an alternative to specific chemical biosignatures at the level of planets, showing the promise of detecting anomalous features at the population-scale. Critically, these anomalous features can be explained by a model with simple hypotheses about what life does, rather than what life is, and is agnostic to our ignorance of living systems as they might exist outside of Earth. This is only an introduction to the kinds of questions which could be investigated with this technique. The basic idea of taking key features of life, and understanding how they might manifest on the scale of planets leads us to imagine properties of life that may manifest across scales, like “planetary phylogenies”—that is, heredities and lineages of planets—and how they could shed light on better understanding living systems generally. Such hypothetical characteristics of planets (and life) could be investigated with our of model. Perhaps one of the biggest limitations in our model is assuming we can reliably map observable characteristics of a planet to something as comprehensible and malleable as a vector of real numbers. For example, how the chemistry of the atmosphere, biosphere, geosphere, etc. could be reflected via the atmosphere of a planet, and what the mappings are from observed atmospheric properties (the planet’s “phenotype”) to the underlying chemistries that put constraints on the compatibility of planets (the planet’s “genotype”), and the kinds of life which can emerge on or coexist with a planet. This would further strengthen the realism of our agnostic approach with minimal assumptions, and in future work we would like to explore these ideas more. While we only walk through a simulation under a single set of parameters here, our goal was to identify a best possible case scenario for applying a technique where we assume life should be able to drive correlations in planetary position-composition space based on hallmarks of life like proliferation and environmental bi-directional feedback. Our model results show promise that life could be detected at the scale of a population of planets, using information from only ≈ 1000 (perhaps fewer) planetary atmospheres, even in the absence of any information about what kinds of planetary environments are most suitable to life, or without knowing anything about the origins of life, or the peculiarities of life’s metabolic outputs. We showed how our distributed biosignature can be further refined to detect specific clusters of terraformed planets in this space, even when they only comprise a few percent of all planets. Again, this approach does not require an independent way to “detect” life, like a smoking gun biosignature atmospheric gas. Instead this approach depends on two key assumptions about what life can do, and derives observable consequences directly from them, providing a statistical approach which can be refined by astronomical surveys, and therein lies its biggest promise. Authors:
(1) Harrison B. Smith, Earth-Life Science Institute, Tokyo Institute of Technology, Ookayama, Meguro-ku, Tokyo, Japan, and Blue Marble Space Institute of Science, Seattle, Washington, USA (hbs@elsi.jp);
(2) Lana Sinapayen, Sony Computer Science Laboratories, Kyoto, Japan and National Institute for Basic Biology, Okazaki, Japan (lana.sinapayen@gmail.com). This paper is available on arxiv under CC BY-NC-ND 4.0 Deed license. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Methods 2.1. Modeling Panspermia and Terraformation 2.1. Modeling Panspermia and Terraformation 2.2. Identifying the Presence of Terraformed Planets and 2.3. Software and Availability 2.2. Identifying the Presence of Terraformed Planets and 2.3. Software and Availability 3. Results 3.1. Panspermia can increase the correlation between planets’ compositions and positions 3.1. Panspermia can increase the correlation between planets’ compositions and positions 3.2. Likely terraformed planets can be identified from clustering 3.2. Likely terraformed planets can be identified from clustering 4. Summary and Discussion 4. Summary and Discussion 5. Acknowledgements and References 5. Acknowledgements and References APPENDIX APPENDIX A. Appendix A. Appendix 4. SUMMARY AND DISCUSSION We showed how a positive Mantel correlation coefficient with a low p-value can be used to identify when a region of space contains life, under the postulates that life can spread between planetary systems and terraform planets. Going beyond (or below) a population-scale biosignature, we demonstrated how to select, from a population of planets, clusters which have high-likelihood to contain terraformed planets. More importantly, these selected clusters have a low-likelihood to contain non -terraformed planets. This is increasingly important as the field of astrobiology reconciles with unavoidable false positives for more traditional biosignatures. non We assumed that the background non-terraformed distribution of planetary compositions is uncorrelated and drawn from a flat random distribution of compositions (a reasonable prior given the high sensitivity of planet formation to initial conditions of the planetary disk; (Tasker et al. 2020; Pacetti et al. 2022)). In the event that underlying abiotic planets are correlated in position and composition space (perhaps from some undiscovered astrophysical phenomena), then the value of the Mantel correlation coefficient considered to be a statistically significant biosignature would have to increase. It’s possible that if the underlying coefficient of non-terraformed planets were too high, the spread of life would actually increase the Mantel p-value, even early in the simulation. If planetary compositions are actually clustered around a few characteristic planet types, we could represent the initial non-terraformed planets as being drawn from a distribution punctuated with narrow Gaussians in composition space, and the results might be different yet. Our results underscore the importance of better understanding the baseline abiotic diversity of planetary populations, including better constraining which features of planet formation best predict planetary characteristics over time. Our model demonstrates that such information could be used to aid life detection in the absence of making progress on a theory of living systems. Our approach here also provides an alternative to specific chemical biosignatures at the level of planets, showing the promise of detecting anomalous features at the population-scale. Critically, these anomalous features can be explained by a model with simple hypotheses about what life does, rather than what life is, and is agnostic to our ignorance of living systems as they might exist outside of Earth. This is only an introduction to the kinds of questions which could be investigated with this technique. The basic idea of taking key features of life, and understanding how they might manifest on the scale of planets leads us to imagine properties of life that may manifest across scales, like “planetary phylogenies”—that is, heredities and lineages of planets—and how they could shed light on better understanding living systems generally. Such hypothetical characteristics of planets (and life) could be investigated with our of model. Perhaps one of the biggest limitations in our model is assuming we can reliably map observable characteristics of a planet to something as comprehensible and malleable as a vector of real numbers. For example, how the chemistry of the atmosphere, biosphere, geosphere, etc. could be reflected via the atmosphere of a planet, and what the mappings are from observed atmospheric properties (the planet’s “phenotype”) to the underlying chemistries that put constraints on the compatibility of planets (the planet’s “genotype”), and the kinds of life which can emerge on or coexist with a planet. This would further strengthen the realism of our agnostic approach with minimal assumptions, and in future work we would like to explore these ideas more. While we only walk through a simulation under a single set of parameters here, our goal was to identify a best possible case scenario for applying a technique where we assume life should be able to drive correlations in planetary position-composition space based on hallmarks of life like proliferation and environmental bi-directional feedback. Our model results show promise that life could be detected at the scale of a population of planets, using information from only ≈ 1000 (perhaps fewer) planetary atmospheres, even in the absence of any information about what kinds of planetary environments are most suitable to life, or without knowing anything about the origins of life, or the peculiarities of life’s metabolic outputs. We showed how our distributed biosignature can be further refined to detect specific clusters of terraformed planets in this space, even when they only comprise a few percent of all planets. Again, this approach does not require an independent way to “detect” life, like a smoking gun biosignature atmospheric gas. Instead this approach depends on two key assumptions about what life can do, and derives observable consequences directly from them, providing a statistical approach which can be refined by astronomical surveys, and therein lies its biggest promise. Authors: (1) Harrison B. Smith, Earth-Life Science Institute, Tokyo Institute of Technology, Ookayama, Meguro-ku, Tokyo, Japan, and Blue Marble Space Institute of Science, Seattle, Washington, USA (hbs@elsi.jp); (2) Lana Sinapayen, Sony Computer Science Laboratories, Kyoto, Japan and National Institute for Basic Biology, Okazaki, Japan (lana.sinapayen@gmail.com). Authors: Authors: (1) Harrison B. Smith, Earth-Life Science Institute, Tokyo Institute of Technology, Ookayama, Meguro-ku, Tokyo, Japan, and Blue Marble Space Institute of Science, Seattle, Washington, USA ( hbs@elsi.jp ); hbs@elsi.jp (2) Lana Sinapayen, Sony Computer Science Laboratories, Kyoto, Japan and National Institute for Basic Biology, Okazaki, Japan ( lana.sinapayen@gmail.com ). lana.sinapayen@gmail.com This paper is available on arxiv under CC BY-NC-ND 4.0 Deed license. This paper is available on arxiv under CC BY-NC-ND 4.0 Deed license. available on arxiv

A Biosignature Based on Modeling Panspermia and Terraformation: Summary and Discussion

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. CONCLUSION We propose a 3D Gaussian segmentation method guided by 2D segmentation maps, attaching a probability distribution vector for each 3D Gaussian on various categories to enable the segmentation of the majority of 3D Gaussians in the scene. Meanwhile, we employ KNN clustering to utilize the spatial continuity of objects, ensuring that nearby 3D Gaussians belong to the same category. Additionally, optional statistical filtering is used to help remove those 3D Gaussians that are incorrectly segmented. As an initial step in 3D understanding and editing, this method has a wide range of potential applications in downstream tasks. We demonstrate the effectiveness of our method on common NeRF datasets. References [1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis, “3D Gaussian ¨ Splatting for Real-Time Radiance Field Rendering,” ACM Transactions on Graphics, vol. 42, no. 4, pp. 1–14, July 2023. [2] Weiping Liu, Jia Sun, Wanyi Li, Ting Hu, and Peng Wang, “Deep learning on point clouds and its application: A survey,” Sensors, vol. 19, no. 19, 2019. [3] Dawar Khan, Alexander Plopski, Yuichiro Fujimoto, Masayuki Kanbara, Gul Jabeen, Yongjie Jessica Zhang, Xiaopeng Zhang, and Hirokazu Kato, “Surface remeshing: A systematic literature review of methods and research directions,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 3, pp. 1680–1713, 2022. [4] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. [5] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, no. 1, pp. 99–106, dec 2021. [6] Johannes L. Schonberger and Jan-Michael Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. [7] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke, “Gaussian Grouping: Segment and Edit Anything in 3D Scenes,” arXiv e-prints, p. arXiv:2312.00732, Dec. 2023. [8] Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian, “Segment Any 3D Gaussians,” arXiv e-prints, p. arXiv:2312.00860, Dec. 2023. [9] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann, “Decomposing nerf for editing via feature field distillation,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds. 2022, vol. 35, pp. 23311–23330, Curran Associates, Inc. [10] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and P. J. Narayanan, “Interactive segmentation of radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 4201–4211. [11] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng, “DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation,” arXiv eprints, p. arXiv:2309.16653, Sept. 2023. [12] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang, “GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models,” arXiv e-prints, p. arXiv:2310.08529, Oct. 2023. [13] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, “High-resolution im- ¨ age synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10684–10695. [14] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang, “4D Gaussian Splatting for RealTime Dynamic Scene Rendering,” arXiv e-prints, p. arXiv:2310.08528, Oct. 2023. [15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick, “Segment Anything,” ´ arXiv e-prints, p. arXiv:2304.02643, Apr. 2023. [16] Bing Wang, Lu Chen, and Bo Yang, “DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images,” arXiv e-prints, p. arXiv:2208.07227, Aug. 2022. [17] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui, “Learning object-compositional neural radiance field for editable scene rendering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 13779–13788. [18] Youtan Yin, Zhoujie Fu, Fan Yang, and Guosheng Lin, “OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields,” arXiv e-prints, p. arXiv:2305.10503, May 2023. [19] Jesus Zarzar, Sara Rojas, Silvio Giancola, and Bernard Ghanem, “SegNeRF: 3D Part Segmentation with Neural Radiance Fields,” arXiv e-prints, p. arXiv:2211.11215, Nov. 2022. [20] Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, and Alex Levinshtein, “Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 20669–20679. [21] Zhenxing MI and Dan Xu, “Switch-neRF: Learning scene decomposition with mixture of experts for largescale neural radiance fields,” in The Eleventh International Conference on Learning Representations, 2023. [22] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve´ Jegou, Julien Mairal, Piotr Bojanowski, and Armand ´ Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 9650–9660. [23] Yuying Hao, Yi Liu, Yizhou Chen, Lin Han, Juncai Peng, Shiyu Tang, Guowei Chen, Zewu Wu, Zeyu Chen, and Baohua Lai, “Eiseg: An efficient interactive segmentation tool based on paddlepaddle,” arXiv e-prints, pp. arXiv–2210, 2022. [24] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo OrtizCayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar, “Local light field fusion: Practical view synthesis with prescriptive sampling guidelines,” ACM Trans. Graph., vol. 38, no. 4, jul 2019. [25] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 5470–5479. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References 5. CONCLUSION We propose a 3D Gaussian segmentation method guided by 2D segmentation maps, attaching a probability distribution vector for each 3D Gaussian on various categories to enable the segmentation of the majority of 3D Gaussians in the scene. Meanwhile, we employ KNN clustering to utilize the spatial continuity of objects, ensuring that nearby 3D Gaussians belong to the same category. Additionally, optional statistical filtering is used to help remove those 3D Gaussians that are incorrectly segmented. As an initial step in 3D understanding and editing, this method has a wide range of potential applications in downstream tasks. We demonstrate the effectiveness of our method on common NeRF datasets. References [1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis, “3D Gaussian ¨ Splatting for Real-Time Radiance Field Rendering,” ACM Transactions on Graphics, vol. 42, no. 4, pp. 1–14, July 2023. [2] Weiping Liu, Jia Sun, Wanyi Li, Ting Hu, and Peng Wang, “Deep learning on point clouds and its application: A survey,” Sensors, vol. 19, no. 19, 2019. [3] Dawar Khan, Alexander Plopski, Yuichiro Fujimoto, Masayuki Kanbara, Gul Jabeen, Yongjie Jessica Zhang, Xiaopeng Zhang, and Hirokazu Kato, “Surface remeshing: A systematic literature review of methods and research directions,” IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 3, pp. 1680–1713, 2022. [4] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. [5] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, no. 1, pp. 99–106, dec 2021. [6] Johannes L. Schonberger and Jan-Michael Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. [7] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke, “Gaussian Grouping: Segment and Edit Anything in 3D Scenes,” arXiv e-prints, p. arXiv:2312.00732, Dec. 2023. [8] Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian, “Segment Any 3D Gaussians,” arXiv e-prints, p. arXiv:2312.00860, Dec. 2023. [9] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann, “Decomposing nerf for editing via feature field distillation,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds. 2022, vol. 35, pp. 23311–23330, Curran Associates, Inc. [10] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and P. J. Narayanan, “Interactive segmentation of radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 4201–4211. [11] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng, “DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation,” arXiv eprints, p. arXiv:2309.16653, Sept. 2023. [12] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang, “GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models,” arXiv e-prints, p. arXiv:2310.08529, Oct. 2023. [13] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, “High-resolution im- ¨ age synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10684–10695. [14] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang, “4D Gaussian Splatting for RealTime Dynamic Scene Rendering,” arXiv e-prints, p. arXiv:2310.08528, Oct. 2023. [15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick, “Segment Anything,” ´ arXiv e-prints, p. arXiv:2304.02643, Apr. 2023. [16] Bing Wang, Lu Chen, and Bo Yang, “DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images,” arXiv e-prints, p. arXiv:2208.07227, Aug. 2022. [17] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui, “Learning object-compositional neural radiance field for editable scene rendering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 13779–13788. [18] Youtan Yin, Zhoujie Fu, Fan Yang, and Guosheng Lin, “OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields,” arXiv e-prints, p. arXiv:2305.10503, May 2023. [19] Jesus Zarzar, Sara Rojas, Silvio Giancola, and Bernard Ghanem, “SegNeRF: 3D Part Segmentation with Neural Radiance Fields,” arXiv e-prints, p. arXiv:2211.11215, Nov. 2022. [20] Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, and Alex Levinshtein, “Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 20669–20679. [21] Zhenxing MI and Dan Xu, “Switch-neRF: Learning scene decomposition with mixture of experts for largescale neural radiance fields,” in The Eleventh International Conference on Learning Representations, 2023. [22] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve´ Jegou, Julien Mairal, Piotr Bojanowski, and Armand ´ Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 9650–9660. [23] Yuying Hao, Yi Liu, Yizhou Chen, Lin Han, Juncai Peng, Shiyu Tang, Guowei Chen, Zewu Wu, Zeyu Chen, and Baohua Lai, “Eiseg: An efficient interactive segmentation tool based on paddlepaddle,” arXiv e-prints, pp. arXiv–2210, 2022. [24] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo OrtizCayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar, “Local light field fusion: Practical view synthesis with prescriptive sampling guidelines,” ACM Trans. Graph., vol. 38, no. 4, jul 2019. [25] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 5470–5479. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Conclusion and References

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 4. EXPERIMENT 4.1. Setups Due to the scarcity of 3D Gaussian segmentation methods and the lack of open source code for Gaussian Grouping [7] and SAGA [8], we chose to compare our method with previous NeRF segmentation methods [10]. For this purpose, we selected well-known NeRF datasets for our experiments, including LLFF [24], NeRF-360 [5], and Mip-NeRF 360 [25]. Both LLFF and NeRF-360 are centered on objects in the scene, with the difference that the camera viewpoint of the former varies in a small range, while the latter contains a 360° image around the object. Mip-NeRF 360 features an unbounded scene, and its camera viewpoint also varies in a large range. In the Gaussian Clustering stage, the probability threshold β of each 3D Gaussian is set at 0.65, while the 50 3D Gaussians closest to its distance are filtered for subsequent computation. The 3D Gaussians are built and trained on a single Nvidia Geforce RTX 3090 GPU. 4.2. Result Fig. 2 illustrates the segmentation effects of this method in various scenes. The first two rows demonstrate the segmentation performance when the camera position varies within a small range. The third row depicts the segmentation effect in a 360° scene. The final row highlights the results of multiobject segmentation, where distinct objects such as the TV, desk, and table are segmented separately. Our method’s efficiency is enhanced by the addition of object code, a simple yet effective tool for handling complex scenes. In the first row, this code enables the successful removal of complex background elements like the leaves behind the flower. In the third row, it ensures accurate segmentation even when there is a significant change in the viewing angle. Moreover, the object code, which encapsulates the probability distribution of the 3D Gaussian across all classes, facilitates the simultaneous segmentation of multiple objects in a scene. Fig. 3 illustrates the comparative results between our method and ISRF [10]. Owing to the explicit representation of 3D Gaussians, our segmentation results are more precise in detail compared to those of ISRF, which is particularly evident in the leaf section of Fig. 3. 4.3. Ablations Fig. 4 presents the results of the ablation experiments, clearly demonstrating the effectiveness of KNN clustering and statistical filtering. Fig. 4(b) shows the initial segmented foreground object obtained without KNN clustering or statistical filtering, where it is noticeable that some leaves behind the flower are erroneously segmented. Fig. 4(c) displays the segmented foreground image after KNN clustering. Since KNN primarily addresses Gaussians with ambiguous semantic information, its impact on the visualization result is minimal. However, it can be observed that some incorrectly segmented Gaussians have been removed. Finally, Fig. 4(d) shows the result obtained after applying both KNN clustering and statistical filtering, which successfully filters out those Gaussians that were incorrectly segmented. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References 4. EXPERIMENT 4.1. Setups Due to the scarcity of 3D Gaussian segmentation methods and the lack of open source code for Gaussian Grouping [7] and SAGA [8], we chose to compare our method with previous NeRF segmentation methods [10]. For this purpose, we selected well-known NeRF datasets for our experiments, including LLFF [24], NeRF-360 [5], and Mip-NeRF 360 [25]. Both LLFF and NeRF-360 are centered on objects in the scene, with the difference that the camera viewpoint of the former varies in a small range, while the latter contains a 360° image around the object. Mip-NeRF 360 features an unbounded scene, and its camera viewpoint also varies in a large range. In the Gaussian Clustering stage, the probability threshold β of each 3D Gaussian is set at 0.65, while the 50 3D Gaussians closest to its distance are filtered for subsequent computation. The 3D Gaussians are built and trained on a single Nvidia Geforce RTX 3090 GPU. 4.2. Result Fig. 2 illustrates the segmentation effects of this method in various scenes. The first two rows demonstrate the segmentation performance when the camera position varies within a small range. The third row depicts the segmentation effect in a 360° scene. The final row highlights the results of multiobject segmentation, where distinct objects such as the TV, desk, and table are segmented separately. Our method’s efficiency is enhanced by the addition of object code, a simple yet effective tool for handling complex scenes. In the first row, this code enables the successful removal of complex background elements like the leaves behind the flower. In the third row, it ensures accurate segmentation even when there is a significant change in the viewing angle. Moreover, the object code, which encapsulates the probability distribution of the 3D Gaussian across all classes, facilitates the simultaneous segmentation of multiple objects in a scene. Fig. 3 illustrates the comparative results between our method and ISRF [10]. Owing to the explicit representation of 3D Gaussians, our segmentation results are more precise in detail compared to those of ISRF, which is particularly evident in the leaf section of Fig. 3. 4.3. Ablations Fig. 4 presents the results of the ablation experiments, clearly demonstrating the effectiveness of KNN clustering and statistical filtering. Fig. 4(b) shows the initial segmented foreground object obtained without KNN clustering or statistical filtering, where it is noticeable that some leaves behind the flower are erroneously segmented. Fig. 4(c) displays the segmented foreground image after KNN clustering. Since KNN primarily addresses Gaussians with ambiguous semantic information, its impact on the visualization result is minimal. However, it can be observed that some incorrectly segmented Gaussians have been removed. Finally, Fig. 4(d) shows the result obtained after applying both KNN clustering and statistical filtering, which successfully filters out those Gaussians that were incorrectly segmented. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Setups, Result and Ablations

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 3.2. Gaussian Clustering During experiments, we observed that employing 2D segmentation maps as the sole guide for learning 3D semantic information may lead to inaccuracies in the semantic information of some 3D Gaussians. These inaccuracies manifest either as 3D Gaussians approximating an initial state of uniform distribution across all categories or as exhibiting similar probabilities in a limited number of categories. To address this issue, and considering that objects are continuously distributed in space, we posit that each 3D Gaussian should typically be classified within the same category as other 3D Gaussians located within a certain proximity. To remedy the inaccuracies in semantic information, we refer to the KNN clustering algorithm. For a 3D scene with pre-learned semantic information, we initially retrieve the object code, denoted as o, of each 3D Gaussian used to represent the scene. These codes then undergo softmax processing to deduce the probability distribution of each 3D Gaussian across various categories. 3D Gaussians with maximum probability values max(sof tmax(o)) < β are selected. Finally, we fed the object codes of these selected 3D Gaussians along with their center coordinates into KNN for clustering. For a query 3D Gaussian, we calculate its distance from the surrounding 3D Gaussians, and the k 3D Gaussians closest in distance are selected, the object code of the query Gaussian is set to the mean of these 3D Gaussians’ object code. 3.3. Gaussian Filtering During experiments, We also found that after 3D semantic information learning and Gaussian clustering, some 3D Gaussians not belonging to the object intended for segmentation were incorrectly segmented out. We observed that these erroneously segmented 3D Gaussians are spatially distant from the rest of the segmented 3D Gaussians, as shown in Fig. 4(a). Therefore, we employ a statistical filtering algorithm similar to that used in point cloud segmentation to solve this problem. For each segmented Gaussian, we calculate its average distance D from the neighboring 3D Gaussians. Then, we compute the mean µ and variance σ of these average distances. Finally, we remove those 3D Gaussians whose average distance D > µ + σ from the current segmentation results. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References 3.2. Gaussian Clustering During experiments, we observed that employing 2D segmentation maps as the sole guide for learning 3D semantic information may lead to inaccuracies in the semantic information of some 3D Gaussians. These inaccuracies manifest either as 3D Gaussians approximating an initial state of uniform distribution across all categories or as exhibiting similar probabilities in a limited number of categories. To address this issue, and considering that objects are continuously distributed in space, we posit that each 3D Gaussian should typically be classified within the same category as other 3D Gaussians located within a certain proximity. To remedy the inaccuracies in semantic information, we refer to the KNN clustering algorithm. For a 3D scene with pre-learned semantic information, we initially retrieve the object code, denoted as o, of each 3D Gaussian used to represent the scene. These codes then undergo softmax processing to deduce the probability distribution of each 3D Gaussian across various categories. 3D Gaussians with maximum probability values max(sof tmax(o)) < β are selected. Finally, we fed the object codes of these selected 3D Gaussians along with their center coordinates into KNN for clustering. For a query 3D Gaussian, we calculate its distance from the surrounding 3D Gaussians, and the k 3D Gaussians closest in distance are selected, the object code of the query Gaussian is set to the mean of these 3D Gaussians’ object code. max(sof tmax(o)) < β k 3.3. Gaussian Filtering During experiments, We also found that after 3D semantic information learning and Gaussian clustering, some 3D Gaussians not belonging to the object intended for segmentation were incorrectly segmented out. We observed that these erroneously segmented 3D Gaussians are spatially distant from the rest of the segmented 3D Gaussians, as shown in Fig. 4(a). Therefore, we employ a statistical filtering algorithm similar to that used in point cloud segmentation to solve this problem. For each segmented Gaussian, we calculate its average distance D from the neighboring 3D Gaussians. Then, we compute the mean µ and variance σ of these average distances. Finally, we remove those 3D Gaussians whose average distance D > µ + σ from the current segmentation results. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Gaussian Clustering and Filtering

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 3. METHOD Given a well-trained scene using 3D Gaussian representation, scene rendering images, and corresponding camera parameters, we initially employed an interactive 2D segmentation model [23] to segment the rendered images. Then, the obtained 2D segmentation maps are used as guidance to facilitate the learning of semantic information (object code) added to the 3D Gaussians. Finally, we use KNN clustering to address issues of semantic ambiguity in certain 3D Gaussians, while optional statistical filtering can help eliminate those 3D Gaussians that have been erroneously segmented. The pipeline is depicted in Fig. 1. 3.1. Point-Based rendering and Semantic Information Learning Gaussian Splatting [1] employs a point-based rendering technique (α-blending) to render a 3D scene onto a plane, and the color of a pixel on the plane can be calculated as: To achieve segmentation of a 3D scene, semantic information needs to be incorporated into the representation of the scene. Inspired by 2D segmentation, we assign an object code To use 2D segmentation maps as supervision for learning the added 3D semantic information, it is necessary to project the added semantic information from 3D onto a 2D plane. Inspired by α-blending, we consider the pixel categories in the rendered 2D segmentation map as a weighted sum of the categories of multiple 3D Gaussians along the current ray during rendering. We assume that the first 3D Gaussian contributes the most, with each subsequent 3D Gaussian’s contribution diminishing in accordance with its distance from the rendering plane, and this contribution is also proportional to the size of the 3D Gaussian itself. The category of each pixel on the rendered image can be represented by the object code of o the 3D Gaussians as: which simply replaces the color c of each 3D Gaussian in Eq. (1) with the object code of each 3D Gaussian. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References 3. METHOD Given a well-trained scene using 3D Gaussian representation, scene rendering images, and corresponding camera parameters, we initially employed an interactive 2D segmentation model [23] to segment the rendered images. Then, the obtained 2D segmentation maps are used as guidance to facilitate the learning of semantic information (object code) added to the 3D Gaussians. Finally, we use KNN clustering to address issues of semantic ambiguity in certain 3D Gaussians, while optional statistical filtering can help eliminate those 3D Gaussians that have been erroneously segmented. The pipeline is depicted in Fig. 1. 3.1. Point-Based rendering and Semantic Information Learning Gaussian Splatting [1] employs a point-based rendering technique (α-blending) to render a 3D scene onto a plane, and the color of a pixel on the plane can be calculated as: To achieve segmentation of a 3D scene, semantic information needs to be incorporated into the representation of the scene. Inspired by 2D segmentation, we assign an object code To use 2D segmentation maps as supervision for learning the added 3D semantic information, it is necessary to project the added semantic information from 3D onto a 2D plane. Inspired by α-blending, we consider the pixel categories in the rendered 2D segmentation map as a weighted sum of the categories of multiple 3D Gaussians along the current ray during rendering. We assume that the first 3D Gaussian contributes the most, with each subsequent 3D Gaussian’s contribution diminishing in accordance with its distance from the rendering plane, and this contribution is also proportional to the size of the 3D Gaussian itself. The category of each pixel on the rendered image can be represented by the object code of o the 3D Gaussians as: which simply replaces the color c of each 3D Gaussian in Eq. (1) with the object code of each 3D Gaussian. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Rendering and Semantic Information

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 2. RELATED WORK 3D Gaussian, a recently proposed explicit representation method, has attained remarkable achievements in three-dimensional scene reconstruction [1]. Its biggest advantage is the capability of real-time rendering. Utilizing a series of scene images and corresponding camera data, it employs 3D Gaussians to depict scene objects. Each 3D Gaussian is defined by parameters including mean, covariance matrix, opacity, and spherical harmonics. The mean pinpoints the Gaussian’s central position in the 3D scene. Expressed by a scaling matrix S and a rotation matrix R, the covariance matrix describes the Gaussian’s size and shape, while the spherical harmonics encode its color information. Gaussian Splatting then utilizes point-based rendering for efficient 3D to 2D projection. Recent developments have seen numerous advancements in Gaussian Splatting. Innovations like DreamGaussian [11] and GaussianDreamer [12] merge this technique with Diffusion model [13], facilitating text-to-3D generation. 4D Gaussian Splatting [14] extends these methods to dynamic scene representation and rendering. Focusing on segmentation, Gaussian Grouping [7] and SAGA [8] have made significant strides. They both employ the Segment Anything Model (SAM) [15] to derive 2D prior segmentation data, guiding the learning of added semantic information in 3D Gaussians. In Gaussian Grouping, this information is conveyed similarly to coefficients of spherical harmonic functions, whereas SAGA uses learnable low-dimensional features. However, SAM’s reliance on geometric structures limits its semantic inclusivity in each mask. Thus, both methods propose strategies to ensure consistency of SAM’s segmentation outcomes from various perspectives. Gaussian Grouping treats images from different angles as a sequence of video frames, utilizing a pre-trained model for mask propagation and matching. In contrast, SAGA consolidates consistent, multi-granularity segmentation information across viewpoints, employing a custom-designed SAM-guidance loss. 3D Segmentation in Radiance Fields. Prior to the advent of 3D Gaussians, NeRF [5] stood as a prominent method in 3D characterization, sparking a plethora of derivative works [10, 16–21], including several focusing on decomposing and segmenting NeRF. A notable example is Object NeRF [17], which introduced a dual-pathway neural radiance field adept at object decomposition. Its scene branch processes spatial coordinates and viewing directions, outputting density and color details of a point from the viewer’s perspective, primarily encoding the background of the 3D scene and offering geometric context for the object branch. Uniquely, the object branch, in addition to spatial and directional inputs, integrates a learnable object activation code, enabling the independent learning of neural radiance fields for each scene object. And the 3D guard msak helps mitigate occlusion issues between objects during the learning phase. Similarly, Switch-NeRF [21] demonstrates the decomposition of large-scale neural radiance fields through a trainable gating network. DM-NeRF [16] introduces an object field for NeRF segmentation, using it to generate a one-hot vector indicating mentation, using it to generate a one-hot vector indicating the ownership of each spatial point by an object. SPInNeRF [20] employs a semantic radiance field, assessing the likelihood of scene locations being associated with specific objects. ISRF [10] adds semantic features to specific points and incorporates DINO [22] features of rendered images into this framework through a teacher-student model, allowing for feature interpolation at any given point. Techniques such as K-means clustering, nearest neighbor matching, and bilateral search are integrated, enabling interactive NeRF segmentation. Additionally, OR-NeRF[18] chooses to back-project 2D segmentation results into a 3D space, propagating them across different viewpoints, and then re-rendering them onto a 2D plane. These 3D Gaussian and NeRF segmentation methods either take a long time or struggle to preserve the detailed features of the scene in the segmentation result. For this reason, we propose a method that can segment multiple objects while preserving the detailed features in a short time. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References 2. RELATED WORK 3D Gaussian , a recently proposed explicit representation method, has attained remarkable achievements in three-dimensional scene reconstruction [1]. Its biggest advantage is the capability of real-time rendering. Utilizing a series of scene images and corresponding camera data, it employs 3D Gaussians to depict scene objects. Each 3D Gaussian is defined by parameters including mean, covariance matrix, opacity, and spherical harmonics. The mean pinpoints the Gaussian’s central position in the 3D scene. Expressed by a scaling matrix S and a rotation matrix R, the covariance matrix describes the Gaussian’s size and shape, while the spherical harmonics encode its color information. Gaussian Splatting then utilizes point-based rendering for efficient 3D to 2D projection. 3D Gaussian Recent developments have seen numerous advancements in Gaussian Splatting. Innovations like DreamGaussian [11] and GaussianDreamer [12] merge this technique with Diffusion model [13], facilitating text-to-3D generation. 4D Gaussian Splatting [14] extends these methods to dynamic scene representation and rendering. Focusing on segmentation, Gaussian Grouping [7] and SAGA [8] have made significant strides. They both employ the Segment Anything Model (SAM) [15] to derive 2D prior segmentation data, guiding the learning of added semantic information in 3D Gaussians. In Gaussian Grouping, this information is conveyed similarly to coefficients of spherical harmonic functions, whereas SAGA uses learnable low-dimensional features. However, SAM’s reliance on geometric structures limits its semantic inclusivity in each mask. Thus, both methods propose strategies to ensure consistency of SAM’s segmentation outcomes from various perspectives. Gaussian Grouping treats images from different angles as a sequence of video frames, utilizing a pre-trained model for mask propagation and matching. In contrast, SAGA consolidates consistent, multi-granularity segmentation information across viewpoints, employing a custom-designed SAM-guidance loss. 3D Segmentation in Radiance Fields. Prior to the advent of 3D Gaussians, NeRF [5] stood as a prominent method in 3D characterization, sparking a plethora of derivative works [10, 16–21], including several focusing on decomposing and segmenting NeRF. A notable example is Object NeRF [17], which introduced a dual-pathway neural radiance field adept at object decomposition. Its scene branch processes spatial coordinates and viewing directions, outputting density and color details of a point from the viewer’s perspective, primarily encoding the background of the 3D scene and offering geometric context for the object branch. Uniquely, the object branch, in addition to spatial and directional inputs, integrates a learnable object activation code, enabling the independent learning of neural radiance fields for each scene object. And the 3D guard msak helps mitigate occlusion issues between objects during the learning phase. Similarly, Switch-NeRF [21] demonstrates the decomposition of large-scale neural radiance fields through a trainable gating network. 3D Segmentation in Radiance Fields. DM-NeRF [16] introduces an object field for NeRF segmentation, using it to generate a one-hot vector indicating mentation, using it to generate a one-hot vector indicating the ownership of each spatial point by an object. SPInNeRF [20] employs a semantic radiance field, assessing the likelihood of scene locations being associated with specific objects. ISRF [10] adds semantic features to specific points and incorporates DINO [22] features of rendered images into this framework through a teacher-student model, allowing for feature interpolation at any given point. Techniques such as K-means clustering, nearest neighbor matching, and bilateral search are integrated, enabling interactive NeRF segmentation. Additionally, OR-NeRF[18] chooses to back-project 2D segmentation results into a 3D space, propagating them across different viewpoints, and then re-rendering them onto a 2D plane. These 3D Gaussian and NeRF segmentation methods either take a long time or struggle to preserve the detailed features of the scene in the segmentation result. For this reason, we propose a method that can segment multiple objects while preserving the detailed features in a short time. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Related Work

Authors:
(1) Kun Lan, University of Science and Technology of China;
(2) Haoran Li, University of Science and Technology of China;
(3) Haolin Shi, University of Science and Technology of China;
(4) Wenjun Wu, University of Science and Technology of China;
(5) Yong Liao, University of Science and Technology of China;
(6) Lin Wang, AI Thrust, HKUST(GZ);
(7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References ABSTRACT Recently, 3D Gaussian, as an explicit 3D representation method, has demonstrated strong competitiveness over NeRF (Neural Radiance Fields) in terms of expressing complex scenes and training duration. These advantages signal a wide range of applications for 3D Gaussians in 3D understanding and editing. Meanwhile, the segmentation of 3D Gaussians is still in its infancy. The existing segmentation methods are not only cumbersome but also incapable of segmenting multiple objects simultaneously in a short amount of time. In response, this paper introduces a 3D Gaussian segmentation method implemented with 2D segmentation as supervision. This approach uses input 2D segmentation maps to guide the learning of the added 3D Gaussian semantic information, while nearest neighbor clustering and statistical filtering refine the segmentation results. Experiments show that our concise method can achieve comparable performances on mIOU and mAcc for multi-object segmentation as previous single-object segmentation methods. Index Terms— 3D Gaussian, 3D Segmentation 1. INTRODUCTION The recently emerged 3D Gaussian technique [1] marks a significant advancement over previous 3D representation methods such as point clouds [2], meshes [3], signed distance functions (SDF) [4], and neural radiance fields (NeRF) [5], especially in terms of training time and scene reconstruction quality. The mean of each 3D Gaussian represents the position of its center point, the covariance matrix indicates rotation and size, and spherical harmonics express color. Starting with point clouds obtained from SFM [6], 3D Gaussians inherently contain the scene’s geometric information, thus saving time in locating areas with concentrated objects in space. Moreover, their explicit expression method further accelerates calculations of color and density for every 3D Gaussian in space, enabling real-time rendering. Additionally, adaptive density control endows them with the capability to express detailed features. These advantages make it widely applicable in 3D understanding and editing. Nonetheless, there is little research on 3D Gaussian segmentation, which is another critical pillar of the realm. A few Gaussian segmentation methods have been proposed recently, yet they require further improvement. For example, Gaussian Grouping [7] requires an extended training period of about 15 minutes. SAGA [8] is complex in its implementation and struggles with segmenting multiple objects simultaneously. Additionally, the explicit expression of 3D Gaussians leads to storage overhead, preventing it from directly transferring 2D semantic features into 3D, as in NeRF segmentation [9, 10]. Finally, the scarcity of datasets and the lack of annotations impede the application of supervised segmentation methods, commonly utilized in 2D and point cloud segmentation. In light of the aforementioned challenges, we propose leveraging a pre-trained 2D segmentation model to guide 3D Gaussian segmentation. Inspired by the 2D segmentation approach, which assigns a probability distribution vector for each pixel across different categories, we first assign an object code to each 3D Gaussian to indicate the Gaussian’s categorical probability distribution. Subsequently, we employ an algorithm that guides the classification of each 3D Gaussian by minimizing the error between the 2D segmentation map and the rendered segmentation map at a given pose. Finally, we employ KNN clustering to resolve semantic ambiguity in 3D Gaussians and statistical filtering to remove erroneously segmented 3D Gaussians. We validated the effectiveness of our approach through experiments in object-centric and 360° scenes. Our contributions can be summarized as follows. • We propose an efficient 3D Gaussian segmentation method supervised by 2D segmentation, which can learn the semantic information of a 3D scene in less than two minutes and segment multiple objects in 1-2 seconds for a given viewpoint. • Extensive experiments on LLFF, NeRF-360, and MipNeRF 360 have demonstrated the effectiveness of our method, obtaining an mIOU of 86%. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Authors: Authors: (1) Kun Lan, University of Science and Technology of China; (2) Haoran Li, University of Science and Technology of China; (3) Haolin Shi, University of Science and Technology of China; (4) Wenjun Wu, University of Science and Technology of China; (5) Yong Liao, University of Science and Technology of China; (6) Lin Wang, AI Thrust, HKUST(GZ); (7) Pengyuan Zhou, University of Science and Technology of China. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3. Method and 3.1. Point-Based rendering and Semantic Information Learning 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 3.2. Gaussian Clustering and 3.3. Gaussian Filtering 4. Experiment 4.1. Setups, 4.2. Result and 4.3. Ablations 4.1. Setups, 4.2. Result and 4.3. Ablations 5. Conclusion and References 5. Conclusion and References ABSTRACT Recently, 3D Gaussian, as an explicit 3D representation method, has demonstrated strong competitiveness over NeRF (Neural Radiance Fields) in terms of expressing complex scenes and training duration. These advantages signal a wide range of applications for 3D Gaussians in 3D understanding and editing. Meanwhile, the segmentation of 3D Gaussians is still in its infancy. The existing segmentation methods are not only cumbersome but also incapable of segmenting multiple objects simultaneously in a short amount of time. In response, this paper introduces a 3D Gaussian segmentation method implemented with 2D segmentation as supervision. This approach uses input 2D segmentation maps to guide the learning of the added 3D Gaussian semantic information, while nearest neighbor clustering and statistical filtering refine the segmentation results. Experiments show that our concise method can achieve comparable performances on mIOU and mAcc for multi-object segmentation as previous single-object segmentation methods. Index Terms— 3D Gaussian, 3D Segmentation Index Terms— Index Terms— 1. INTRODUCTION The recently emerged 3D Gaussian technique [1] marks a significant advancement over previous 3D representation methods such as point clouds [2], meshes [3], signed distance functions (SDF) [4], and neural radiance fields (NeRF) [5], especially in terms of training time and scene reconstruction quality. The mean of each 3D Gaussian represents the position of its center point, the covariance matrix indicates rotation and size, and spherical harmonics express color. Starting with point clouds obtained from SFM [6], 3D Gaussians inherently contain the scene’s geometric information, thus saving time in locating areas with concentrated objects in space. Moreover, their explicit expression method further accelerates calculations of color and density for every 3D Gaussian in space, enabling real-time rendering. Additionally, adaptive density control endows them with the capability to express detailed features. These advantages make it widely applicable in 3D understanding and editing. Nonetheless, there is little research on 3D Gaussian segmentation, which is another critical pillar of the realm. A few Gaussian segmentation methods have been proposed recently, yet they require further improvement. For example, Gaussian Grouping [7] requires an extended training period of about 15 minutes. SAGA [8] is complex in its implementation and struggles with segmenting multiple objects simultaneously. Additionally, the explicit expression of 3D Gaussians leads to storage overhead, preventing it from directly transferring 2D semantic features into 3D, as in NeRF segmentation [9, 10]. Finally, the scarcity of datasets and the lack of annotations impede the application of supervised segmentation methods, commonly utilized in 2D and point cloud segmentation. In light of the aforementioned challenges, we propose leveraging a pre-trained 2D segmentation model to guide 3D Gaussian segmentation. Inspired by the 2D segmentation approach, which assigns a probability distribution vector for each pixel across different categories, we first assign an object code to each 3D Gaussian to indicate the Gaussian’s categorical probability distribution. Subsequently, we employ an algorithm that guides the classification of each 3D Gaussian by minimizing the error between the 2D segmentation map and the rendered segmentation map at a given pose. Finally, we employ KNN clustering to resolve semantic ambiguity in 3D Gaussians and statistical filtering to remove erroneously segmented 3D Gaussians. We validated the effectiveness of our approach through experiments in object-centric and 360° scenes. Our contributions can be summarized as follows. • We propose an efficient 3D Gaussian segmentation method supervised by 2D segmentation, which can learn the semantic information of a 3D scene in less than two minutes and segment multiple objects in 1-2 seconds for a given viewpoint. • Extensive experiments on LLFF, NeRF-360, and MipNeRF 360 have demonstrated the effectiveness of our method, obtaining an mIOU of 86%. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Effortless 2D-Guided, 3D Gaussian Segmentation: Abstract and Introduction

Welcome to a comprehensive exploration of multi-object tracking. Given the complex nature of the environment, handling the uncertainties and fluctuations in a sensor's field of view has been a long-standing challenge in the field of sensor fusion and tracking. Traditional techniques have struggled to maintain an optimal balance between accuracy and computational efficiency. In this article, as an outcome, you can consider the PMBM filter as one of the most modern and high-performance algorithms in this domain. We will be unraveling its mathematical foundation and highlighting its practical implications. Whether you're an engineer, a researcher, a student, or simply a curious mind fascinated by the world of multi-object tracking, I hope to equip you with a solid understanding of the Poisson Multi-Bernoulli Mixture filter by the end of this article, thus enabling you to appreciate its remarkable contributions to the realm of multi-target tracking. Problem statement The primary task at hand revolves around multi-object tracking, a problem rooted in the complex and dynamic nature of real-world environments. This problem involves continuously estimating the states of multiple objects over time using sensor observations. The four primary challenges in this problem are clutter, object appearance and disappearance, the unknown number of objects, and sensor fusion. : Tracking can be complicated by 'clutter,' which refers to misleading sensor readings that are not associated with the objects of interest. This can be due to various factors such as environmental noise, sensor errors, or the presence of irrelevant objects. For instance, in radar tracking, clutter could be caused by reflections from buildings, terrain, or other non-target objects. In image processing, clutter can be introduced by reflections in mirrors, glass surfaces, or water bodies Clutter This occurs when an object of interest is not detected by the object detection system. For example, a camera might miss a person who briefly steps behind a pillar. Miss Detection: The spontaneous appearance or disappearance of objects within the sensor's field of view adds complexity to tracking. For instance, in a public square, people can enter or leave at any time. This dynamic nature, coupled with the uncertainty in the total number of objects, makes consistent and accurate multi-object tracking a challenging task." Dynamics and Uncertainty in Object Count: This refers to the complex task of merging data from multiple sensors, each with unique characteristics. For example, a self-driving car might need to integrate data from cameras, radars, and lidar, despite their potential inconsistencies and time delays. Sensor Fusion: The goal is to develop a robust multi-object tracking system that can accurately track multiple objects in cluttered environments, handle object dynamics, adapt to changing object counts, and effectively fuse data from various sensors. Models In this article, we focus on model-based methods. These algorithms are effective under the assumption that the processes involved can be accurately represented by a model. Sensor model In our simplified 2D model, we're examining an algorithm for a basic problem: tracking the 2D movement of an object starting from a known position, amidst significant sensor inaccuracies and clutter. Before we can start tracking, we first need to model the measurements, which includes modeling the clutter. Missdetections are modeled using the concept of detection probability (P_D). This parameter indicates the likelihood of an object being detected by the sensor. A P_D less than 1 implies a chance of missdetection. For example, a P_D of 0.9 suggests the object will be detected 90% of the time, and missed 10% of the time. class SensorModelConfig:
    def __init__(self, P_D, lambda_c, range_c):
        # P_D: object detection probability
        self.P_D = P_D
        # lambda_c: average number of clutter measurements per time scan, Poisson distributed
        self.lambda_c = lambda_c
        # range_c: range of surveillance area
        self.range_c = range_c
        # V: Volume of the surveillance area
        self.V = np.prod(np.diff(self.range_c))  
        # pdf_c: Spatial PDF of clutter
        self.pdf_c = 1 / self.V  
        # intensity_c: expected number of clutter detections per unit volume
        self.intensity_c = self.lambda_c / self.V  
    
    def handle_missdetection(self, objects):
        """Handles missdetection based on the detection probability."""
        # Generate a random number for each object
        random_numbers = np.random.uniform(size=len(objects))
        # Determine which objects are detected
        detected_objects = objects[random_numbers < self.P_D]
        return detected_objects

    def generate_clutter(self):
        """Generates clutter based on the clutter intensity."""
        # N_c: Number of clutter measurements, Poisson distributed
        N_c = np.random.poisson(lam=self.lambda_c)
        # Generate clutter within the surveillance area
        clutter = np.random.uniform(self.range_c[0], self.range_c[1], (N_c, 2))
        return clutter To model clutter, we assume it's uniformly distributed within a certain area. The quantity of clutter is modeled using a Poisson distribution, which represents the number of events (in this case, clutter) occurring in a fixed interval, given a constant mean rate. This aligns with our assumption of uniformly distributed clutter. The average number of clutter measurements per time scan is defined by the parameter . We randomly generate the number of clutter measurements, , based on this Poisson distribution. This provides us with the flexibility to model varying levels of clutter in different scenarios. lambda_c N_c Let s how it looks like for single object and P_G = 0.8, lambda_c = 0.5, range_c = [[-1000, 1000],[-1000, 1000]. Single object tracking We consider a state space model where we have one object and one measurement from the real object at each step. To represent our state, we adopt a Gaussian distribution. This choice is optimal under the assumption of normally distributed measurement noise, as it allows us to effectively estimate the object's true state. By incorporating motion and measurement models, we can predict future states based on previous observations. Assuming we know the initial position of the real object, we create a Gaussian distribution around it. As time progresses, we observe multiple measurements, some from the real object and others from clutter. To update the state, we employ the nearest measurement heuristic. However, since our state is represented by a Gaussian with mean and covariance, we need a way to measure the distance between the state and the measurements. The Mahalanobis distance, which takes into account the covariance matrix, provides a suitable solution. In scenarios where we have missdetections and can only rely on measurements from objects part of the time, we need a sensor model. We introduce a detection probability, denoted as P_D, to represent the likelihood of detecting the object. Additionally, we model the clutter by utilizing the Poisson distribution. The clutter rate, lambda_c, and the intensity_c, representing the expected number of clutter in one time interval, play crucial roles in this modeling approach. Returning to our problem, we calculate the weights of detection and missdetection as w_detection and w_missdetection, respectively. If w_detection is greater than w_missdetection, we update the state using the measurements. Otherwise, we return the state after the prediction step. This decision is based on a comparison between the weights, enabling us to make informed updates to the state estimate in the presence of missdetections. Object tracking with known number of objects The n-object tracking problem involves tracking multiple objects simultaneously. Unlike the previous single-object tracking problem where we worked with Gaussian distributions, in this case, we need to handle n Gaussians representing the multiple objects. At each step, we receive k measurements and consider missdetections and k association hypotheses for each Gaussian component. We calculate weights for each hypothesis using a similar approach to likelihood calculation. The challenge lies in finding the association between n Gaussians and k + n hypotheses. To solve this, we formulate the problem as a linear assignment problem. We construct a cost matrix of size n x (k + n) and assign a value of +inf to each field. We fill each field with negative likelihood values. Using the Hungarian algorithm, we minimize the sum of the costs to find the associations between Gaussians and hypotheses. While the Hungarian algorithm provides a greedy approach, it may not be suitable when dealing with unknown objects due to uncertainties. In such cases, alternative algorithms like Murty or Gibbs sampling can be used. These algorithms calculate the top k good solutions, taking into account the uncertain nature of the problem. Multi object tracking In the more complex scenario of multi-object tracking with unknown object appearances and disappearances, we employ the Poisson Multi-Bernoulli Mixture (PMBM) algorithm. This algorithm handles the uncertainty in the number of objects and their dynamic behavior. To begin, we introduce the concepts of the birth model and the death model within the framework of Random Finite Sets (RFS). The birth model can be represented by either a Bernoulli or a Poisson point process, while the death model incorporates a survival probability to represent the likelihood of an object continuing to exist in the next time step. In the case of a Bernoulli Random Finite Set, we describe the state of each object using Bernoulli components. These components consist of a Gaussian density representing the object's state and an existence probability (r) indicating the likelihood of the object's presence. Consider an example where we have one Bernoulli and receive a single measurement. This measurement generates two hypotheses: one assumes it is from a new object, while the other associates it with the existing Bernoulli (representing a detection). By extending this concept, if we have a set of Bernoullis, we create new Bernoullis for missdetections and associations for each existing Bernoulli, resulting in a Multi-Bernoulli Mixture. The PMBM algorithm allows us to consider uncertainty in associations, providing multiple possible outcomes. Each association represents a global hypothesis. We construct a cost matrix and aim to find the top m associations since the problem may not have a unique solution. The output of the tracker is the most probable hypothesis. For each global hypothesis, we repeat the same procedures in the next step, resulting in an exponential growth of global hypotheses. However, we can store only the most probable ones. The core of the PMBM filter relies on conjugate distributions. By parameterizing a distribution, the model remains unchanged after an update, and only the parameters are updated. This enables us to calculate these parameters and utilize them in subsequent steps. The PMBM algorithm involves several steps: Prediction Step: Predict the states using the motion model, including the prediction methods for both the Poisson Point Process and the Multi-Bernoulli Mixture. Update Step: Update the predicted states based on new measurements. Pruning Step: Remove unlikely hypotheses to manage computational complexity. Estimation Step: Generate estimates of the current state based on the updated and pruned hypotheses. By following these steps, the PMBM algorithm effectively handles the complexities of multi-object tracking with unknown object appearances and disappearances. References García-Fernández, Ángel F., Jason L. Williams, Karl Granström, and Lennart Svensson. 2017. “Poisson Multi-Bernoulli Mixture Filter: Direct Derivation and Implementation.” . arXiv. https://doi.org/10.1109/TAES.2018.2805153. arXiv [cs.CV] . Multiple Object Tracking Youtube Channel

Demystifying the Poisson Multi-Bernoulli Mixture Filter: A Leap Forward in Multi-Target Tracking

An Introduction to Variational Autoencoders Using Keras

Hello, today I'd like to explain briefly how we use artificial intelligence to count sunflower seeds in a photo taken with a mobile device. Agenda: 1. Business needs 2. Data preparation 3. Model structure 4. Used libs and tools 5. Results 6. Error analysis 7. Fails/Hypotheses 8. Conclusion 9. References 1. Business needs Fortunately for me, I am working at . Where I am developing Computer Vision (CV) and other models to solve business problems and challenges. One of them is to count seeds on sunflower. Kernel Kernel – the world’s leading and the largest in Ukraine producer and exporter of sunflower oil, and major supplier of agricultural products from the Black Sea region to world markets. Kernel exports its products into more than 80 countries of the world. It is a common task for agronomist to count seeds on sunflower and corn, as these calculations will be used to predict "biologycal harvest". Separately, agronomists calculate weight of 1000 kernels, which will further be used to estimate the overall yield per area of field. Moreover, yield prediction is the primary function of agro analytics. The faster and more precise they make a forecast, the more money the company will earn. These inspections should be carried out for every filed. And this is how it works now: 1. Agronomists go to the field 2. Take about 10 sunflowers from different parts of the field 3. Split every sunflower into 4 quarters 4. Count the seeds and multiply them by 4 to find the total number of seeds on every sunflower 5. Take random 1000 seeds and weight them 5. We get the density of plants on the field per hectare from other inspections (or use planned density), btw we also will use AI to solve this task, and this could be topic for next post 6. And calculate (density per ha * avg seed quantity on sunflower * weight of 1000) / 1000 = yield weight per ha Here is an example of a common sunflower photo White (unformed) and black (formed) seeds can be seen, however only black seeds will be used to produce sunflower oil. One disadvantage of this calculation is that agronomists must count black, formed seeds on sunflower, which is very inaccurate and inconvenient. Furthermore, each agronomist may count a different number of seeds on the same sunflower because some seeds may be unformed (white, non-pollinated by bees) and it is difficult to determine which kernel is formed and which is not. 2. Data preparation To solve any Machine Learning problem, we need to get data and train the model. And for every specific task you cant find any free available labeled dataset. In our case, the dataset should contain hundreds of sunflower photos with label for every seed related to right class (black or white, for easy understanding), so we gathered our own dataset. Our agronomists have made about 1000 photos of different sunflower hybrids from different regions of Ukraine. And subsequently, another team labeled photos in free tool for data labeling. There should be for each image, as well as , and CVAT bounding_box (purple box) black seeds (green points in the picture below) white seeds (red points in the example below). Following this, we can check distribution of kernels in sunflowers looks normal with mean value - 1271. And correlation between number of black and white seeds. Weak dependence exists, but it can not be used further. 3. Model structure For business needs, all calculations should be done on a mobile device. That's why I used with encoder pretrained on What does this all means? U-net [1] mobilenet_v2 [2] imagenet . is Neural Net structure where NN takes an input image and returns a segmentation map (in our case heatmap), with downsampling and upsampling data. Our heatmap represents probability that pixel is related to some kernel. U-net Where up/down sampling blocks used weights from mobilenet_v2 Imagenet is a dataset with millions of images and tens thousands of classes. Frankly speaking, for this task I trained 2 NN and 1 algorithm. The first NN cut the sunflower from the image, like this: The second NN build 2 heatmaps, one per class (black and white). For training, input for this net is cropped image (from previous NN), and target is heatmap with builded gaussians on labeled points. For black kernels it looks like this: According to this heatmap, the lighter the pixel, the more model is sure that pixel is related to kernel. And the last algorithm (Laplacian of Gaussian approach) searches for centers of probability blobs (groups of bright pixels that are adjacent to each other), and on next image I used green points as the blob center above the cropped image. With white kernels prediction looks like this: 4. Used libs and tools For training, I used Python libs such as: and for training models - for image augmentation - to use pretrained models - to work with blobs on heatmap - to tune hyperparams for blob detection - to track training experiments PyTorch PyTorch Lightning albumentations segmentation_models_pytorch scikit_image hyperopt wandb 5. Results How do we validate the whole solution? To calculate mean absolute percentage error (MAPE) I made prediction for every image and compared the number of found seeds by model and number of seeds found by assessors. As a result, we got normal distribution with mean 0, which is extremely good. Blue bars - are for black kernels and MAPE is 3.6%, Orange bars - are for white kernels and MAPE is 8.8%. 6. Error analysis The biggest errors will have unusual sunflowers, that have with abnormally large number of kernels, or too many grey kernels, or with rounded edges, as it can be seen in the example below: Here, the model found 16% more than the assessor, and as we can see, the majority of extra seeds are on the edge of the sunflower. Or biggest error -257% for white seeds in this case: The total number of white seeds is 14, and the model found 50. As a  percentage, it is a really big number, but in absolute terms, it is not critical. 7. Fails/Hypotheses Once, I have found mathematically explained seeds positions in sunflowers and some other plants with fractal structure. Positions could be explained by which has an equation in polar coordinates Fermat's spiral where is index number of kernel and - constant, I guess for every sunflower it is different. As result, we should get the following positions n c But in practice it is doesn't work. I have modified that equations and added new constants a, b and the best result that I got was this: where blue - dot is point. So it is close in some way but not applicable for to real case. (0, 0) 8. Conclusion Currently, this task has been speeded up and standardized in our company in order to exclude human factor from counting. The MAPE is acceptable for business needs and this solution is working in mobile application now. For me, it is still unbelievable that we can count more than 1000 kernels, separate points into 2 different classes and all of this can be done via mobile device. Many thanks to for mentor ship in learning and support. Markiyan Kostiv 9. Referances 1. (2015) 2. (2019) U-Net: Convolutional Networks for BiomedicalImage Segmentation MobileNetV2: Inverted Residuals and Linear Bottlenecks

Stories, News & Mentions about GAUSSLABS

Gausslabs’s Stories on HackerNoon