Table of links ABSTRACT 1 INTRODUCTION 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 3 PRELIMINARY EXPERIMENT 3.1 Experiment Setup 3.2 Observations 3.3 Summary and Challenges 4 OVERVIEW OF PANOPTICUS 5 MULTI-BRANCH OMNIDIRECTIONAL 3D OBJECT DETECTION 5.1 Model Design 6 SPATIAL-ADAPTIVE EXECUTION 6.1 Performance Prediction 5.2 Model Adaptation 6.2 Execution Scheduling 7 IMPLEMENTATION 8 EVALUATION 8.1 Testbed and Dataset 8.2 Experiment Setup 8.3 Performance 8.4 Robustness 8.5 Component Analysis 8.6 Overhead 9 RELATED WORK 10 DISCUSSION AND FUTURE WORK 11 CONCLUSION AND REFERENCES 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 3D object detection aims to identify objects in space and predict their properties such as 3D location, size, and velocity. The predicted object information is utilized by application functionalities such as obstacle avoidance for robot navigation. Safe navigation cannot be solely ensured by Simultaneous Localization and Mapping (SLAM), lacking the ability to model the object sizes or movements in real-time. A robot must plan its navigation path based on obstacles’ location and size, or even their predictive trajectory, to prevent collisions beforehand. Moreover, in complex outdoor environments where objects can approach from multiple directions, the ability to detect surrounding objects becomes essential. Existing methods for omnidirectional 3D object detection utilize LiDAR sensors or multiple cameras providing a 360° perception range. While LiDAR sensors offer accurate object localization based on depth measurements, the camera-based solutions have recently drawn attention due to their costeffectiveness. Recent camera-based detectors aggregate information from multiple camera images into a bird’s-eye-view (BEV) space, providing a top-down representation of the surrounding 3D space. Early work [38] proposed an end-toend trainable method to extract BEV features directly from multi-view images. Building upon [38], BEVDet [24] enables the detection of surrounding 3D objects using the extracted BEV features. Due to its simplified and scalable architecture, many of the latest BEV-based 3D detectors [23, 29, 30, 52] followed BEVDet’s inference pipeline, as shown in Figure 2. Such detectors have overcome the monocular ambiguity of camera-based approaches by introducing enhanced methods for each stage of the baseline BEVDet pipeline, achieving accuracy comparable to the LiDAR counterpart [54]. Table 1 lists these methods, which are described in the following. The first stage involves extracting 2D feature maps from multi-view images using backbone neural networks, such as ResNet [21], widely used in vision tasks. It is well-known that increasing the backbone capacity, e.g., the number of layers, or input image resolution leads to accuracy improvements. For example, combining a 152-layer ResNet with a 720×1,280 (height×width) resolution yields a more detailed feature map than a 34-layer ResNet with 256×448 resolution thereby enhancing the detection of small and distant objects. The second stage of BEV-based detection is to transform the extracted image features into 3D space using predicted depth. For this stage, prior work [30] employed a depth estimation neural network, i.e., DepthNet, supervised by dense depth data generated from 3D point clouds. Accurate metric depth predictions (in meters) from images allow the detector to better distinguish objects from the backgrounds. The third stage involves projecting features scattered in each 3D camera coordinate into a unified BEV grid using camera parameters, generating a BEV feature map. Recent works [23, 31] have proposed techniques that fuse the BEV feature map from a previous frame with the feature map of the current frame. By exploiting temporal cues, the technique improves perception robustness, enabling the detection of temporarily occluded objects and the accurate prediction of object velocities. Lastly, the neural networks in the BEV head generate 3D bounding boxes and their properties, e.g., location and velocity, using the BEV features. Authors:

Jeho Lee
Chanyoung Jung
Jiwon Kim
Hojung Cha This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. Table of links Table of links ABSTRACT ABSTRACT ABSTRACT 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 3 PRELIMINARY EXPERIMENT 3 PRELIMINARY EXPERIMENT 3 PRELIMINARY EXPERIMENT 3.1 Experiment Setup 3.2 Observations 3.3 Summary and Challenges 4 OVERVIEW OF PANOPTICUS 4 OVERVIEW OF PANOPTICUS 4 OVERVIEW OF PANOPTICUS 5 MULTI-BRANCH OMNIDIRECTIONAL 3D OBJECT DETECTION 5 MULTI-BRANCH OMNIDIRECTIONAL 3D OBJECT DETECTION 5 MULTI-BRANCH OMNIDIRECTIONAL 3D OBJECT DETECTION 5.1 Model Design 6 SPATIAL-ADAPTIVE EXECUTION 6 SPATIAL-ADAPTIVE EXECUTION 6 SPATIAL-ADAPTIVE EXECUTION 6.1 Performance Prediction 5.2 Model Adaptation 6.2 Execution Scheduling 7 IMPLEMENTATION 7 IMPLEMENTATION 7 IMPLEMENTATION 8 EVALUATION 8 EVALUATION 8 EVALUATION 8.1 Testbed and Dataset 8.2 Experiment Setup 8.3 Performance 8.4 Robustness 8.5 Component Analysis 8.6 Overhead 9 RELATED WORK 9 RELATED WORK 9 RELATED WORK 10 DISCUSSION AND FUTURE WORK 10 DISCUSSION AND FUTURE WORK 10 DISCUSSION AND FUTURE WORK 11 CONCLUSION AND REFERENCES 11 CONCLUSION AND REFERENCES 11 CONCLUSION AND REFERENCES 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 2 BACKGROUND: OMNIDIRECTIONAL 3D OBJECT DETECTION 3D object detection aims to identify objects in space and predict their properties such as 3D location, size, and velocity. The predicted object information is utilized by application functionalities such as obstacle avoidance for robot navigation. Safe navigation cannot be solely ensured by Simultaneous Localization and Mapping (SLAM), lacking the ability to model the object sizes or movements in real-time. A robot must plan its navigation path based on obstacles’ location and size, or even their predictive trajectory, to prevent collisions beforehand. Moreover, in complex outdoor environments where objects can approach from multiple directions, the ability to detect surrounding objects becomes essential. Existing methods for omnidirectional 3D object detection utilize LiDAR sensors or multiple cameras providing a 360° perception range. While LiDAR sensors offer accurate object localization based on depth measurements, the camera-based solutions have recently drawn attention due to their costeffectiveness. Recent camera-based detectors aggregate information from multiple camera images into a bird’s-eye-view (BEV) space, providing a top-down representation of the surrounding 3D space. Early work [38] proposed an end-toend trainable method to extract BEV features directly from multi-view images. Building upon [38], BEVDet [24] enables the detection of surrounding 3D objects using the extracted BEV features. Due to its simplified and scalable architecture, many of the latest BEV-based 3D detectors [23, 29, 30, 52] followed BEVDet’s inference pipeline, as shown in Figure 2. Such detectors have overcome the monocular ambiguity of camera-based approaches by introducing enhanced methods for each stage of the baseline BEVDet pipeline, achieving accuracy comparable to the LiDAR counterpart [54]. Table 1 lists these methods, which are described in the following. The first stage involves extracting 2D feature maps from multi-view images using backbone neural networks, such as ResNet [21], widely used in vision tasks. It is well-known that increasing the backbone capacity, e.g., the number of layers, or input image resolution leads to accuracy improvements. For example, combining a 152-layer ResNet with a 720×1,280 (height×width) resolution yields a more detailed feature map than a 34-layer ResNet with 256×448 resolution thereby enhancing the detection of small and distant objects. The second stage of BEV-based detection is to transform the extracted image features into 3D space using predicted depth. For this stage, prior work [30] employed a depth estimation neural network, i.e., DepthNet, supervised by dense depth data generated from 3D point clouds. Accurate metric depth predictions (in meters) from images allow the detector to better distinguish objects from the backgrounds. The third stage involves projecting features scattered in each 3D camera coordinate into a unified BEV grid using camera parameters, generating a BEV feature map. Recent works [23, 31] have proposed techniques that fuse the BEV feature map from a previous frame with the feature map of the current frame. By exploiting temporal cues, the technique improves perception robustness, enabling the detection of temporarily occluded objects and the accurate prediction of object velocities. Lastly, the neural networks in the BEV head generate 3D bounding boxes and their properties, e.g., location and velocity, using the BEV features. Authors: Jeho Lee
Chanyoung Jung
Jiwon Kim
Hojung Cha Authors: Jeho Lee Chanyoung Jung Jiwon Kim Hojung Cha Jeho Lee Jeho Lee Chanyoung Jung Chanyoung Jung Jiwon Kim Jiwon Kim Hojung Cha Hojung Cha This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Arweave

Optimizing Omnidirectional 3D Object Detection for Edge AI

This AI Model Gives Edge Devices Eyes on the Back of Their Heads

Omnidirectional Technology

Too Long; Didn't Read

Make resilience your competitive advantage

How 3D Object Detection Helps AI Avoid Obstacles

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Next-Gen 3D Detection: Challenges, Innovations & Future Directions

This AI Model Gives Edge Devices Eyes on the Back of Their Heads

Optimizing Omnidirectional 3D Object Detection for Edge AI

Efficient 3D Object Detection for Edge Devices

Optimizing AI for Real-Time Object Tracking

Next-Gen 3D Detection: Challenges, Innovations & Future Directions

This AI Model Gives Edge Devices Eyes on the Back of Their Heads

Optimizing Omnidirectional 3D Object Detection for Edge AI

Efficient 3D Object Detection for Edge Devices

Optimizing AI for Real-Time Object Tracking

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps