How I Made Autonomous Vehicle AI 40% Faster Without Losing Accuracy

Autonomous vehicles need to see the world fast and with precision. But real-time perception is tough when your model is drowning in millions of LiDAR points and high-res images. The solution? Sensor fusion, model quantization, and a sprinkle of TensorRT acceleration. Let me walk you through how I optimized a multi-modal perception model to run 40% faster while keeping it sharp enough to dodge pedestrians and parked cars. The Bottleneck: A Million Points, A Million Problems Autonomous vehicles rely on LiDAR and cameras, each with strengths and weaknesses. LiDAR excels at depth estimation, but it spits out millions of points per second. Cameras provide rich textures and colors but lack depth. Fusing both in a single pipeline means balancing speed and precision. Problem 1: The Lag Between LiDAR and Camera Data By the time LiDAR finishes scanning, the camera has already captured a new frame. This misalignment means object positions don’t match. Fix: I used ego-motion compensation with Kalman filters to adjust LiDAR points dynamically. Here’s a snippet of how it works in Python: import numpy as np class EgoMotionCompensation: def __init__(self, initial_pose, process_noise, measurement_noise): self.state = initial_pose self.P = np.eye(5) * 0.01 self.Q = np.diag(process_noise) self.R = np.diag(measurement_noise) def predict(self, dt): # Unpack state x, y, yaw, vx, vy = self.state x_pred = x + vx * dt y_pred = y + vy * dt yaw_pred = yaw self.state = np.array([x_pred, y_pred, yaw_pred, vx, vy]) F = np.eye(5) F[0, 3] = dt F[1, 4] = dt self.P = F @ self.P @ F.T + self.Q def update(self, measurement): #measurement = [x_meas, y_meas, yaw_meas] H = np.array([ [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0] ]) z = np.array(measurement) y_err = z - H @ self.state S = H @ self.P @ H.T + self.R K = self.P @ H.T @ np.linalg.inv(S) self.state += K @ y_err I = np.eye(5) self.P = (I - K @ H) @ self.P def apply_correction(self, lidar_points, dt): #predict and update using IMU/odometry self.predict(dt) #Suppose we have a measurement from the IMU or odometry #assume partial measurement [x_meas, y_meas, yaw_meas] measurement = [self.state[0], self.state[1], self.state[2]] self.update(measurement) x, y, yaw, _, _ = self.state cos_yaw, sin_yaw = np.cos(yaw), np.sin(yaw) transformed = [] for px, py, pz in lidar_points: #rotate and translate x_new = (px * cos_yaw - py * sin_yaw) + x y_new = (px * sin_yaw + py * cos_yaw) + y transformed.append((x_new, y_new, pz)) return np.array(transformed) This tweak reduced misalignment by 85%, boosting object detection accuracy. Problem 2: Sensor Fusion: Making LiDAR and Cameras Play Nice Now that we fixed the time lag, we have another issue: LiDAR gives point clouds, while cameras provide RGB images. We needed a single representation for both. We built a Bird’s Eye View (BEV) model that projects LiDAR points onto the image plane and aligns them with camera pixels. How We Did It: Calibrated LiDAR and camera with intrinsic & extrinsic matrices. Projected LiDAR points into the image space. Fused features using a cross-attention transformer. Here’s how the projection works in Python: import torch import torch.nn.functional as F import cv2 import numpy as np def voxelize_points(lidar_points, voxel_size=(0.1, 0.1, 0.2)): #Transform LiDAR points into 3D voxel indices coords = np.floor(lidar_points / np.array(voxel_size)).astype(int) coords -= coords.min(0) voxels = np.zeros((coords[:,0].max()+1, coords[:,1].max()+1, coords[:,2].max()+1)) for (x_idx, y_idx, z_idx) in coords: voxels[x_idx, y_idx, z_idx] += 1 return voxels def project_voxel_to_image(voxel_coords, camera_matrix, rvec, tvec, voxel_size): """ Projects 3D voxel centers into image space using a known camera matrix (including distortion). """ #compute center for each voxel voxel_centers = [] for (x_idx, y_idx, z_idx) in voxel_coords: center_x = x_idx * voxel_size[0] center_y = y_idx * voxel_size[1] center_z = z_idx * voxel_size[2] voxel_centers.append([center_x, center_y, center_z]) voxel_centers = np.array(voxel_centers, dtype=np.float32).reshape(-1, 1, 3) #project to image plane img_pts, _ = cv2.projectPoints(voxel_centers, rvec, tvec, camera_matrix, None) return img_pts def fuse_features(camera_features, lidar_features): """ cross-attention: queries come from camera_features, keys/values from lidar_features, to refine camera representation based on 3D data. """ #camera_features, lidar_features are [Batch, Channels, Height, Width] fused = torch.cat([camera_features, lidar_features], dim=1) fused = F.relu(F.conv2d(fused, torch.randn(128, fused.size(1), 3, 3))) return fused This improved detection by 29%, helping the model understand its surroundings better. Slashing Inference Time by 40% With TensorRT and Quantization Even with better fusion, inference time was killing us. Running deep learning models on embedded hardware isn’t the same as cloud GPUs. We needed to shrink the model without losing accuracy. The Solution: TensorRT + Quantization Converted PyTorch to TensorRT: TensorRT optimizes computation graphs, fuses layers, and reduces memory overhead. Applied INT8 Quantization: This compressed 32-bit floating points into 8-bit representations making computations 4x faster. Quantization technique that was used: import torch from torch.ao.quantization import ( get_default_qconfig_mapping, prepare_fx, convert_fx ) def apply_quantization(model, calibration_data): model.eval() #Use a QConfigMapping that sets INT8 or mixed precision qconfig_mapping = get_default_qconfig_mapping("fbgemm") example_inputs = (calibration_data[0].unsqueeze(0), calibration_data[1].unsqueeze(0)) prepared_model = prepare_fx(model, qconfig_mapping, example_inputs) #Run calibration with torch.no_grad(): for data in calibration_data: fused_input = data[0].unsqueeze(0), data[1].unsqueeze(0) prepared_model(*fused_input) quantized_model = convert_fx(prepared_model) return quantized_model An example of how I converted the PyTorch model: import tensorrt as trt import os def build_tensorrt_engine(onnx_file_path, precision="int8", max_batch_size=4, input_shape=(3, 256, 256)): trt_logger = trt.Logger(trt.Logger.INFO) builder = trt.Builder(trt_logger) network_flags = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) network = builder.create_network(flags=network_flags) parser = trt.OnnxParser(network, trt_logger) if not os.path.exists(onnx_file_path): raise FileNotFoundError(f"ONNX file not found: {onnx_file_path}") with open(onnx_file_path, "rb") as f: if not parser.parse(f.read()): for i in range(parser.num_errors): print(parser.get_error(i)) raise ValueError("Failed to parse the ONNX file.") config = builder.create_builder_config() if precision == "int8": config.set_flag(trt.BuilderFlag.INT8) elif precision == "fp16": config.set_flag(trt.BuilderFlag.FP16) profile = builder.create_optimization_profile() #define min/opt/max batch sizes min_shape = (1,) + input_shape opt_shape = (max_batch_size // 2,) + input_shape max_shape = (max_batch_size,) + input_shape input_name = network.get_input(0).name profile.set_shape(input_name, min_shape, opt_shape, max_shape) config.add_optimization_profile(profile) engine = builder.build_engine(network, config) return engine Results: Inference time dropped from 250ms → 75ms per frame. Memory footprint reduced by 40%. Real-time performance improved to 5 FPS. Next Steps: End-to-End Learning We’re still exploring end-to-end models, where perception and planning happen in one deep learning pipeline. The idea? Instead of multiple modules for object detection, lane estimation, and path planning, we let a neural network learn everything from scratch. Some early results show promise but training needs massive datasets and interpretability is a challenge. Still, we’re inching toward fully AI-driven navigation. Final Thoughts: AI That Sees and Thinks Fast By blending sensor fusion, ego-motion correction, and TensorRT acceleration, we made autonomous AI faster and smarter. The next step? Scaling it beyond research and getting it on the road. If you’re working on AV models, let’s connect, I’d love to hear how you’re tackling inference speed and perception challenges. Further Reading: NVIDIA TensorRT for High-Performance Inference Multi-Modal Fusion in Autonomous Vehicles End-to-End Learning for Self-Driving Autonomous vehicles need to see the world fast and with precision. But real-time perception is tough when your model is drowning in millions of LiDAR points and high-res images. The solution? Sensor fusion, model quantization, and a sprinkle of TensorRT acceleration. Autonomous vehicles need to see the world fast and with precision. Sensor fusion, model quantization, and a sprinkle of TensorRT acceleration. Let me walk you through how I optimized a multi-modal perception model to run 40% faster while keeping it sharp enough to dodge pedestrians and parked cars. multi-modal perception model The Bottleneck: A Million Points, A Million Problems The Bottleneck: A Million Points, A Million Problems Autonomous vehicles rely on LiDAR and cameras , each with strengths and weaknesses. LiDAR excels at depth estimation , but it spits out millions of points per second . Cameras provide rich textures and colors but lack depth. Fusing both in a single pipeline means balancing speed and precision . LiDAR and cameras depth estimation millions of points per second rich textures and colors speed and precision Problem 1: The Lag Between LiDAR and Camera Data Problem 1: The Lag Between LiDAR and Camera Data By the time LiDAR finishes scanning, the camera has already captured a new frame. This misalignment means object positions don’t match. Fix: I used ego-motion compensation with Kalman filters to adjust LiDAR points dynamically. Here’s a snippet of how it works in Python: Fix: ego-motion compensation Kalman filters import numpy as np class EgoMotionCompensation: def __init__(self, initial_pose, process_noise, measurement_noise): self.state = initial_pose self.P = np.eye(5) * 0.01 self.Q = np.diag(process_noise) self.R = np.diag(measurement_noise) def predict(self, dt): # Unpack state x, y, yaw, vx, vy = self.state x_pred = x + vx * dt y_pred = y + vy * dt yaw_pred = yaw self.state = np.array([x_pred, y_pred, yaw_pred, vx, vy]) F = np.eye(5) F[0, 3] = dt F[1, 4] = dt self.P = F @ self.P @ F.T + self.Q def update(self, measurement): #measurement = [x_meas, y_meas, yaw_meas] H = np.array([ [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0] ]) z = np.array(measurement) y_err = z - H @ self.state S = H @ self.P @ H.T + self.R K = self.P @ H.T @ np.linalg.inv(S) self.state += K @ y_err I = np.eye(5) self.P = (I - K @ H) @ self.P def apply_correction(self, lidar_points, dt): #predict and update using IMU/odometry self.predict(dt) #Suppose we have a measurement from the IMU or odometry #assume partial measurement [x_meas, y_meas, yaw_meas] measurement = [self.state[0], self.state[1], self.state[2]] self.update(measurement) x, y, yaw, _, _ = self.state cos_yaw, sin_yaw = np.cos(yaw), np.sin(yaw) transformed = [] for px, py, pz in lidar_points: #rotate and translate x_new = (px * cos_yaw - py * sin_yaw) + x y_new = (px * sin_yaw + py * cos_yaw) + y transformed.append((x_new, y_new, pz)) return np.array(transformed) import numpy as np class EgoMotionCompensation: def __init__(self, initial_pose, process_noise, measurement_noise): self.state = initial_pose self.P = np.eye(5) * 0.01 self.Q = np.diag(process_noise) self.R = np.diag(measurement_noise) def predict(self, dt): # Unpack state x, y, yaw, vx, vy = self.state x_pred = x + vx * dt y_pred = y + vy * dt yaw_pred = yaw self.state = np.array([x_pred, y_pred, yaw_pred, vx, vy]) F = np.eye(5) F[0, 3] = dt F[1, 4] = dt self.P = F @ self.P @ F.T + self.Q def update(self, measurement): #measurement = [x_meas, y_meas, yaw_meas] H = np.array([ [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0] ]) z = np.array(measurement) y_err = z - H @ self.state S = H @ self.P @ H.T + self.R K = self.P @ H.T @ np.linalg.inv(S) self.state += K @ y_err I = np.eye(5) self.P = (I - K @ H) @ self.P def apply_correction(self, lidar_points, dt): #predict and update using IMU/odometry self.predict(dt) #Suppose we have a measurement from the IMU or odometry #assume partial measurement [x_meas, y_meas, yaw_meas] measurement = [self.state[0], self.state[1], self.state[2]] self.update(measurement) x, y, yaw, _, _ = self.state cos_yaw, sin_yaw = np.cos(yaw), np.sin(yaw) transformed = [] for px, py, pz in lidar_points: #rotate and translate x_new = (px * cos_yaw - py * sin_yaw) + x y_new = (px * sin_yaw + py * cos_yaw) + y transformed.append((x_new, y_new, pz)) return np.array(transformed) This tweak reduced misalignment by 85% , boosting object detection accuracy. reduced misalignment by 85% Problem 2: Sensor Fusion: Making LiDAR and Cameras Play Nice Problem 2: Sensor Fusion: Making LiDAR and Cameras Play Nice Now that we fixed the time lag, we have another issue: LiDAR gives point clouds , while cameras provide RGB images . We needed a single representation for both. point clouds RGB images single representation We built a Bird’s Eye View (BEV) model that projects LiDAR points onto the image plane and aligns them with camera pixels. Bird’s Eye View (BEV) model LiDAR points onto the image plane How We Did It: How We Did It: Calibrated LiDAR and camera with intrinsic & extrinsic matrices. Projected LiDAR points into the image space. Fused features using a cross-attention transformer. Calibrated LiDAR and camera with intrinsic & extrinsic matrices. Calibrated LiDAR and camera Projected LiDAR points into the image space. Projected LiDAR points Fused features using a cross-attention transformer. Fused features Here’s how the projection works in Python: import torch import torch.nn.functional as F import cv2 import numpy as np def voxelize_points(lidar_points, voxel_size=(0.1, 0.1, 0.2)): #Transform LiDAR points into 3D voxel indices coords = np.floor(lidar_points / np.array(voxel_size)).astype(int) coords -= coords.min(0) voxels = np.zeros((coords[:,0].max()+1, coords[:,1].max()+1, coords[:,2].max()+1)) for (x_idx, y_idx, z_idx) in coords: voxels[x_idx, y_idx, z_idx] += 1 return voxels def project_voxel_to_image(voxel_coords, camera_matrix, rvec, tvec, voxel_size): """ Projects 3D voxel centers into image space using a known camera matrix (including distortion). """ #compute center for each voxel voxel_centers = [] for (x_idx, y_idx, z_idx) in voxel_coords: center_x = x_idx * voxel_size[0] center_y = y_idx * voxel_size[1] center_z = z_idx * voxel_size[2] voxel_centers.append([center_x, center_y, center_z]) voxel_centers = np.array(voxel_centers, dtype=np.float32).reshape(-1, 1, 3) #project to image plane img_pts, _ = cv2.projectPoints(voxel_centers, rvec, tvec, camera_matrix, None) return img_pts def fuse_features(camera_features, lidar_features): """ cross-attention: queries come from camera_features, keys/values from lidar_features, to refine camera representation based on 3D data. """ #camera_features, lidar_features are [Batch, Channels, Height, Width] fused = torch.cat([camera_features, lidar_features], dim=1) fused = F.relu(F.conv2d(fused, torch.randn(128, fused.size(1), 3, 3))) return fused import torch import torch.nn.functional as F import cv2 import numpy as np def voxelize_points(lidar_points, voxel_size=(0.1, 0.1, 0.2)): #Transform LiDAR points into 3D voxel indices coords = np.floor(lidar_points / np.array(voxel_size)).astype(int) coords -= coords.min(0) voxels = np.zeros((coords[:,0].max()+1, coords[:,1].max()+1, coords[:,2].max()+1)) for (x_idx, y_idx, z_idx) in coords: voxels[x_idx, y_idx, z_idx] += 1 return voxels def project_voxel_to_image(voxel_coords, camera_matrix, rvec, tvec, voxel_size): """ Projects 3D voxel centers into image space using a known camera matrix (including distortion). """ #compute center for each voxel voxel_centers = [] for (x_idx, y_idx, z_idx) in voxel_coords: center_x = x_idx * voxel_size[0] center_y = y_idx * voxel_size[1] center_z = z_idx * voxel_size[2] voxel_centers.append([center_x, center_y, center_z]) voxel_centers = np.array(voxel_centers, dtype=np.float32).reshape(-1, 1, 3) #project to image plane img_pts, _ = cv2.projectPoints(voxel_centers, rvec, tvec, camera_matrix, None) return img_pts def fuse_features(camera_features, lidar_features): """ cross-attention: queries come from camera_features, keys/values from lidar_features, to refine camera representation based on 3D data. """ #camera_features, lidar_features are [Batch, Channels, Height, Width] fused = torch.cat([camera_features, lidar_features], dim=1) fused = F.relu(F.conv2d(fused, torch.randn(128, fused.size(1), 3, 3))) return fused This improved detection by 29% , helping the model understand its surroundings better. improved detection by 29% Slashing Inference Time by 40% With TensorRT and Quantization Slashing Inference Time by 40% With TensorRT and Quantization Even with better fusion, inference time was killing us. Running deep learning models on embedded hardware isn’t the same as cloud GPUs. We needed to shrink the model without losing accuracy . shrink the model without losing accuracy The Solution: TensorRT + Quantization The Solution: TensorRT + Quantization Converted PyTorch to TensorRT: TensorRT optimizes computation graphs, fuses layers, and reduces memory overhead. Applied INT8 Quantization: This compressed 32-bit floating points into 8-bit representations making computations 4x faster. Converted PyTorch to TensorRT: TensorRT optimizes computation graphs, fuses layers, and reduces memory overhead. Converted PyTorch to TensorRT : TensorRT optimizes computation graphs, fuses layers, and reduces memory overhead. Converted PyTorch to TensorRT Applied INT8 Quantization: This compressed 32-bit floating points into 8-bit representations making computations 4x faster. Applied INT8 Quantization : This compressed 32-bit floating points into 8-bit representations making computations 4x faster . Applied INT8 Quantization 8-bit representations 4x faster Quantization technique that was used: import torch from torch.ao.quantization import ( get_default_qconfig_mapping, prepare_fx, convert_fx ) def apply_quantization(model, calibration_data): model.eval() #Use a QConfigMapping that sets INT8 or mixed precision qconfig_mapping = get_default_qconfig_mapping("fbgemm") example_inputs = (calibration_data[0].unsqueeze(0), calibration_data[1].unsqueeze(0)) prepared_model = prepare_fx(model, qconfig_mapping, example_inputs) #Run calibration with torch.no_grad(): for data in calibration_data: fused_input = data[0].unsqueeze(0), data[1].unsqueeze(0) prepared_model(*fused_input) quantized_model = convert_fx(prepared_model) return quantized_model import torch from torch.ao.quantization import ( get_default_qconfig_mapping, prepare_fx, convert_fx ) def apply_quantization(model, calibration_data): model.eval() #Use a QConfigMapping that sets INT8 or mixed precision qconfig_mapping = get_default_qconfig_mapping("fbgemm") example_inputs = (calibration_data[0].unsqueeze(0), calibration_data[1].unsqueeze(0)) prepared_model = prepare_fx(model, qconfig_mapping, example_inputs) #Run calibration with torch.no_grad(): for data in calibration_data: fused_input = data[0].unsqueeze(0), data[1].unsqueeze(0) prepared_model(*fused_input) quantized_model = convert_fx(prepared_model) return quantized_model An example of how I converted the PyTorch model: import tensorrt as trt import os def build_tensorrt_engine(onnx_file_path, precision="int8", max_batch_size=4, input_shape=(3, 256, 256)): trt_logger = trt.Logger(trt.Logger.INFO) builder = trt.Builder(trt_logger) network_flags = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) network = builder.create_network(flags=network_flags) parser = trt.OnnxParser(network, trt_logger) if not os.path.exists(onnx_file_path): raise FileNotFoundError(f"ONNX file not found: {onnx_file_path}") with open(onnx_file_path, "rb") as f: if not parser.parse(f.read()): for i in range(parser.num_errors): print(parser.get_error(i)) raise ValueError("Failed to parse the ONNX file.") config = builder.create_builder_config() if precision == "int8": config.set_flag(trt.BuilderFlag.INT8) elif precision == "fp16": config.set_flag(trt.BuilderFlag.FP16) profile = builder.create_optimization_profile() #define min/opt/max batch sizes min_shape = (1,) + input_shape opt_shape = (max_batch_size // 2,) + input_shape max_shape = (max_batch_size,) + input_shape input_name = network.get_input(0).name profile.set_shape(input_name, min_shape, opt_shape, max_shape) config.add_optimization_profile(profile) engine = builder.build_engine(network, config) return engine import tensorrt as trt import os def build_tensorrt_engine(onnx_file_path, precision="int8", max_batch_size=4, input_shape=(3, 256, 256)): trt_logger = trt.Logger(trt.Logger.INFO) builder = trt.Builder(trt_logger) network_flags = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) network = builder.create_network(flags=network_flags) parser = trt.OnnxParser(network, trt_logger) if not os.path.exists(onnx_file_path): raise FileNotFoundError(f"ONNX file not found: {onnx_file_path}") with open(onnx_file_path, "rb") as f: if not parser.parse(f.read()): for i in range(parser.num_errors): print(parser.get_error(i)) raise ValueError("Failed to parse the ONNX file.") config = builder.create_builder_config() if precision == "int8": config.set_flag(trt.BuilderFlag.INT8) elif precision == "fp16": config.set_flag(trt.BuilderFlag.FP16) profile = builder.create_optimization_profile() #define min/opt/max batch sizes min_shape = (1,) + input_shape opt_shape = (max_batch_size // 2,) + input_shape max_shape = (max_batch_size,) + input_shape input_name = network.get_input(0).name profile.set_shape(input_name, min_shape, opt_shape, max_shape) config.add_optimization_profile(profile) engine = builder.build_engine(network, config) return engine Results: Results: Inference time dropped from 250ms → 75ms per frame. Memory footprint reduced by 40%. Real-time performance improved to 5 FPS. Inference time dropped from 250ms → 75ms per frame. Memory footprint reduced by 40%. Real-time performance improved to 5 FPS. Next Steps: End-to-End Learning Next Steps: End-to-End Learning We’re still exploring end-to-end models , where perception and planning happen in one deep learning pipeline . The idea? Instead of multiple modules for object detection, lane estimation, and path planning , we let a neural network learn everything from scratch . end-to-end models perception and planning happen in one deep learning pipeline object detection, lane estimation, and path planning learn everything from scratch Some early results show promise but training needs massive datasets and interpretability is a challenge . Still, we’re inching toward fully AI-driven navigation. training needs massive datasets interpretability is a challenge fully AI-driven navigation. Final Thoughts: AI That Sees and Thinks Fast Final Thoughts: AI That Sees and Thinks Fast By blending sensor fusion, ego-motion correction, and TensorRT acceleration , we made autonomous AI faster and smarter . sensor fusion, ego-motion correction, and TensorRT acceleration faster and smarter The next step? Scaling it beyond research and getting it on the road . If you’re working on AV models, let’s connect, I’d love to hear how you’re tackling inference speed and perception challenges. Scaling it beyond research getting it on the road Further Reading: Further Reading: NVIDIA TensorRT for High-Performance Inference Multi-Modal Fusion in Autonomous Vehicles End-to-End Learning for Self-Driving NVIDIA TensorRT for High-Performance Inference NVIDIA TensorRT for High-Performance Inference Multi-Modal Fusion in Autonomous Vehicles Multi-Modal Fusion in Autonomous Vehicles End-to-End Learning for Self-Driving End-to-End Learning for Self-Driving