Lidar-Camera Calibration (single target and Targetless)
PreviousData Collection for Lidar-Camera Calibration (Single Target and Targetless)NextData Collection for Multi Target Lidar-Camera Calibration
Last updated
Last updated
This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.
Click on New Calibration to create a new calibration dataset.
Select LiDAR-Camera Calibration to create a new dataset.
Upon selecting LiDAR-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.
Users can choose either the target-based or the targetless calibration. The target-based calibration uses the checkerboard/charucoboard as the calibration target, and the targetless calibration uses the scene captured in both LiDAR and the camera sensor data.
Intrinsic parameters for the camera are to be added here. Users have three options.
Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.
Users can also load the JSON file.
Users can manually enter the intrinsic parameters if they already have them.
Horizontal corners: Total number of inner corners from left to right. The blue dots shown in the above preview correspond to the horizontal corners.
Vertical corners: Total number of inner corners from top to bottom. The red dots shown in the above preview correspond to the vertical corners.
Square size: This is the length of the square's arm in meters. It corresponds to the length of the yellow square highlighted in the preview.
Left padding: The distance from the leftmost side of the board to the left-most corner point in meters. Corresponds to the left blue line in the preview.
Right padding: The distance from the rightmost side of the board to the rightmost corner point in meters. Corresponds to the right blue line in the preview.
Top padding: The distance from the topmost side of the board to the topmost corner point in meters. Corresponds to the top red line in the preview.
Bottom padding: The distance from the bottom-most side of the board to the bottom-most corner point in meters. Corresponds to the bottom red line in the preview.
On ground: Enable this if the checkerboard is placed on the ground and the point cloud has the ground points in the scene around the checkerboard placement.
Tilted: Enable this if the checkerboard is tilted.
Rows: Total number of squares in the horizontal direction.
Columns: Total number of squares in the vertical direction.
Square size: It is the length of the arm of the square in meters.
Marker size: It is the length of the arm of the aruco marker in meters. This is usually 0.8 times the Square size.
Left padding: The distance from the board's left edge to the left of the first square in the row.
Right padding: The distance from the board's right edge to the right of the last square in the row.
Top padding: The distance from the board's bottom edge to the bottom of the last square in the column.
Bottom padding: The distance from the board's top edge to the top of the first square in the column.
On ground: Enable this if the checkerboard is placed on the ground and the point cloud has the ground points in the scene around the checkerboard placement.
Tilted: Enable this if the charcoboard is tilted.
Intrinsic parameters for the camera are to be added here. Users have three options.
Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.
Users can also load the JSON file.
Users can manually enter the intrinsic parameters if they already have them.
Add point cloud files from the LiDAR and images from the camera sensor. After adding, pair the point cloud files with the matching image files before continuing.
Users can click on Detect corners to detect the corners in the target. This is an automated process, and our algorithm usually detects the corners in the image accurately.
Suppose the target corners are not auto-detected. Users can follow the steps below and add the four boundary markers to get the inner checkerboard corners.
The extrinsic parameter space is vast, so we need an estimated entry point for optimization. The user can provide estimated extrinsic parameters in three ways.
Users can map the target corner points in the point cloud and get the initial estimates of the extrinsic parameters. Only one point cloud mapping is sufficient to get the initial estimates.
Our algorithms can automatically detect targets in the point cloud if the lidar channel data is provided on the configuration page. Please note that the auto-detection might not work properly if there are many flat surfaces, like walls, ceilings, etc., in the scene.
Add estimated extrinsic parameters
Users can manually enter estimated extrinsic parameters.
To get the initial estimates, users can map any four corresponding points from the image and the point cloud data.
Alternatively, users can add the initial estimates if they know them. In such a case, users can skip manually adding the markers. Users can click Add estimated extrinsic parameters to add the initial estimates.
Once the estimated extrinsic parameters are in the tool, users can visualize the parameters by clicking on the visualize button. In the visualization, we have a few sensor fusion techniques through which the accuracy of the extrinsic parameters can be visualized. For more details, visit Sensor fusion techniques.
Estimated extrinsic parameters are crucial in generating accurate extrinsic parameters.
To get good initial estimates, users must clear the markers and redo the markings if the estimated parameters are way off.
There are two types of segmentation approaches available for the user to select:
Auto segmentation: This approach automates the segmentation of vehicles in point clouds and images using a deep learning model trained on various datasets.
Manual segmentation:
Lidar: In this approach, the user needs to add bounding boxes in the lidar frame and fit the boxes to vehicles in the point cloud. The bounding boxes must be added for all the point clouds uploaded for calibration. This can be done by selecting the Bounding box mode, adding bounding boxes, and clicking Save Labels.
Image: There are two ways to do manual segmentation
Semantic Painting: Users can use the brush to paint the vehicles in the image and click on Save Labels.
Segment anything: In this approach, Users place a cluster of points on each vehicle. The same vehicle points should be placed under the same category. Please place at least one point on each surface of the car, such as the windshield, sides, roof, etc., so that when the model runs, it doesn't miss any part of the vehicle. After placing the points in each image, please click on the save label to save the data.
Note: Auto segmentation is suggested initially. Based on the segmented vehicles in the point clouds and images, the user can decide whether to proceed with auto-segmentation or perform the segmentation manually.
Users need to click on Calibrate to optimize the estimated extrinsic parameters further. All the uploaded pairs are used in the optimization process.
Deep Optimization: Users can select deep optimization to optimize the extrinsic further for datasets with the Tilted option enabled on the configuration page.
Max correspondence: This value is used as input for the algorithm. Users can tweak the value by analyzing the fused point cloud LiDAR files. Suppose the difference between the input and the generated cloud is more significant; the user can try to increase the value of the max correspondence and retry to improve the calibration results.
Users can optimize only angles by selecting the Angles only check box. It is observed that enabling Angles only results in achieving better Sensor angle accuracy (note that sensor position is not optimized in this case).
Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. The closer the error stats to zero, the better the extrinsic parameters.
Translation Error: Mean of difference between the centroid of points of checkerboard in the LiDAR and the projected corners in 3D from an image. Values are shown in meters. This calculation happens in the LiDAR coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.
Plane Translation Error: Mean of the Euclidean distance between the centroid of projected corners in 3D from an image and plane of the target in the LiDAR. Values are shown in meters. Note: If the board is only partially covered by the LiDAR or the LiDAR scan lines are non-uniformly distributed, translation and reprojection errors are inaccurate, but this plane translation error is accurate even in these scenarios.
Rotation Error: Mean difference between the normals of the target in the point cloud and the projected corners in 3D from an image. Values are shown in degree. This calculation happens in the LiDAR coordinate system. Note: All LiDARs have noise when measuring distance. This will, in turn, cause noise in the target's point clouds and the target's normals. Usually, this metric cannot measure accurately below 1 degree. For an accurate rotation error, we suggest using a faraway straight edge such as a building edge, roofline, or straight pole and projecting the point cloud onto the image. The rotation error can be calculated from the number of pixels between the image edges and the projected points.
Reprojection Error: Mean difference between the centroid of the target corners from the image and the centroid of the projected target from the LiDAR space onto the image. This is calculated in the image coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.
Individual error stats for each image/LiDAR pair can be seen. The average shows the mean of the errors of all the eligible image/LiDAR pairs.
Once the entire calibration is done, users can download all intrinsic and extrinsic parameters by clicking the Export button in the header.
Users can use the following techniques to visualize the extrinsic parameters.
Frustum: Users can see the image's field of view in the LiDAR frame. This uses both the camera matrix and the extrinsic parameters. Image axes are also displayed according to the extrinsic parameters.
LiDAR points in image: Users can see the LiDAR points projected in the camera image using extrinsic parameters.
color points from camera: Users can see the camera's color points in the lidar space using the extrinsic parameters.
Show target in LiDAR: Users can see the checkerboard points projected in the LiDAR frame using the extrinsic params.
The target in the image is filled with points. If the target configuration the user provides is correct, there will be no overflow or underflow.
This shows the extracted target from the original lidar file. We use this to calculate the error statistics. We compare the extracted target with the projected target.
Targets from all the point clouds are cropped and fused into a single point cloud.
Input cloud: This contains the fuse of all input clouds filtering the target area. If the target is not in the LiDAR file, the user has to fix the extrinsic parameters by going back to the mapping step or manually updating them.
Generated target: This contains the fuse of all generated targets. If the target is inaccurate, the user has to fix the target configuration or the inner corner detection.
Input and generated target: This contains the fused output of the Input cloud and Generated target. This helps us to analyze the difference between the input and the generated output before optimization.
Target begin vs after optimization: This helps to know the difference between the generated target, using the extrinsic values before and after the optimization step.
To verify the extrinsic parameters obtained from the calibration, we have an additional step that shows how close the final extrinsic values are to the actual extrinsic values of the setup.
Steps to be followed to validate ground truth:
Select the Validate Ground Truth option displayed at the top panel in the visualizer
From the image, select any edge of the board that will be used for error estimation
Draw a line that exactly matches the edge selected from the image, which is called the Ground Truth line
Draw another line joining the edge of the points that are projected from lidar onto the image (Projected Line)
After adding both lines to the image, click the Validate Ground Truth button in the right panel. This generates the ground truth Angle and Pixel errors.
roll, pitch, and yaw are in degrees, and px, py, and pz are in meters.
lidarPoint3D is the 3d coordinates of a point in the lidar coordinate system.
imagePoint3D is the 3d coordinates of a point in the camera coordinate system.
We currently show three different types of camera coordinate systems. The extrinsic parameters change according to the selected Camera coordinate system. The export option exports the extrinsic parameters based on the selected camera coordinate system.
Optical coordinate system: It's the default coordinate system that we follow.
ROS REP 103: This is the coordinate system followed by ROS. When you change to this, you can see the change in the visualization and the extrinsic parameters.
NED: This follows the North-East-Down coordinate system.
This is a sample Python script to project lidar points on an image using extrinsic parameters. It uses the open3d and opencv libraries.
