Lidar-Camera Calibration (single target and Targetless)

Calibration Homepage

  • This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

  • Click on New Calibration to create a new calibration dataset.

Calibration selection

Select LiDAR-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting LiDAR-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

Users can choose either the target-based or the targetless calibration. The target-based calibration uses the checkerboard/charucoboard as the calibration target, and the targetless calibration uses the scene captured in both LiDAR and the camera sensor data.

Configuration

Target-based

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

  • Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

  • Users can also load the JSON file.

  • Users can manually enter the intrinsic parameters if they already have them.

Checkerboard target configuration

  • Horizontal corners: Total number of inner corners from left to right. The blue dots shown in the above preview correspond to the horizontal corners.

  • Vertical corners: Total number of inner corners from top to bottom. The red dots shown in the above preview correspond to the vertical corners.

  • Square size: It is the length of the arm of the square in meters. The square size corresponds to the length of the yellow square highlighted in the preview.

  • Left padding: The distance from the leftmost side of the board to the left-most corner point in meters. Corresponds to the left blue line in the preview.

  • Right padding: The distance from the rightmost side of the board to the rightmost corner point in meters. Corresponds to the right blue line in the preview.

  • Top padding: The distance from the topmost side of the board to the topmost corner point in meters. Corresponds to the top red line in the preview.

  • Bottom padding: The distance from the bottom-most side of the board to the bottom-most corner point in meters. Corresponds to the bottom red line in the preview.

  • On ground: Enable this if the checkerboard is placed on the ground. Also, the point cloud has the ground points in the scene around the checkerboard placement.

  • Tilted: Enable this if the checkerboard is tilted.

Charucoboard target configuration

  • Rows: Total number of squares in the horizontal direction.

  • Columns: Total number of squares in the vertical direction.

  • Square size: It is the length of the arm of the square in meters.

  • Marker size: It is the length of the arm of the aruco marker in meters. This is usually 0.8 times the Square size.

  • Left padding: The distance from the board's left edge to the left of the first square in the row.

  • Right padding: The distance from the board's right edge to the right of the last square in the row.

  • Top padding: The distance from the board's bottom edge to the bottom of the last square in the column.

  • Bottom padding: The distance from the board's top edge to the top of the first square in the column.

  • On ground: Enable this if the charcoboard is placed on the ground. Also, the point cloud has the ground points in the scene around the charcoboard placement.

  • Tilted: Enable this if the charcoboard is tilted.

Targetless

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

  • Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

  • Users can also load the JSON file.

  • Users can manually enter the intrinsic parameters if they already have them.

Upload files from LiDAR and Camera

Add point cloud files from the LiDAR and images from the camera sensor. After adding, pair the point cloud files with the matching image files before continuing.

sample CSV format

X, Y, Z
0,-0,-0
62.545,-3.5064,-3.5911
62.07,-3.5133,-4.1565
32.773,-1.8602,-3.4055

Detect target corners in images (Only for target-based)

Users can click on Detect corners to detect the corners in the target. This is an automated process, and our algorithm usually detects the corners in the image accurately.

Suppose the target corners are not auto-detected. Users can follow the steps below and add the four boundary markers to get the inner checkerboard corners.

Estimated extrinsic parameters

Target-based

The extrinsic parameter space is huge, so we need an estimated entry point to optimize. There are three ways for the user to provide estimated extrinsic parameters.

Mapping of target corner points

Users can map the target corner points in the point cloud and get the initial estimates of the extrinsic parameters. Only one point cloud mapping is sufficient to get the initial estimates.

Auto detect target

Our algorithms can automatically detect targets in the point cloud if the lidar channel data is provided on the configuration page. Please note the auto-detection might not work properly if there are a lot of flat surfaces, like walls, ceilings, etc., in the scene.

Add estimated extrinsic parameters

Users can manually enter estimated extrinsic parameters.

Targetless calibration

Mapping of corresponding points

Users can map any four corresponding points from the image and the point cloud data to get the initial estimates.

Manually enter extrinsic parameters

Alternatively, users can add the initial estimates if they know them. In such a case, users can skip manually adding the markers. Users can click Add estimated extrinsic parameters to add the initial estimates.

Verifying the accuracy of the estimated extrinsic parameters

Once the estimated extrinsic parameters are in the tool, users can visualize the parameters by clicking on the visualize button. In the visualization, we have a few sensor fusion techniques through which the accuracy of the extrinsic parameters can be visualized. For more details, visit Sensor fusion techniques.

Estimated extrinsic parameters are crucial in generating accurate extrinsic parameters.

Users must clear the markers and redo the markings if the estimated parameters are way off to get good initial estimates.

Segmentation: (Only for Targetless Calibration)

There are two types of segmentation approaches available for the user to select:

  1. Auto segmentation: This approach performs automated segmentation of vehicles in point clouds and images using a deep learning model trained on various datasets.

  2. Manual segmentation: Lidar: In this approach, the user needs to add bounding boxes in the lidar frame and fit the boxes to vehicles in the point cloud. The bounding boxes must be added for all the point clouds uploaded for calibration. This can be done selecting bounding box in mode. Creating bounding boxes and clicking on save label

    Image: There are two ways to do manual segmentation 1. Semantic Painting: In this you can use a brush and paint the vehicles in the image and click on save labels to save label. 2. Segment anything: In this approach you place a cluster of points on each vehicle. Same Vehicle points should be placed under same category. Please place at least one point on each surface of the car such as windshield, sides, roof etc so that when the model runs it doesn't miss any part of the vehicle . (better segmentation = better results). after placing the points in each image please click on save label to save the data.

Note: It is suggested that auto segmentation be performed initially. Based on the segmented vehicles in the point clouds and images, the user can decide to go ahead with auto segmentation or perform the segmentation manually.

Run Calibration

Users need to click on Run calibration to optimize the estimated extrinsic parameters further. All the uploaded pairs are used in the optimization process.

Additional options in the run calibration

Target-based

Deep Optimization: Users can select deep optimization to optimize the extrinsic further for datasets with the Tilted option enabled on the configuration page.

Max correspondence: This value is used as an input to the algorithm. Users can tweak the value by analyzing the fused point cloud LiDAR files. If the difference between the input and the generated cloud is greater, the user can try to increase the value of the max correspondence and retry, improving the calibration results.

Targetless

Users can optimize only angles by selecting the Angles only check box. It is observed that enabling Angles only results in achieving better Sensor angle accuracy (note that sensor position is not optimized in this case).

Error stats (target based)

Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. The closer the error stats to zero, the better the extrinsic parameters.

  • Translation Error: Mean of difference between the centroid of points of checkerboard in the LiDAR and the projected corners in 3D from an image. Values are shown in meters. This calculation happens in the LiDAR coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

  • Plane Translation Error: Mean of the Euclidean distance between the centroid of projected corners in 3D from an image and plane of the target in the LiDAR. Values are shown in meters. Note: If the board is only partially covered by the LiDAR or the LiDAR scan lines are non-uniformly distributed, translation and reprojection errors are inaccurate, but this plane translation error is accurate even in these scenarios.

  • Rotation Error: Mean difference between the normals of the target in the point cloud and the projected corners in 3D from an image. Values are shown in degree. This calculation happens in the LiDAR coordinate system. Note: All LiDARs have noise in distance measurement. This will, in turn, cause noise in the target's point clouds and the target's normals. Usually, this metric cannot measure accurately below 1 degree. For an accurate rotation error, we suggest using a faraway straight edge such as a building edge, roofline, or straight pole and projecting the point cloud onto the image. The rotation error can be calculated from the number of pixels between the image edges and the projected points.

  • Reprojection Error: Mean difference between the centroid of the target corners from the image and the centroid of the projected target from the LiDAR space onto the image. This is calculated in the image coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

  • Individual error stats for each image/LiDAR pair can be seen. The average shows the mean of the errors of all the eligible image/LiDAR pairs.

Download calibration parameters

Once the entire calibration is done, users can download all intrinsic and extrinsic parameters by clicking the Export button in the header.

Analyzing the extrinsic parameters in Visualization Mode:

Sensor fusion techniques

Users can use the following techniques to visualize the extrinsic parameters.

Frustum: Users can see the field of view of the image in the LiDAR frame. This uses both the camera matrix and the extrinsic parameters. Image axes are also displayed according to the extrinsic parameters.

LiDAR points in image: Users can see the LiDAR points projected in the camera image using the extrinsic params.

color points from camera: Users can see the camera color points from the camera in the lidar space using the extrinsic params.

  • Show target in LiDAR: Users can see the checkerboard points projected in the LiDAR frame using the extrinsic params.

Image: Target Identification

The target in the image is filled with points. There will not be any overflow or underflow if the target configuration provided by the user is correct.

LiDAR: Extracted target:

This shows the extracted target from the original lidar file. This is used for the error stats calculation. We compare the extracted target with the projected target.

Fused Point Cloud:

Targets from all the point clouds are cropped and fused into a single point cloud.

  • Input cloud: This contains the fuse of all input clouds filtering the target area. If the target is not in the LiDAR file, the user has to fix the extrinsic parameters by going back to the mapping step or manually updating the extrinsic parameters.

  • Generated target: This contains the fuse of all generated targets. If the target is inaccurate, the user has to fix the target configuration or the inner corner detection.

  • Input and generated target: This contains the fused output of the Input cloud and Generated target. This helps us to analyze the difference between the input and the generated output before optimization.

  • Target begin vs after optimization: This helps to know the difference between the generated target, using the extrinsic values before and after the optimization step.

Validate Ground Truth:

To verify the extrinsics obtained from the calibration, we have an additional step that gives how close the final extrinsic values are to the actual extrinsics of the setup.

Steps to be followed to validate ground truth:

  1. Select the Validate Ground Truth option displayed at the top panel in the visualizer

  2. From the image select any edge of the board that will be used for error estimation

  3. Draw a line that exactly matches the with edge selected from the image which is called the Ground Truth line

  1. Draw another line which is the line joining the edge of the points that are projected from lidar onto the image (Projected Line)

  1. After Adding both the lines in the image, click on the Validate Ground Truth button in the right panel which generates the ground truth Angle Error and Pixel Error.

Extrinsic Calibration Output.

  • roll, pitch, and yaw are in degrees, and px, py, and pz are in meters.

  • lidarPoint3D is the 3d coordinates of a point in the lidar coordinate system.

  • imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Camera coordinates system

We currently show three different types of camera coordinate systems. The extrinsic parameters change according to the selected Camera coordinate system. The export option exports the extrinsic parameters based on the selected camera coordinate system.

  • Optical coordinate system: It's the default coordinate system that we follow.

  • ROS REP 103: It is the coordinate system followed by ROS. On changing to this, you can see the change in the visualization and the extrinsic parameters.

  • NED: This follows the north-east-down coordinate system.

Sample Script

This is a sample Python script to project lidar points on an image using extrinsic parameters. It uses the open3d and opencv libraries.

Tool usage guide for old UX

Last updated