What is Deepen AI Suite? Deepen AI is the fastest way to annotate data to build and ship artificial intelligence applications. Deepen AI Suite provides enterprise-grade AI-accelerated tools focusing on image, video. and sensor fusion (camera, LiDAR, radar, and more) training data. Who is it for?

1. Believe commercial solutions will be a far better choice than building and maintaining internal tools and infrastructure.

2. Want the highest quality standards for creating training data.

3. Want to standardize creating and managing training data across various teams within the organization.

4. Want to leverage labeling automation and reduce costs in human labeling.

5. Want the ability to work with internal labeling teams (usually experts) as well as fully managed external labeling service.

6. Want to leverage state of the art labeling tools for creating training data.

Overview: Share your labeled dataset on your own site

Steps to embed link:

1. To enter the embed labeling screen, go to “Share” on the dataset page.

2. Enter the title and number of frames you wish to display. A unique embedding link wrapped around <iframe> tag will be automatically generated.

3. Paste the link in your HTML or JavaScript editor.

General FAQ:

1. What is Deepen AI? Deepen is an industry-leading computer vision training data solution with fast AI enabled labeling tools, labeling automation, human workforce, and data management.

2. Does Deepen only sell labeling services? Deepen sells both AI-enabled labeling tools and labeling services, which includes: -An image and video annotation tool and services -Multi-sensor (Lidar, Radar, Sonar, Camera and more) fusion annotation tool and services -Quality and performance management tools.

3. How do I set up a Deepen project? Once you have created a Deepen account, follow the simple steps on this page to set up a project.

4. Do I need to be an AI engineer to use Deepen tools? No, we designed our user-friendly platform to require minimal technical knowledge and expertise such that anyone can annotate data using our out-of-the-box web application. However, it may be beneficial to have someone experienced in data science on your team to help ensure that your machine learning project is successfully developed and deployed.

5. What is the difference between Deepen 2D and Deepen 3D? Deepen 2D includes annotation capability for camera data such as images and videos. Deepen 3D supports multi-data annotation which includes the camera, LiDAR, radar, and more.

6. Which internet browser should I use? Google Chrome will be recommended when using Deepen tools.

Our data management workflow is designed to help managers manage a team of annotators in a seamless manner, yet produce high-quality results. It includes features such as task assignments on a user level, multi-stage quality assurance down to each label, and exporting your labeled data with just a click.

Overview: Creating a group allows you to create a group within a workspace.

Creating a Group: Users can create a group by clicking on the create group on the Users page and add other users to it.

User Permissions: Permissions can be changed at the datasets levels for different Users by using user permissions.

User Group: Permissions can be set for a group of users by selecting a group and every user has the same permissions for the selected datasets.

How to enable auto-task assignments for datasets?

Users with admin permission can go to the tasks screen in the dataset details page, and turn on the auto-task assignment feature using the Auto-assign option. Users can also set an assignment config for each stage by clicking on the menu settings icon beside the Auto-assign option.

How does this work? When auto-assign is enabled. Users having labelling permission can go to the tasks page at the workspace level and send a request for work by clicking on the Request work button. once the user requests work, tasks will be assigned to the user.

Overview: Users can export Labels/Dataset/Profile using the export button.

2D Label Types: 2D Bounding Boxes 2D Polyline/Line 2D Polygon 2D Points 2D Semantic Painting 3D Bounding boxes on images

3D Label Types: 3D Bounding Boxes 3D Lines 3D Polygons 3D Semantic Segmentation 3D Instance Painting

Overview: Tasks can be assigned in both labeling and configured pipeline stages for added users. Steps for Task Assignments: You can assign the tasks to the user in the dataset level under “Tasks”. 1. Click on Manage tasks for assignments. 2. Here we can assign the files to both the labeling and QA and other configured pipeline stages.

3. Admin needs to select the user email from the drop-down boxes and select the files from the unassigned files or enter the range(E.g: 1-4,5,6-11). 4. Labelling users can check the assigned files from the editor by clicking on Your work at the centre of the editor.

5.There are some filters by which the user can sort based on their preferences.

6. Users can move to the assigned frames/labels in the editor by clicking on the next task.

Overview: Users can add different type categories and add attributes at label and category levels.

Steps to create a dataset profile:

1. Click on the “dataset profile” at the left panel.

2. Create a profile name e.g.: My Image profile.

3. When configuring categories, choose a label type. Then, create a category by typing the name and click enter the box below.

4. For 3D bounding boxes, you can configure your default bounding box size by clicking on the category.

5. You can also add the label attributes specific to the category when you select a category.

6. Common attributes are attributes that apply to all categories.

7. Frame Attributes are attributes that apply at a frame level.

8. Dataset Attributes are attributes that apply at the dataset level.

9. Auto-task assignments can be enabled by clicking on Enable auto-task assignments.

10. Next, enter the labeling instructions (optional).

11. Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of pipelines as you want.

12. In the Image Editor user can give the attributes to the labeled objects by pressing shortcut keys or from the tool itself.

Overview: Users can export labels using export labels by using different methods. Steps to export data:

1. Labels that are done can be downloaded from the tool in JSON format.

2. Only labels which are marked as done are exported as JSON files.

3. Labels: Labels in the last pipeline stages will be downloaded. (Files should be marked as done)

4. Labels by file names: Download labels from the last pipeline stage for all label types, with the file name as the key in JSON format.

5. Customize your export: Files can also be downloaded by selecting particular frames, particular annotations, categories, attributes, types etc. (Files need not be marked as done if labels are downloaded using these).

6. 2D Semantic labels can be downloaded by clicking on the 2D Semantic Segmentation option.
7. Video for downloading paint labels https://drive.google.com/file/d/1EUzTqbRlyW85ZMgOJS74jCPTXk70vhH0/view?usp=sharing

Note: 2D Semantic segmentation output can be found here

Overview: Polyline or Line annotations are continuous lines made of one or more line segments. The entire line is treated as one object. Polylines are typically used to annotate open shapes since they don’t have to be closed as polygons do. This tool is often used to label shapes like road lane marks or other similar shapes.

Steps to Draw a Line:

1. Click on Label type and select Line from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a line keeping points on the image with a left mouse click. 4. You can draw another line by pressing the “Esc” key twice. 5. You can select the label by clicking on the label with a left mouse click. 6. Double-click on the line to edit the selected 2D-Line, by dragging the highlighted points of the line with a left mouse click, and also you can delete the selected highlighted point by pressing the “Delete” key. 7. To add a point on the line, click on the line using the left mouse click. 8. Labels are saved automatically. 9. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: Access token is created for admin to access different APIs in your specific workspace.

How to create an access token:

1. Go to the left panel, under "Developer settings" click on "Access Token".

2. Click on "Generate new token" on the top right of the screen to start generating a token.

3. Fill in your note and select the scope of users allowed to access the token.

4. Click on "Generate token" to create token access.

5. This token will not expire, you have to revoke the token access manually.

How to view/revoke created token:

1. Once the token is created, go to "Developer Tokens" to view the token.

2. You can click on the token to copy it and click on the "revoke" button to revoke the token manually.

Overview: A scene label is a behaviour or an event or an action by an object or a scene in the sequence.

Steps to enable scene labeling:

1. To start enabling scene labeling, upload a dataset as a sequence datasets or video that is not split into frames. By checking the "use video directly instead of splitting into frames" option, it will not the video into frames and rather upload the entire video.

2. Create a category under scene labeling. Ex: cut in

3. Clicking on the New Scene Label button at the bottom of the editor creates a new label from the user's current frame with the selected category. Ex: cut in:1

Steps to create a scene label:

1. Click on the label name or track selects the label and will enable the "Start Scene" and "End Scene" buttons.

2. Navigating to the desired frame and clicking on the "End Scene" buttons will extend the track from the already existing start point to the selected frame.

3. Upon selecting the frame number below the already existing frame and click on the "Start Scene" button extends the track to the left towards the user's new start frame.

4. Click on label or track selects the labels which enable the user to click on the edges of the track and drag towards the user's desired frame.

The Deepen reports offer a complete view of the dataset management environment. They track important aspects such as data quality, user contributions, labeling progress, and real-time activity within the workspace. This information is vital for making informed decisions and optimizing the workflow for dataset-related tasks.

Dataset Reports:

The dataset report provides information about the datasets in the workspace. It includes details such as the number of frames, labels, time spent on the dataset, average time across all stages, and completed tasks.

User Reports:

The user report gives information about the users working on the dataset. It provides details such as the total number of labels created and the average daily time spent.

Quality Reports:

The Quality report relates to issues that arise within the dataset, providing information on the status of each issue.

Labelling Reports:

The Labelling report provides information about the labeling categories, label type, label count, and stage ID.

Real-Time Reports:

The Real-time report provides real-time updates about the current workspace. It allows users to track the number of users who are working on the workspace.

Productivity Reports:

The productivity report provides information about the user's productivity across all label types and stages.

Overview: Import Labels allows users to import existing dataset labels from a dataset to another dataset in JSON and DPN(3D Segmentation) format for both 2D and 3D.

• Users can find the import labels feature beside the launch button(If customer review is enabled, it is beside Launch review) on the dataset page.

• Once import labels is selected, the user can select the format and click on Import Labels. Labels will be uploaded in a few minutes based on the file size.

• Users can select Import into label set if they want to import labels into a different Label Set.

• If web paint is selected, a metadata file should also be uploaded in JSON format. The segmentation format can be found at Data Output Format - 3D Semantic Segmentation

Overview: Painting foreground and background allows you to segment an object occluded in the background

Steps to segment foreground or background:

1. Knowing how to use the foreground and background are very useful in data labeling.

2. Start labeling by using left-click.

3. When we label the 1st object which was occluded by another 2nd object. It should be labeled with an extra region of the object.

4. By using the foreground option for the 2nd object which was in front of the 1st object. It will cut the 1st object polygon accurately.

5. By using the background option we can label an occluded object without affecting the polygon in front of the occluded objects

6. Click “control + shift” to draw a Polygon within a polygon.

Overview: Importing a dataset configuration via JSON

1. Go to the Datasets tab, find the import profile feature at the top right under the 3 dot dropdown

2. Click on import profile, there you can click on import profile from JSON and select a JSON file.

3. Customize what you wish to import

4. Once you've confirmed, click on "import config". All the configurations in the dataset will then be changed according to the JSON file you've chosen"

How to add a user:

1. When a workspace is opened there is the Users option on the left side panel.

2. Click on the Users option and at the top right click on Add user.

3. Enter valid email and select the scope for the user and click on Add User.

Permission rights of each user: Deepen AI annotation tools have different user access levels for different tasks. Different roles and their description are as follows:

1. Admin: This role can view/add/edit/delete a project, add/delete a dataset, add users, add payment details, sign up for different pricing plans, assign users, add files and add labels.

2. Reader: This role can view all the labels, reports and profiles but cannot make any changes.

3. Datasets Admin: This role has access to create and delete datasets. They can assign frames to already added users. They cannot label or add any users.

4. Datasets Creator: This role has access to add datasets, but cannot delete datasets.

5. Labels Admin: This role can add and delete labels.

6. Labeller: This role has access to annotate the allocated frames (Labelling and QA)

7. Customer: This role has permission to accept or reject labels in the workspace.

Apart from these, we have additional scopes for Access Tokens. They are:

1. Dataset Labels Admin: This role can add and delete the labels on the dataset level only.

2. Dataset Labeller: This role has access to annotate the allocated frames(Labelling and QA) on the dataset level.

3. Dataset Calibration Edit: This role can view, create, manage and delete calibration datasets at the dataset level.

4. Dataset Calibration Read: This role can view all at the dataset level for calibration dataset.

5. Dataset Reader: This role can view all the labels, reports and profiles but cannot make any changes at the dataset level.

6. Dataset Customer: This role can accept labels at the dataset level.

Overview: Import Profile allows you to import an existing dataset configuration from a dataset to another dataset

You can find the import label feature at the top right under the 3 dot dropdown in a dataset.

Steps for Importing Labels:

1. Click on Select Profile and select an existing profile that is already created.

2. Click Next and Customise profile.

3. Once the above step is done click on Next: Finish and Click on Import config.

4. You can view your imported labels in the label set you selected on the labeling screen.

Users can now stream data from their cloud buckets directly into the app without having to upload it. Whatever state of files users have in the bucket is automatically synced with the corresponding files in users datasets. With this now, users can also add or delete files from datasets by simply adding/deleting them in your bucket.

Users can create a Data stream by clicking on create Data stream button on the data streams page.

Images and Point clouds(JSON Format) are supported as of now, and we are working on adding more file type supports.

User needs to choose which cloud provider they need and then provide few details which are bucket name(bucket where your data is stored), folder name(location of the folder that will be synced to create a dataset present within the bucket), credential file(a json file with the private key of one user that can enable us to authenticate to your bucket location) and optionally user can provide a dataset configuration profile which will be applied to all the datasets that are created using this data stream configuration.

Once a data stream configuration is created, you can click on the "Sync bucket" option to sync your datasets with your bucket instantly. Post that you can use this dataset like normal datasets and it has all the native functionalities that are already supported by Datasets.

Assessments are designed to automatically evaluate the proficiency of users, generate a score, and produce a comprehensive report. At present, we provide support for automatic evaluation of 2D bounding boxes. However, we are committed to expanding our support for additional label types soon. To create an assessment, follow the below steps:

1. Open the datasets page and click on “More”.
2. Click on “Assessments” and then “New” to create a new assessment.
3. Enter the assessment name, attendees, passmark, IoU matching threshold, time limit, and maximum attempts.
4. Select a benchmark dataset that has labels and is marked as “Done”.
5. Once the assessment is created, Users can launch the dataset and start the assessment.
6. Annotate the bounding boxes and submit the assessment once all the labels are annotated.
7. You can go back to the assessment page to view the results. If you fail the assessment, you can reattempt it.
8. To compare the assessment results with the benchmark dataset, the assessment creator must click on the view results button.
9. The results show missing objects and false positives.

We're introducing an improved lifecycle for the tasks in the dataset to streamline your dataset workflow experience. Here are the new task states we support:

Waiting: Indicates the task is in waiting state and needs to be submitted from a previous stage

Ready: Indicates the task is ready for you to start working on

In progress: Indicates the user is working on the task

Submitted: Indicates the task was submitted for review, and the next stage user has to verify and accept it. Please note that you will not be able to modify a task once submitted.

Accepted: Indicates the task was verified by the next stage user and has approved it and no changes would be further required in it. Please note that you will not be able to reject a task once accepted.

Rejected: Indicates the task was rejected by the next stage user and needs to be fixed (users can refer to issues if any were created for the task before rejecting the task). Rejected tasks need re-work and have to be submitted again for further review.

Note: Once a task is submitted user can't edit the task in that stage.

Task options you see for different possibilities:

• If you are in the first stage of the pipeline, you see Submit.

• If you are in a stage after the first one (e.g. second, third)

  • If the previous stage was not accepted yet, you see Accept & modify, Accept & submit and Reject

  • Once you've accepted the previous stage task, you see Submit.

• If you are in the last stage of the pipeline

  • If the previous stage was not accepted yet, you see Accept & modify, Accept & finish and the Reject

  • Once you've accepted the previous stage task, you see Finish.

New tasks page inside a dataset

Users with admin scopes and customers will now have a new tasks screen visualizing the flow of the work in the dataset across all stages. Only users with dataset admin or above scopes will be able to assign users to the tasks.

When clicked on a task in the Task column (left-most column), the task is launched in the editor with all the required options set automatically, so you can start working without worrying about which stage you have to go to or which label set you to need to set to.

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to draw a Bounding Box:

1. Click on Label type and select the Bounding box from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a bounding box by clicking on the image with the left mouse click and Drag the 2D-box up to the corners of the object.

4. You can select the label by clicking on the label with a left mouse click. Then you can edit the 2D-Box, by dragging the highlighted points of the box with a left mouse click.

5. Labels are saved automatically.

6. Click “control + shift” to draw a bounding box within a box.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

Overview: Users can now perform instance segmentation tasks in polygon mode for single-frame 2D image datasets with AI assistance. In order to use the feature, the user needs to follow the steps below.

Creating the Segmentation Mask

• Go to Polygon mode

• Choose the “2D Segmentation” button

• Draw a tight box enclosing the object to be segmented

• AI server will return an output polygon mask covering the object

Refining the Segmentation Mask

Users have an option to edit/refine the masks using the pre-existing tools in the polygon labelling mode or they can use the AI assistance to refine the polygons automatically. To use AI assistance, the user needs to follow the steps below:

• To activate AI refinement mode, select the label which was created using the “2D Segmentation” button.

• Choose the “Refine Polygons” button (pencil) besides the “2D Segmentation” button.

• In the refinement mode, the user can plot two kinds of points as below.

  • Green indicates the regions which the user wants the model to include in the segmentation mask.

  • Red indicates the regions which the user wants the model to exclude from the segmentation mask.

• The user can plot Green points on the image using “Left Mouse Click”.

• Points To add the Red points, the user can use the “Left Mouse Click” with “Alt” hotkey.

• As soon as any of the above points are added, an API call will be made to refine the points.

• The segmentation will be refined using the input provided and the polygon will be updated accordingly. The refinement mode will be deactivated once the polygon is refined. You need to activate the refinement mode again using the “Refine Polygons” button in the secondary menu bar to make further edits.

• To add multiple points at once before refining the polygon, the user can use the "Shift" hotkey along with the process described above. The last point in this scenario needs to be added with “Shift” hotkey released for the polygon to be refined.

• The user can delete existing points using “Ctrl” + “Shift” hotkeys along with "Left Mouse Click" on the point to be deleted.

Semantic Segmentation

Overview: Semantic segmentation is the task of classifying each and every pixel in an image into a class or category. 1. Select an instance or semantic segmentation by clicking “Label” at the top panel

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the semantic if the objects are touching each other than their drawn polygons are merged with each other with a single id. 4. In the below image, you can see the merged polygons with a single id

Instance Segmentation:

Overview: Identify each object instance of each pixel for every known object within an image.

1. Select an instance or semantic segmentation by clicking “Label” at the top panel.

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the Instance segmentation, each object can be labelled distinctly with a unique ID. 4. In the image below, you can see the object’s polygons with different IDs like Car1, Car2 and Car3.

Overview: Create, configure, and upload your dataset to your workspace.

Steps to create a dataset:

1. In your workspace, click on “Create dataset” in the top right corner.

2. Enter a dataset name (use alphanumeric characters only. Special characters are not allowed).

3. Select a dataset type (Images/Video/3D Point Cloud).

4. Select Dataset Format (JSON) when the 3D point cloud is selected.

5. Select Frames per second when Video (only .mp4 files are supported) is selected from the Dataset type.

6. Select a labeling mode (Individual/Sequence).

7. Select a dataset (Only Zip files are allowed) by clicking on browse from the computer, or drag and drop the dataset.

8. Import dataset configuration from profile (if any profile is already created).

9. Categories and Attributes can be configured if there is no Dataset Profile

10. Enter Labelling Instructions if required.

11. You can also manage your team by assigning them certain permissions like Labeller, Labels Admin, or Reader for Local Users or Local Groups.

12. Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of the pipeline as you want.

13. Enable 2D Tools in 3D Editor (only for 3D datasets) to label 2D annotations for the 3D datasets.

14. Users can enable customer review/Enable auto-task assignments while creating the dataset if needed.

15. Add a new tag or select an existing tag to find the dataset easily.

16. Click on Create to create a dataset.

17. Once created, you can see the dataset with the “Pending upload” status and the file count after finishing the upload process.

18. Refresh the browser once the data is uploaded.

Here is the 3D input format for uploading a dataset:

Overview: The Polygon tool is used to create geometric annotations by drawing lines between points.

Steps to Draw a Polygon

1. Click on Label type and select Polygon from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a polygon, keeping points on the image with a left mouse click. You can label the same polygon ID by holding an “Alt” key for the first polygon point or by selecting a label and clicking on Draw selected polygon. Then continue the polygon labeling for the object with the same ID.

4. You can select the label by clicking on the label with a left mouse click. Double-click on a polygon to edit it by dragging the highlighted points of the line with a left mouse click and also you can delete the selected highlighted point by pressing the “Delete” key. To add a point on the line, click on the line using the left mouse click.

5. You can draw another line by pressing the “Esc” key twice. 6.Labels are saved automatically. 7. Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

Polygon (Foreground/Background): Knowing how to use the foreground and background can come in handy in data labelling for polygons. If there is an object at the back of another object, that is when this feature comes into play. Draw a polygon as Foreground:

1. Selecting a foreground option from the image editor page

2. For example, if you want to label a “Car” in the below image without interrupting the “Building” polygon, you want to select a Foreground option before saving the “Car” label as the Building is the Background and car is the foreground object.

3. If you use Background instead of Foreground option, it will not be cut by the Background polygon label(Building).

Draw a polygon as Background:

1. Selecting a Background option from the image editor page.

2. For example, if you want to label a building in the below image without interrupting the car polygon, you want to select a background option before saving the building label because the Building is the Background and the car is the foreground object.

3. When you label a background object with a background option, it will get a polygon accurately to the object without disturbing the foreground label (Car). 4. If you use the foreground instead of the background option, it will interrupt the foreground polygon label (Car).

Segment Anything Model (SAM): A new AI model that can “cut out” any object, in any image, with a single click.

Steps to perform Segment Anything in semantic segmentation:

1. Click on Label type and select Semantic Painting from the top left menu of the editor.
2. Click on Paint and select Paint using Segment Anything.
3. Users can label an object by selecting either Points or Box. The more points that are added, the higher the accuracy of the output. For better accuracy, it is advisable to place more points in the center of the object.
4. After selecting the point/box, click on the Generate label to preview the results. The user can either accept or reject the label.
5. Click on Paint and select Save to save all the changes made. It is important to note that for Semantic painting, users must manually label by clicking on Save.

6. Users can overwrite the painted region by selecting Overwrite painted region.

7. Users can unselect the Paint using the Segment Anything option and make edits using brush/polygon.

User can Hide/Unhide the lidar points from top and bottom in the editor.

Overview: Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: Click on Paint at the top panel and select 'Paint with Polygon' or use right-click at the type selection. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Following are some shortcut keys that can be used in 2D Editor. These can be found from the editor under the Help⇾Keyboard shortcuts menu or on the bottom right of the editor.

We Support 7 different dictionaries. 4x4, 5x5, 6x6 and 7x7 can have a maximum of 1000 markers, The original dictionary supports upto 1024 markers, apriltag_36h11 can have upto 587 markers, and apriltag_25h9 can have upto 35 markers.

• Original

• 4X4

• 5x5

• 6x6

• 7x7

• apriltag_36h11

• apriltag_25h9

How to choose the bits of charuco dictionary?

The more the number of markers/corners, the better the accuracy of your calibration, but your camera should be able to identify your board correctly . So consider the camera's resolution, the target's distance from the camera, and the environment around you while choosing the dictionary to get a clear image and more corners.

while having multiple target avoid using 4x4 as it has been observed to provide ghost points due to its relatively simple pattern.

Labels for 3D instance segmentation can be downloaded as combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The dpn file is a binary file which contains point level information for label cateogry and label id. For a dataset with n number of points, first n bytes of dpn file represent the label category, the next n bytes represent the lower bytes of the label id and the subsequent n bytes represent the higher bytes of the label id the point belongs to.

For example, consider a dataset containing 3 files each with 10,000 points, 11,000 points and 9000 points respectively. Assuming that the point with index 50 in second file of the dataset is annotated as below :

label_category : "paint_category_2" label_id : 1024

For the above scenario, dpn file will contain 90000 bytes ((10,000 + 11,000 + 9000) * 3). The annotation information for the point will be present at 10,050th (10,000 + 50), 40,050th (30,000 + 10,050) and 70,050th (60,000 + 10,050) indexes. The values of bytes at these indexes are as below:

10050 : 2 (1 based index of "paint_category_2" in metadata) 40050 : 0 (1024 % (2^8)) 70050 : 4 (1024 / (2^8)) The value 0 for label category is reserved for unpainted points.

Overview: By using 2D Editor users can annotate 2D Bounding Boxes,2D Polygons, 2D Lines, 2D Points, 3D bounding boxes and Semantic painting for the camera images.

Primary Toolbar: It has Home, Select and Move Tool(V), Pan Tool(2), Select and Move, Zoom In, Zoom Out and Create Issue.

• Home Button: Home button is used to navigate to the Homepage.

• Select and move tools(V): Users can move or edit labels

• Pan(2): This Option is used for moving images. Users can’t label images if this option is selected.

• Zoom In: Users can Zoom in by selecting this option.

• Zoom Out: Users can Zoom Out by selecting this option.

• Create Issue: Users can create issues at the frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar. More information on issues can be found here.

• The Top toolbar consists of Frame, Label, View and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete. Polygon annotation additionally has Attributes, Polygon Segmentation Mode and Polygon Drawing Mode.

View: View consists of Canvas view, Show Labels Text, Show Labels Attributes, Grid Size, Show Grid, and Show Merge Options for Polygons.

Secondary Menu bar: It consists of Label Set, Stage, Label type, Point size, Labelling Instructions, View Editor, View Original Image, View Masked Image(only for Polygons) and Reset View.

Panel Content: It is on the right of the image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these it also has Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular file.If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on the workspace the tool navigates the user to the workspace page in the tool.

Buttons in the lower navigation bar and their usage:

• Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.

• Preload Data: It loads the frames in advance before opening them.

• User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

• User can hide the bottom bar using the Hide bottom bar option for sequence annotations.

Overview: Labels are generated automatically based on the image, this is also known as pre-labeling.

Steps for using AI Sense: 1.Go to the Editor Page and Click on the AI Sense button from the right side menu. 2.Click on Auto Detect. 3.Click on Preview to view the labels or Add All to add the labels. 4.Users can add individual labels or use Confidence Score to detect labels. 5.Make sure the category is given before using AI Sense.

Categories supported for AI Sense: Single Frame Detection categories:(2D Bounding Boxes)

1. person: A person walking in the scene (usually on sidewalks, crosswalks, outside driveable space

2. bicycle: Human or electric-powered 2-wheeled vehicle designed to travel at lower speeds either on the edge of the road surface, sidewalks, or bike paths. A driving bicycle is also considered a bicycle.

3. car: All Sedan, Suv, minivans, and sports cars are marked as cars

4. motorcycle: Gasoline or electric-powered 2-wheeled vehicle designed to move rapidly (at the speed of standard cars) on the road surface.

5. airplane

6. bus: A road vehicle designed to carry many passengers.

7. train

8. truck: Motor vehicles designed to transport cargo, goods, merchandise, and a wide variety of objects.

9. boat

10. traffic light: A road signal for directing vehicular traffic by means of coloured lights, typically red for stop, green for go, and yellow for proceeding with caution.

11. fire hydrant: a fitting in a street or other public place with a nozzle by which a fire hose may be attached to the water main.

12. stop sign: A stop sign is a traffic sign designed to notify drivers that they must come to a complete stop

13. parking meter

14. bench

15. bird

16. cat

17. dog

18. horse

19. sheep

20. cow

21. elephant

22. bear

23. zebra

24. giraffe

25. backpack

26. umbrella

27. handbag

28. tie

29. suitcase

30. frisbee

31. skis

32. snowboard

33. sports ball

34. kite

35. baseball bat

36. baseball glove

37. skateboard

38. surfboard

39. tennis racket

40. bottle

41. wine glass

42. cup

43. fork

44. knife

45. spoon

46. bowl

47. banana

48. apple

49. sandwich

50. orange

51. broccoli

52. carrot

53. hot dog

54. pizza

55. donut

56. cake

57. chair

58. couch

59. potted plant

60. bed

61. dining table

62. toilet

63. tv

64. laptop

65. mouse

66. remote

67. keyboard

68. cell phone

69. microwave

70. oven

71. toaster

72. sink

73. refrigerator

74. book

75. clock

76. vase

77. scissors

78. teddy bear

79. hair drier

80. toothbrush

Sequence tracking model categories:(2D Bounding Boxes)

1. car

2. truck

3. person

4. bus

5. bike

6. rider: A person driving a bicycle or motorcycle is considered as the rider

7. motor

8. train

Categories for 3d Bounding box (Sequence and Single Frame):

1. vehicle: All Sedan, Suv, minivans, and sports cars are marked as cars

2. pedestrian: A person walking in the scene(usually on sidewalks, crosswalks, outside driveable space)

3. cyclist: A Person who is riding a Cycle

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

1. Click on Label type and select Semantic painting from the top left menu of the editor.

2. Select a category/class from the dropdown.

3. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon) to start painting.

4. Selecting Brush/Polygon from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon”.

5. Use mouse left click to paint the image.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You need to save by clicking on Paint at the top panel or click on save or use Enable Autosave.

8. To erase, click on the eraser icon to erase the paint.

Note: The output file can be downloaded by clicking Export ⇾ Labels ⇾ 2D Semantic Segmentation.

Overview: One-click bounding box allows you to create a perfectly labeled bounding box with just one click.

Steps to create a one-click bounding box:

1. To enable one-click bounding boxes, click on “label” at the top panel and select “Enable one click box placement”. Make sure you’re in the bounding box mode.

2. Once selected, place the bounding box at a dynamic object. Ex: Vehicle. Placing at concentrated points will yield better results.

3. Look at the secondary view to check the quality of the placement. Adjust the label accordingly if needed.

Overview: Instance Painting allowed to paint each instance of each object featured in the point cloud instead of categorizing each pixel like in semantic segmentation.

1. Click on Label type and select 3D Instance painting from the top left menu of the editor.

2. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon or Cube) to start painting.

3. Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”

4. Instance Painting: To Create a Paint Label, Select a class and hold C on the keyboard using left-click on the Lidar.

5. To create another instance of the same or another category, hold “C” while painting. It will automatically generate a new instance.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You need to save by click on Paint at the top panel and then click on save.

8. To erase, hold “e” or click on the eraser icon to erase the paint.

9. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Instructions for creating a 3D bounding box within a sequence dataset:

1. Click on Label type and select “3D bounding box” from the top left menu.

2. To create a Bounding Box: Click on Create Tool (C) from the left-hand sidebar to start labelling. Note that this mode only adds bounding boxes.

3. To Edit Bounding Box: Users are only able to edit boxes when using the “Select and Move Tool(V).”

4. To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the space of the box to move the bounding box to the desired direction.

5. To Rotate the Bounding Box: To rotate the bounding box, hold down the “R” key and click the left mouse button. Then, click on the “Rotated arrows” icon.

6. To Increase or Decrease Bounding Box: To enable scaling (increasing or decreasing box size) action, hold “S+Left Click”. Now click “Circle” to adjust the bounding box in various directions.

7. To Delete the Bounding Box: Select the label and press Shift + Delete to remove the label from the sequence, or click on the trash can icon of the label in the right panel to delete a specific key frame or a manual frame.

8. Refer Sequence Timeline to understand the Timeline feature in sequence labelling.

Steps to perform propagate labels in semantic segmentation:

1. Click on “Label type” and select “Semantic painting” from the top left menu of the editor.

2. Paint the complete area in the current frame.

3. On the right side, click on “Propagate paints.”

4. Select the next frame to copy the labels and click on “Copy.”

5. Move to the next frame and the copied labels will be displayed.

6. Make any necessary changes to the current frame, and repeat the above steps to copy the labels to the next frame.

User can add a 3D bounding box to the image by selecting the 3D bounding box label Type.

Steps to Draw a 3D Bounding Box on image:

1. Click on Label type and select the 3D bounding box from the top left menu of the editor.

2. Select a category from the drop-down box.

3. You can label the 3D bounding box by left-clicking the mouse. On mouse left click and drag user can see a 2D box which will be the front face of and another click will generate a corresponding 3D bounding box.

4. Hovering on the box reflects which face/vertex will be moved and you can use Shift + Left Click-drag to move the box.

5. You can select the 3D bounding box by clicking on the box with a left mouse click. Then you can edit the bounding box by dragging the highlighted box with a left mouse click.

6. Labels are saved automatically.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: The brush is best used to paint dynamic objects like vehicles.

Steps to paint using a brush: 1. Select a category from the editor.

2. Selecting a Brush: Click on Paint at the top panel and select 'Paint with Brush'. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. To paint another category change category type and paint using left-click. 6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers. 7. Brush depth: This option lets you adjust how far the brush has to paint the points in the LiDAR. Setting a lower depth value, lets you paint points closest to the brush and a higher depth value lets you paint all the points that fall (including the closest up to the farthest) under the brush. The Default value is 5 which can be adjusted. 8. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 9. You need to save by click on Paint at the top panel and then click on save.

10. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: Polygons are straight-sided shapes (3 or more sides), defined by three-dimensional points (vertices) and the straight lines that connect them (edges).

Steps to draw a polygon:

1. Click on Label type and select 3D polygon from the top left menu of the editor.

2. Select a category from the dropdown.

3. Click on Create Tool(C) to create a new Polygon.

4. Draw a Polygon using the mouse left click.

5. Press the ESC key to end the Polygon.

6. Polygons can be edited only in Select and Edit Tool(V).

7. Use the mouse left click to select a Polygon.

8. User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.

9. Labels will be saved automatically.

10. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: The key point annotation is used to detect shape variations and small objects, it helps to understand each point motion in the targeted object.key points can help in facial recognition and also is used to detect human body parts and estimate their poses with the right accuracy.

Steps to Draw a Point:

1. Click on Label type and select Point from the top left menu of the editor.

2. Select a category from the drop-down box.

3. You can label the points by left-clicking the mouse.

4. You can select the point by clicking on the point with a left mouse click. Then you can edit the point, by dragging the highlighted point with a left mouse click.

5. You can draw another point by pressing the “Esc” key twice.

6. Labels are saved automatically.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: Background feature allows you to paint an object that is occluded

Foreground:

1. Click on Paint and select “Paint Foreground”

2. Once “Paint Foreground” is selected, the user can paint on any area of Lidar.

3. In the image editor, there is an option for selecting the foreground as “Paint LiDAR as Foreground”.

4. You need to save by click on Paint at the top panel and then click on save.

Steps to paint background:

1. Click on Paint and select “Paint Background” at the top panel 2. Once “Paint background” is selected, paint that is already labeled won’t be affected.

3. Paint Background is mostly used when there is some unpainted area in Lidar

4. You need to save by click on Paint at the top panel and then click on save.

Overview: Secondary views enable the user to look at the label from three different views - Top, Front, and Side View.

To Disable secondary views, click on "view" in the top menu and Unselect Enable Secondary Views

Overview: Ground Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: To select the ground polygon, Use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5.Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: Smart Brush is best used to paint a large clustered point quickly.

Steps to paint with Smart Brush:

1. Select a category from the editor.

2. Selecting Smart Brush: Click on Paint at the top panel and select 'Paint with Smart Brush' or use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points.

4. To paint another category change category type and paint using left-click.

5. By increasing the distance, It would increase the search radius to expand the smart brush.

6. By increasing the number of points, it would increase the difficulty for the cluster to grow as more number of points are required to be considered as the core point of the cluster.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P”

8. You need to save by click on Paint at the top panel and then click on save.

9. To erase, hold “e” or click on the eraser icon to erase the paint

1. Mount camera on the vehicle (Camera direction should at least be partially facing the front or back of the vehicle).

2. Capture images in such a way that at least 3 equidistant straight lane boundaries are visible or can be labelled.

3. Capture at least 10 images that satisfy the above condition for better accuracy (More the number of images, better the accuracy)

4. Try to avoid scenarios where the lane boundaries are curved and not in a straight line.

Overview: The Camera View Feature allows the user to check 3D Lidar Images to view as an image on the side. This provides user clarity on each of the Lidar Images.

Add Images Page:

Pan: To move the image Zoom in Zoom out

Mapping Mode:

Image Files Toolbar:

Add point(+): To add new points on the image files. Select point: To select a required point. User can also delete a point using the delete key.

Lidar Toolbar:

Paint: To add a point on the lidar with respect to the selected point on the image file. Eraser: To remove the points in the lidar. Rotate: Used for rotating the lidar. Paint Brush Size: Slide right to increase or slide left to decrease the size of the Paintbrush. Point Size: Slide right to increase or slide left to decrease the size of the lidar points.

A timeline feature is a useful tool for visualizing your labeling activity and improving labeling speed and efficiency for sequence datasets. By providing information about the labels across datasets, it allows you to easily identify, navigate, and edit manual frames. With the timeline, you can perform these operations much faster and with greater accuracy. For more detailed information on how to use the timeline, please refer to the instructions below.

Frame Navigation

Using Frame Number

To navigate across frames using the frame number, you can click on the frame number and input the desired frame. Alternatively, you can hover over the frame number and drag left or right to move to the previous or next frame.

Using Frame Slider

You can navigate across frames in the timeline view by dragging the blue slider left or right.

Using Navigation Buttons

You can navigate across frames using the play, previous, and next buttons located on the timeline view. See the button interactions below.

You can use the ctrl/cmd key with mouse scroll to zoom in or zoom out the scale present in the timeline view.

Extend/Clip Labels

To extend or clip labels to particular frames, hover on the right or left edges of the label track (coloured bars) to see a black bar. Drag this black bar in the left/right directions to extend or clip the label to some particular frames from the sequence.

Delete label in specific frame:

You can delete the interpolated label in a specific frame by holding the command (for Mac) or Ctrl (for others) and left-clicking to a specific frame of the labelling timeline.

Note: By disabling “fill new labels”, new labels created will not be interpolated.

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

How to enable customer review for datasets? While creating a new dataset, turn on the Enable customer Review option. If you would like to use this feature for an existing dataset, go to the Settings screen in the dataset you would like to enable this, and turn on Enable customer review option.

How does this work? When customer review is enabled, users can submit the dataset for review once all the tasks are complete in all the pipeline stages. It is recommended for users to complete their existing tasks and resolve their pending issues across all stages before submitting them to review. Please note that once submitted, users will not be able to modify labels and tasks in the editor.

Users will not be able to access the review app when the work is in progress in any of the pipeline stages. When the review app is launched for the first time, you're greeted with a screen to set your review configuration. Users are provided with two types of review options: Sampling review, Full review.

In the sampling review, you can review a sample size of frames to review which are randomly chosen instead of all the frames like in the Full review. Once you have set the configuration, you are ready to get started!

When reviewing, if you notice any labels where re-work might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Issue Creation:

• Users can create issues if they find any errors in the frame/dataset.

• To create an issue, click on the Issues icon.

• Click on the area and Name the issue and select priority and click on Create.

• On the right side menu, all the created issues can be viewed and managed.

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to create a 3D bounding box:

1. Click on Label type and select 3D bounding box from the top left menu of the editor.

2. To Create a Bounding Box: Start labelling by clicking on Create Tool(C) using Left-click on the left-hand sidebar. Note that it only adds bounding boxes in this mode.
3. To Edit Bounding Box: Users can edit boxes only in “Select and Move Tool(V)”.
4. To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the empty space of the box to move the bounding box to the desired direction.
5. To Rotate Bounding Box: Hold “R+Left Click”. Now Click “Rotated arrows” to rotate the bounding box.
6. To Increase or Decrease Bounding Box: To Enable only Scale(Increase or Decrease Box Size)action, Hold “S+Left Click”. Now Click “Circle” to adjust the bounding box in different directions.
7. To Delete Bounding Box: Select the label and click delete(shift+delete for sequence) or click on the trash can icon of the label in the right panel.

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

How to enable customer review for datasets? While creating a new dataset, turn on the Enable customer Review option. If you would like to use this feature for an existing dataset, go to the Settings screen in the dataset you would like to enable this, and turn on Enable customer review option.

How does this work? When customer review is enabled, users can submit the dataset for review once all the tasks are complete in all the pipeline stages. It is recommended for users to complete their existing tasks and resolve their pending issues across all stages before submitting them to review. Please note that once submitted, users will not be able to modify labels and tasks in the editor.

Users will not be able to access the review app when the work is in progress in any of the pipeline stages. When the review app is launched for the first time, you're greeted with a screen to set your review configuration. Users are provided with two types of review options: Sampling review, Full review.

In the sampling review, you can review a sample size of frames to review which are randomly chosen instead of all the frames like in the Full review. Once you have set the configuration, you are ready to get started!

When reviewing, if you notice any labels where rework might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good, you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Issue Creation:

• Users can create issues if they find any errors in the frame/dataset.

• To create an issue, click on the Issues icon.

• Click on the area or the label and name the issue. select the priority and click on Create.

• On the right side menu, all the created issues can be viewed and managed.

Calibration Target

1. Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically.

2. Charucoboard of more than two horizontal and vertical squares. for supported Charuco dictionaries.

Target-based calibration

Data collection

Place the target roughly 3m to 5m from the camera. For the closest target position, the closer should be far enough so that all the board's edges are visible from the camera and lidar. So, it is highly recommended to do this capture inside a building rather than outside. No target position should be occluded in the camera or lidar view.

The same target should be used in all camera and lidar frames.

For example, please take images with a single board at various positions like the following.

The boards and all sensors should be static while collecting the data. To avoid time-synchronization problems, please keep the boards and the sensors stationary for at least 10 seconds while collecting each data pair.

For example, these are the steps to collect one set of calibration data:

1. Orient the camera toward the target. Start recording. Wait for 10 seconds (Don't move/rotate your car/robot/sensors). Stop recording. You must have a recording of images and lidar data for 10 seconds. Extract one image from the camera, and one frame of lidar data captured 5 seconds after the recording has started (e.g., if you start recording at 3:00:00, you stop recording at 3:00:10. We need a frame captured at 3:00:05) and save them.

2. Change the target's location and orientation. Start recording. Wait 10 seconds (Don't move/rotate your car/robot). Stop recording. Again, you must have a recording of images and lidar data for 10 seconds. Extract one image from the camera, and one frame of lidar data captured 5 seconds after the recording starts and save them.

3. Repeat the process for at least 5 data pairs.

Optional Board Position

on-ground: The target is placed on the ground (touching the ground). In such a case, enable the on ground flag in the target configuration. Also, make sure that the lidar data captures the ground points. By doing so, we can optimize the extrinsic parameters using the ground points.

Tilted: A holder holds the target up in the air and tilts right by around 45 degrees. In such a case, enable the Tilted flag in the target configuration. This approach enables deep optimization, and the extrinsic parameters are further optimized using the edge points of the board.

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

1. Click on Label type and select 3D Semantic painting from the top left menu of the editor.

2. To Create a Paint Label, Select a class and use left-click to paint on Lidar.

3. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon or Cube or smart brush) to start painting.

4. Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”.

5. To paint another category change category type and paint using left-click.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P”.

8. You need to save by clicking on Paint at the top panel and then click on save or use Enable Autosave.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

10. Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

This calibration is deprecated. Please use the lane-based targetless vehicle camera calibration instead.

Data collection for targetless vehicle-camera calibration:

1. Mount camera on the vehicle (direction should at least be partially facing the front ).

2. Input the Camera Intrinsics in the config page.

3. Record a scene with 5 to 60 where the vehicle is moving.

4. Keep the speed of the vehicle such that the images captured are not blurry and there is enough overlap between frames of images for feature extraction. Avoid stationary vehicle clips.

5. Having good edges (like buildings) in the images will result in better calibration.

6. The Processing and the feature extraction for the necessary frames from the video takes time.

7. Select a frame where you can easily draw the horizon line and adjust the vanishing point ( Horizon line can be considered as the line segregating the sky and the earth and vanishing point is point where the parallel lines of the road meet at infinity)

8. Click on calibrate

Example data:

Extrinsic Calibration Output:

• roll, pitch, yaw are in degrees .

• roll, pitch, yaw are the extrinsic parameters downloaded from the calibration tool.

• vehiclePoint3D is the 3d coordinates of a point in the vehicle coordinate system.

Steps to paint with Smart Brush: 1. Select a category from the editor. 2. Selecting 3D Brush/Cube: Click on Paint at the top panel and select 'Paint with 3D Brush' or use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points. 4. To paint another category change the category type and paint using the cube. 5. Use the 'F' key to start painting. 6. The size of the cube can be adjusted.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P” 8. You need to save by click on Paint at the top panel and then click on save.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: Issue Creation allows you to create an issue for a label, notify the annotator about the issue, and then discuss the issue with our comment feature.

Steps to create an issue:

1. You can create an issue using by clicking on the "create issue" icon in the left panel on the labeling screen.

2. Once you've created the issue using the "create issue" feature. You can go into the QC view by clicking on "issues" in the right panel.

3. If the current stage is not the first stage, then the assignee email is the email to which this task is assigned in the previous stage.

1. What are the Recommended Browsers? Chrome (preferably the latest or at least version 97+)

2. What is the required Screen Resolution? Recommended 1920x1080 pixels or more (at least 1366x768 pixels)

3. What is the list of domains which need to be whitelisted? Domain Whitelist: The following domains and any requests from its child URLs have to be whitelisted:

Overview: Boxes can be adjusted in all frames at a single place.

Steps for using Label View:

1. In Label View the selected bounding box can be viewed and adjusted in all the preloaded frames.

2. The box will be displayed in Top, Front and Side view.

3. Select a bounding box and Click on ’Label view’ to enable it.

Following are some shortcut keys which can be used in 3D Editor

Target-based

Calibration target

The Checkerboard is the calibration target. You can use the attached PDF. It has seven internal corners horizontally and nine internal corners vertically.

Data Collection

Place the target in the overlapping field of view of the cameras and take an image from both cameras. Then, move the target all across the overlapping field of view of the cameras and take images from both cameras. For good extrinsic parameters, a minimum of 25 pairs of images is recommended.

Here is an example set of files used during this calibration process.

Calculating the intrinsic parameters for the left and right cameras is also recommended. Although the tool supports on-the-fly intrinsics calculations, we use the uploaded images for the intrinsic parameters, which tends to raise inaccuracies in the intrinsics calibration and overall Overlapping Calibration results.

Targetless

It is recommended that the scene have sufficient objects, textures, and unique patterns for feature detectors to identify and match. For example, the above calibration target example is not an ideal dataset for targetless as it has just a checkerboard in front of a plain white wall.

Here is an example set of files used during this calibration process.

Before:

Accessing Show Ground Mesh:

After:

Labels for 2D semantic segmentation can be downloaded as the combination of a binary (.npy) file for each frame inside the dataset and two supporting metadata (.json), and colors (.json) files per dataset.

The JSON data present in the metadata file is in the below format :

                                      {
                                          "sensor_id": {
                                              "file_id_1": [
                                                    "paint_category_1",
                                                    "paint_category_2",
                                                    "paint_category_3",
                                              ],
                                              "file_id_2": [
                                                  "paint_category_2",
                                                    "paint_category_3",
                                              ]
                                          } 
                                     }

Here, paint_categories is a list of paint categories that are painted in that particular frame, which are configured at the dataset level.

The JSON data present in the colors file is in the below format :

                                   {
                      "format": [
                        "b",
                        "g",
                        "r"
                      ],
                      "paint_category_1": [
                        66,
                        45,
                        115
                      ],
                      "paint_category_2": [
                        70,
                        199,
                        184
                      ]
                }

Here, paint_category_1 is a list of r, g, and b values of the particular category listed in format order, for example here it's blue, green and red.

The npy file is a binary file which contains pixel-level information for the label category. For a file with image_width w pixel and image_height h pixel, npy file contains (h*w) bytes that represent the label category. This npy file will be per frame.

For example, consider a dataset containing a file (file_id_1) with an image_width of 1216 and image_height 2560 pixels, respectively. Assuming that the pixel point at image_width 1200 and image_height 2500 in the file of the dataset is annotated as below :

                                      label_category : "paint_category_2"

For the above scenario, the npy file will contain 3112960 bytes (hw). The annotation information for the point will be present at 3041200th (y * image Width + x) (25001216 + 1200) index. The byte value at this index would be 2 which is the based index of "paint_category_2" in the paint categories provided in the metadata for a particular file(file_id_1). The value 0 for the label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header

< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": [“Drivable region”, “Uneven terrain”]}

Remember the following information:

• You can obtain the paint_categories and compression format from the data above.

• The compression format is "pako_compressed,” and you can use pako decompression to retrieve the paint labels for each point.

• For visualization purposes, you can use colors.json.

• Regarding the paint categories, your understanding is correct. We assign 0 for unlabeled points and use the index from the paint_categories field in the paint labels.

Overview: In the 3D coordinate system, lines can be described using vector equations or parametric equations. Lines in 3D have equations similar to lines in 2D and can be found given two points on the line.

Steps to draw lines:

1. Click on Label type and select 3D line from the top left menu of the editor.

2. Select the category from the dropdown.

3. Click on Create Tool(C) to create a new line.

4. Draw a line using the mouse left click.

5. Press the ESC key for drawing another line.

6. Lines can be edited only in Select and Edit Tool(V).

7. Use mouse left click to select a line.

8. User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.

9. Labels will be saved automatically.

10. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Calibration Target

Charucoboard is the calibration target. Charucoboard of the following dictionaries of any size and squares can be used as the target. Targets should be as flat as possible and the surface of the target should be anti-glare (matte finish)

Click here for supported charuco dictionaries.

Note: When multiple targets are used, make sure targets are of different dictionaries.

We suggest that users make at least 1.0m*1.0m charcoboards for good corner detection.

Data for camera-to-lidar extrinsic calibration

Place the targets in the focus of the camera and the LiDAR's field of view. The targets should be placed on the ground or on a stand. Avoid holding the targets with your hands.

The targets and all sensors should be static while collecting the data. To avoid time-synchronization problems, please keep the boards and the sensors stationary for at least 10 seconds while collecting each calibration data set.

For example, these are the steps to collect one set of calibration data:

1. Orient the camera toward the targets. Start recording. Wait for 10 seconds (Don't move/rotate your car/robot/sensors). Stop recording. You must have a recording of images and lidar data for 10 seconds. Extract one image from the camera and one frame of lidar data captured 5 seconds after the recording has started (e.g., if you start recording at 3:00:00, you stop recording at 3:00:10. We need a frame captured at 3:00:05) and save them.

2. Change the targets' location and orientation. Start recording. Wait 10 seconds (Don't move/rotate your car/robot). Stop recording. Again, you must have a recording of images and lidar data for 10 seconds. Extract one image from the camera and one frame of lidar data captured 5 seconds after the recording starts and save them. (Dataset front_2)

3. In the same manner, save Dataset front_3, front_4, and front_5.

Multi-target LiDAR Camera calibration works even with a single pair of images and a lidar frame. The more pairs, the better the accuracy will be.

Optional board position

on ground

The target is placed on the ground (touching the ground). In such a case, enable the on ground flag in the configuration page. Also, make sure that the lidar data captures the ground points. By doing so, we can optimize the extrinsic parameters using the ground points.

Tilted

A holder holds the target up in the air and tilts right by around 45 degrees. In such a case, enable the Tilted flag in the target configuration. This approach enables deep optimization, and the extrinsic parameters are further optimized using the edge points of the board.

Steps to show fused cloud:

1. Click on Fused cloud settings or click 'Alt+f' and select “Show Fused Cloud”.

2. Once the Fused cloud is selected, the accumulated points in the lidar will be shown in the current frame.

3. The points will be shown only for the frames which are preloaded.

Fused Tile Size: Click on Fused cloud settings and change X and Y values under Fused Tile Size for increasing or decreasing tile size or click on Edit Fused Tile and increase the size of the tile in any direction.

Profile homepage

• This page lets users view, create, launch, and delete calibration profiles.

• Click on New to create a new calibration profile.

Profile selection

Users can create three types of profiles

1. Camera Intrinsic Profile

2. IMU Intrinsic Profile

3. Target Profile

Camera Intrinsic Profile

We support two camera lens models. For wide-angle cameras, use Fish-eye and Standard for the rest.

• Standard is the Brown-Conrady camera model

• Fish-eye is the Kannala-Brandt camera model

IMU Intrinsic Profile

Target Profile

We support three types of targets

1. Checkerboard

2. Charuco board

3. Plain board

There are multiple types of aruco markers from which the charuco board can be made. Please visit here for supported charuco dictionaries.

Auto-Checker:

We get a list of suggestions when we click the Auto-check button available in the Issues tab. If there are any issues, the user gets a list of issues.

From the suggestion list, for each suggestion, we can either create an Issue by clicking the Accept button or reject the issue by clicking the Reject button.

We can create Issues from the rejected suggestions as well.

Configure Auto-checker Tab

Based on the configuration given here, we can control Auto checker suggestions List Item.

1. From the options in the first section, we can see either Active suggestions or Rejected suggestions

2. We have the following filters for the Auto checker suggestions.

1. Issue Type

2. Label Type

3. Issue Title

Other Important Issue Flow Features

Visual Sync between Editor and Issues Panel

To see the visible sync for the same issue displayed in the Editor and Issues Panel,

1. Select Create Issue primary toolbar mode.

2. Open the Issue panel and select the Issues tab to see the issues list.

Calibration Target

1. Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically.

2. Charucoboard of more than two horizontal and vertical squares. for supported Charuco dictionaries.

Data for camera intrinsic and distortion calibration

Checkerboard

Place the checkerboard in the field of view of the camera and make sure the checkerboard is in the camera's focus. Move the checkerboard or the camera to different positions and take images so that the coverage of the checkerboard in the camera's field of view is as high as possible. While moving the checkerboard or camera, it is necessary that the entire checkerboard is present in the camera's field of view. In addition, the checkerboard should be tilted slightly in different directions for each image. Do not move the board or the camera while taking the images. Also, do not hold the checkerboard by hand to minimize blur due to shaking. You can also use a computer monitor instead of a physical checkerboard for intrinsic calibration. Monitors are very accurate and flat. You can show this on the monitor for calibration.

Charucoboard

Place the charucoboard in the field of view of the camera and make sure the charucoboard is in the camera's focus. Move the charucoboard or the camera to different positions and take images so that the coverage of the charucoboard in the camera's field of view is as high as possible. While moving the charucoboard or camera, it is not necessary that the entire charucoboard is present in the camera's field of view. In addition, the charucoboard should be tilted slightly in different directions for each image. Do not move the board or the camera while taking the images. Also, do not hold the charucoboard by hand to minimize blur due to shaking. You can also use a computer monitor instead of a physical charucoboard for intrinsic calibration. Monitors are very accurate and flat. Please visit our website to download charucoboard:

Checkerboard sample data collection

Fisheye and wide angle cameras

For cameras with high distortions, such as fisheye and wide-angle cameras, we need to pay special attention to the field of distortion. If each board only covers a small portion of the image, it is likely to lead to overfitting of the field of distortion. i.e. Locally correct distortion but globally incorrect distortion. We recommend to have at least one image where the board covers most of the field of view. It can give useful information on the distortion across the entire field of view.

Long focal length cameras

For cameras with long focal length and narrow (<= 30 degree) field-of-view, it is difficult to obtain the focal length and principal point accurately. If all boards face the camera almost directly, there may be large errors in the vanishing points and focal length. We need some images where the board is very tilted to obtain accurate vanish points, focal length, and principal point.

Charucoboard sample data collection

Calibration Target:

Checkerboard is the calibration target. For small vehicles, the minimum checkerboard size is 0.6m^2. For large cars, the minimum checkerboard size is 2.4m^2. If calibrating a small car, you can print the below pdf file on a foam board at 1.0m x 0.6m. Most print shops can print this. https://drive.google.com/file/d/1mTR8HTpvROE1Pv0rmXEBVLSxs_yMDnvf?usp=sharing

For car wheels, use the Aruco markers. Click here for Aruco markers. Note that the ArUco markers should be as large and flat as possible with a white border. The white border is necessary for tag detection.

Data collection for vehicle-camera calibration:

1. We need two cameras,

   1. A mounted camera on the car with respect to which calibration is done against the vehicle.

   2. An external camera to take images from both left and right sides of the vehicle. Note: The external camera should have a fixed focal length. Changing focal length / auto-focus will change the camera intrinsic parameters. A DSLR with manual fixed focal length can make a good external camera. Modern cell phone cameras all have auto-focus so should not be used as an external camera.

2. Intrinsic parameters for both mounted and external cameras before proceeding to vehicle-camera calibration. Intrinsics can be obtained from the Camera Intrinsic calibration as well.

3. Have a checkerboard with known horizontal corners, vertical corners, and square size.

4. Stick Aruco markers to the vehicle wheels to auto-detect the wheel center. Click here for Aruco markers. Note that the ArUco markers should be as large and flat as possible with a white border. Note: The ArUco markers must match the wheel position. Mismatched markers will not be recognized in the application.

Front Left:

Front Right:

Rear Left:

Rear Right:

Setup and Data collection from the Left-side of the vehicle

1. Place and fix the checkerboard position in the mounted camera's field of view.

2. Take an image from the mounted camera.

3. Take an image from the external camera having a front-left wheel, rear-left wheel, and checkerboard in its field of view. Move the external camera to a different location and take another image. Repeat the process for at least three iterations. Note: Using a tripod with the the external camera can reduce motion blur and improve calibration accuracy.

Note: The vehicle, ArUco tags, and checkerboard should all be static during these 3 steps.

Setup and Data collection from the Right-side of the vehicle

1. Place and fix the checkerboard position in the mounted camera's field of view.

2. Take an image from the mounted camera.

3. Take an image from the external camera having a front-right wheel, rear-right wheel, and checkerboard in its field of view. Move the external camera to a different location and take another image. Repeat the process for at least three iterations.

Note: The vehicle, ArUco tags, and checkerboard should all be static during these 3 steps.

What are the label sets?

A label set is a property that is set on every label and task. Label sets help multiple users to label the same object using their label set id. This will help admins to export labels from a dataset for a particular label set and compare it with the labels from a different label set. When a dataset is created, a “default” label set id is created along with it.

How do I create a label set?

• Go to the editor page

• Click on the dropdown “Default”

• Click on “Create new label set”

• Enter the name of the new label set

• Select an existing label set from “Include labels from the existing label set” if the user wants to copy labels from an existing label set to the new label set.

• Clicking on “create” will create a new label set.

How do I delete a label set?

• Go to the editor page.

• Click on the Label set.

• Click on the delete icon beside the label set name which the user wants to delete.

How do I copy labels between two label set?

• Go to the editor page

• Click on the Label set

• Select the option “Copy labels to label set”

• Select the label set from which the labels would be copied.

• Select a label set from the dropdown “Select label set”

• Clicking on “copy” will copy all the labels

How do I export labels from a label set?

• Go to dataset export page

• Select a label set from the dropdown box

• Click on “Download” to export labels.

Label set Comparison:

How do I compare two label sets?

User can only compare label sets for 3d bounding box labels

• Go to the editor page

• Click on the Label set from the top left menu of the editor.

• Select “Compare”.

• Select a label set to compare with

• Clicking “compare” will show labels from both the label sets in the lidar viewer and a table as shown below.

Calibration Target

Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically.

Prerequisites

Reference Points for Data Collection

Paste the adhesive tape on the ground along the vehicle's edges based on the camera-facing direction. Assume calibrating the front-facing camera sensor as an example for this collection process. Paste the adhesive tape shown in the image below for the front-facing camera. (See

To maintain a fixed distance between the target and the vehicle, paste the adhesive tape parallel to the vehicle with a 1-10m distance between the target and the vehicle, as shown in the below image. It acts as a Reference Line (RL) for the target.

Definitions

Vehicle Reference Point (VRP)

The adhesive tape's left intersection point (from the camera perspective) along the vehicle's edges, as shown in the below image.

Intersection Reference Point (IRP)

This is the left intersection point of the adhesive tape from the vehicle with the adhesive tape from the reference line, as shown in the below image.

Target Reference Point (TRP)

This is the bottom-left edge corner of the target, as shown in the below image.

Vehicle dimensions

Left wheelbase

Distance between the center of the front left wheel to the center of the rear left wheel.

Right wheelbase

Distance between the center of the front right wheel to the center of the rear right wheel.

Note: For a rectangular vehicle, Left wheelbase = Right wheelbase

Front track

Distance between the outer edge of the front left wheel to the outer edge of the front right wheel.

Rear track

Distance between the outer edge of the rear left wheel to the outer edge of the rear right wheel.

Note: For a rectangular vehicle, Front track = Rear track

Front overhang

Distance between the center of the front left/right wheel to the vehicle's front.

Rear overhang

Distance between the center of the rear left/right wheel to the vehicle's rear.

Data collection

1. Mount the camera onto the vehicle and note the distance between VRP and IRP.

2. Place the board at the Reference Line and ensure the board is parallel or perpendicular to the car.

3. Take an image from the mounted camera and note the distance between IRP and TRP.

4. Move the board along the Reference Line, and at every placement, take an image from the mounted camera and note the distance between IRP and TRP.

Note: We recommend capturing at least three images with the target perpendicular to the ground and at least three images parallel to the ground in different positions on the Reference Line to maximize accuracy.

How to paste adhesive tape along the vehicle for different camera-facing directions?

Annotation data is often collected by ROS and stored as ROS bags. In order to use deepen.ai’s system for annotation, the data must be first converted into the JSON format: https://help.deepen.ai/documentation/3d-input-output/3d-upload-json-format

This JSON file specifies the exact data for annotation. It includes point clouds, images, timestamps, intrinsic and extrinsic calibrations, as well as localization information.

Usually, each user writes a script to convert their bag data into the JSON format. We cannot use a common script for conversion for several reasons:

1. Each ROS bag may have many topics. The user needs to specify the exact topics for annotation.

2. Some information such as intrinsic and extrinsic calibrations may not be in the ROS bags.

3. If a special camera model is used, the user may wish to only send the rectified images for annotation.

4. The JSON format links the point cloud and images shown to the annotators. It does not sync using timestamps like RVIZ.

5. If accurate localization is available, the point cloud is transformed into a fixed world coordinate.

6. Advanced processing such as LiDAR ego-motion compensation is typically performed in the script.

Users not familiar with the JSON format may find the conversion script difficult to develop. At deepen, we have worked with many clients on their data conversion scripts. Although each script is different, we have found many common themes in these scripts. If you are developing a new script, this tutorial will help you to get started and get your data ready for annotation. ROS has good Python support. Most conversion scripts are written in Python, so this tutorial also assumes that. We will walk you through the various steps of the conversion script. We will try to describe the simplest processing. There are many advanced techniques which can improve the data, but we will not cover them here.

Reading the Bag file(s)

The first step is very obvious. You need to specify the path of the files and ROS topics to annotate. The topics should include point clouds and images. Images are very important for annotation speed and accuracy.

Synchronizing the point clouds and images

In the 3D annotation tool, the annotator is shown a point cloud and one image from each camera. Thus, the JSON format specifies the connection between the point cloud and images. To calculate this, our script needs to explicitly synchronize the data. Often, the synchronization is done by ROS timestamps. Let us assume that there is only one LiDAR. Thus, there is only a single point cloud sequence. In this stage, we make a single pass through the point cloud data and collect its timestamps. This is our primary timestamp sequence.

We then make a pass through each image topic that we can show to the annotators. As we go through the images, we find the closest timestamp to each LiDAR timestamp. This will be the image attached to each point cloud. Thus, there is one image from each camera for each point cloud. The image sequence is synchronized to the LiDAR sequence.

Temporal Calibration (Advanced Topic)

Technically, each timestamp occurs when the LiDAR finishes a spin or a camera closes its shutter and ROS records its data. The sensors, network, and ROS itself all add latency to this process. Thus, the timestamps usually occur a few milliseconds after the actual events. This latency is typically different for each sensor and introduces inconsistency in timestamps. Temporal calibration is the method to adjust the timestamps, so all sensors have consistent timestamps. We will not cover it here, and you may skip it for your initial conversion script.

Multi-LiDAR

In the case when we have multiple LiDARs that we need to annotate, we can pick one LiDAR as the primary one and synchronize all other LiDARs to it. Note that there is the “d” field for each point in deepen’s JSON format, to which we can assign the LiDAR ID. If you use the “d” field, our annotation UI allows you to distinguish between point clouds from different LiDARs.

Localization

The JSON format specifies all points in a single world coordinate because it would make annotation much easier and more accurate: All static objects would be in a fixed location. All dynamic objects would have their simple motions, without complications from the motion of the ego vehicle itself. In order to achieve this, we would need accurate localization data. The best localization usually comes from an expensive INS/GNSS. Otherwise, LiDAR SLAM combined with other sensors such as IMU and odometers can also give accurate localization in most situations.

If an accurate localization is available as a ROS topic, we just need to find the LiDAR pose with a timestamp closest to that of the LiDAR point cloud. We can use the pose to transform the point cloud into the world coordinate. Note that accurate LiDAR poses require LiDAR to INS calibration, which we will not cover here.

If localization is unavailable, we suggest that you try one of the LiDAR SLAM algorithms. If you skip this step and use the LiDAR point cloud as-is, it is still possible to annotate the data, but the cost and accuracy would both suffer.

Ego Motion Compensation (Advanced topic)

Most LiDARs have a slow “rolling shutter” where each scan or spin can take tens or hundreds of milliseconds. During this time interval, the LiDAR itself may be going through a complicated motion. If we treat the LiDAR as stationary for the entire time interval, the point cloud would be inaccurate. Ego motion compensation is the technique to solve this problem, but we will not cover it here. You can ignore this issue unless your vehicle was moving at a high speed such as on a highway.

Intrinsic and Extrinsic Calibrations

Accurate intrinsic calibration is required for each camera. We support the common plumb-bob (Brown & Conrady) and equidistant fisheye (Kannaly & Brandt) camera models. If you choose a model we don’t currently support, you can submit rectified images instead.

Extrinsic calibration is for specifying the camera pose for each image. If localization and ROS tf transforms are available, you just need to obtain the camera pose at the timestamp of the image. Note that we are using the camera coordinate system in OpenCV. It is identical to the ROS optical coordinate system, not the standard ROS coordinate system.

If the tf transforms cannot give you correct camera poses, but you have the extrinsic LiDAR-camera calibration, you can apply the extrinsic calibration to obtain the camera pose from the LiDAR pose. It is just a simple matrix multiplication. Since the point cloud and image usually have different timestamps, it would be more accurate to interpolate the LiDAR or Camera poses, but we will skip it for this tutorial.

Output

After obtaining the information above, we just need to make a pass through the point clouds and images to output the JSON and image files. Note that the JSON format supports BASE64 encoding, which can make your JSON files much smaller and the script run much faster.

Debugging

Debugging your first conversion script often takes a long time. We have some tips to find the most common bugs.

The first tip to debug is to output a small number of frames. Your script will finish quickly. You can upload this subset into deepen’s tool quickly and visualize the data.

Localization

To verify your localization accuracy, the easiest way is to load a moving point cloud sequence into deepen’s annotation tool. You can use the “fuse” option to accumulate multiple point clouds together. If the static objects are sharp in the fused point cloud, you are likely to have the correct localization. If not, there are many common mistakes:

1. Low quality or misconfigured INS/GNSS

2. Wrong usage of INS output

3. Wrong coordinate frame is used

4. Bad output from LiDAR SLAM algorithm

Calibration

To validate your camera calibrations, you can load a sequence into the annotation tool. Add a 3D box to an object. If the projection of the box on the image looks correct, you probably have the correct calibrations. Otherwise, there are many common mistakes:

1. Giving rectified images and also non-zero distortion coefficients

2. Using the ROS coordinate system instead of the optical coordinate system

3. Scaled the images but did not change the intrinsic parameters

4. Bad LiDAR-camera calibration. You can use deepen’s calibration tool to redo or correct the calibration. https://help.deepen.ai/documentation/calibration/camera-lidar

5. The pose of the camera may be off due to localization error.

Synchronization

In order to validate the synchronization of your data, you can load a sequence into the annotation tool. Add a box to a moving object. If the projections on all camera images look correct, you should have a proper synchronization between point cloud and camera. For a high-precision check, the object should have a high angular velocity relative to the sensors. Therefore, you can use an object moving quickly from left to right or use a sequence where the ego vehicle is turning rapidly. If you found a synchronization error, there are several common mistakes:

1. For big errors, there is likely to be some bugs in the synchronization logic in the script.

2. For small errors, temporal calibration may be needed.

3. For very small errors, we may need to consider the exact time the LiDAR scans the object which is different from the timestamp we assigned to the entire point cloud. This is an advanced topic.

Visualization code

We have developed an offline visualization script to visualize the JSON files. Please find the tool and documentation for it here.

Sample Code

We will release a sample conversion script soon.

Please contact us if you run into problems with your conversion script. We will gladly help you debug your script.

Our powerful and user-friendly data annotation platform is available as a managed service, in the cloud, or on-premise. SaaS Offering: Host your data on Deepen servers and receive automatic updates and access to all features. Hybrid On-premise: Host your data on your own servers and provide Deepen a URL to an asset that is behind a VPN. All source data will be unreachable by Deepen but generated assets(annotation data) will be stored on Deepen servers. Full On-premise: Work with our engineering team to deploy

Labels for 3D semantic segmentation can be downloaded as the combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in the metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The DPN file is a binary file which contains point level information for label category and label id. For a dataset with n number of points, dpn file contains n bytes that represent the label category.

For example, consider a dataset containing 3 files each with 10,000 points, 11,000 points and 9000 points respectively. Assuming that the point with index 50 in second file of the dataset is annotated as below :

label_category : "paint_category_2"

For the above scenario, dpn file will contain 30000 bytes (10,000 + 11,000 + 9000). The annotation information for the point will be present at 10,050th (10,000 + 50) index. The byte value at this index would be 2 which is the 1 based index of "paint_category_2" in the paint categories provided in the metadata. The value 0 for label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header:

< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": ["Drivable region", "Uneven terrain", "Hard Vegetation", "Soft vegetation", "Construction object", "Other Traversable Static", "Ghost", "Static Object", "car", "Vehicle _ Truck", "Pedestrian with Object _ Adult", "Pedestrian _ Adult", "Vehicle _ Trailer", "Construction Object _ Construction Cones and Poles", "Ghost _ Sensor Artifact", "Vehicle _ Van", "Vehicle _ Car", "Ego Car", "Ground", "dynamic_buffer"]}

You can get the paint_categories and compression format from the above data. Here the compression format is pako_compressed and you can use pako decompression to get the paint labels for each point.

For the paint categories, your understanding is correct. We reserve 0 for unlabelled points and use the index from the paint_categories field in the paint labels.

Calibration Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration selection

Select Vehicle-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting Vehicle-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

1. Select the Terrain as Flat and approach as Targetless

1. Select the Lane based calibration option "Atleast 3 equidistant lane boundary lines (New)" option

Configuration

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Upload image files from the mounted camera

Add lanes in the Image

1. Select any of the uploaded image and draw at least 3 straight lines resembling the equidistant lane boundaries present in the image.

2. Repeat the above step for at least 10 images for better accuracy of the final calibrated results.

3. Upon completion, Click 'Continue' to move into the calibration page.

Run Calibration

In the calibration page, Users must click Calibrate to calculate the extrinsic parameters of the camera with respect to the vehicle coordinate system

Visualizer

On successful calibration, click 'Visualize' button on the top right to view the Birds Eye View (BEV) representation of the camera image according to the calibrated extrinsic parameters.

Error function

• For an ideal calibration, the lanes should appear parallel and equidistant when transformed in the BEV (Bird's Eye View) images.

• Based on the above theory, we calculate the Parallelism error and Equidistant error and combine those two to get the final error.

• Parallelism Error

  • $P_e = \frac{1}{N}\sum_{i=1}^{N} |m_i - \hat{m}|$,

  • where $m_i$ is slope of $ith$ line in the BEV image and $\hat{m}$ is the mean slope of all the lanes

• Equidistant error

  • $E_e = \frac{1}{N}\sum_{i=1}^{N} \frac{|c_{i+1} - c_i|}{\sqrt{1 + \hat{m}^2}}$,

  • where $c_i$ is intercept of $ith$ line in its BEV equation ax + by + c = 0 and $\hat{m}$ is the mean slope of all the lanes

• $Error = \frac{\alpha P_e + \beta E_e}{N_c}, \space where \space \alpha, \beta, and \space N_c \space are \space normalisation \space constants$.

Graph

• The below graph is a plot between the actual ground truth error and Deepen error function on 9 observations including the ground truth on a Kitti dataset

• The plot shows a strong correlation between our error function and the actual deviation from the ground truth within 1.3 degree of the ground truth.

• The ground truth for the above mentioned dataset is 0.634 (Roll), -0.430 (Pitch), 0.310 (Yaw) and the estimated extrinsic parameters by optimising our error function is 1.568, -0.215, 0.327 with a deviation of just 0.95 degrees from the ground truth with most of the error being in the estimation of Roll angle.

Extrinsic Calibration Output:

• The extrinsic parameters of the camera are with respect to the vehicle ROS coordinate system.

• In the ROS coordinate system of a vehicle, the X-axis is facing along the vehicle direction, Y-axis is towards the left of the vehicle, and the Z-axis is perpendicular to the road plane facing towards the top.

• In the tool, the extrinsic parameters Roll, Pitch, Yaw are in degrees.

• Roll, Pitch, and Yaw are the extrinsic parameters downloaded from the calibration tool

Overview: 3D Editor is used to label 3D Bounding Boxes, Polygons, Instance & Semantic paint and 3D Lines for Point Cloud.

Primary Toolbar: It is on the left side of the image editor which has Home, Create Tool(C), Select and Move Tool (V), Select and Scale Tool(S), Select and Rotate Tool(R), Orbit Tool(1), Pan Tool(2), Dolly Tool(3) and Create Issue.

• Home Button: Home button is used to navigate to the Homepage.

• Create Tool: The user needs to click on Create Tool to add any new labels. Labels can't be created without selecting this option.

• Select and Move: The user can label frames using this option. The user can’t label if this option is not selected.

• Select and Scale Tool: Selects labels and activates a widget to scale them in 3D space.

• Select and Rotate (R): Selects labels and activates a widget to rotate them in 3D space.

• Orbit Tool(1): Click and drag on it to rotate the view around the lidar.

• Pan Tool(2): Click and drag to pan the lidar up, down, left and right.

• Dolly Tool(3): Click and drag to move the lidar forward and backwards.

• Create issues: Users can create issues at frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar.

Primary Menu bar: It is on the top of the image editor. It consists of Frame, Label, View, LiDAR, and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete, Fit Label to Points, Set box on ground by default, Set box on ground, Enable one click placement, Rotate, Rotate 180 in all frames, Propagate Labels(Only for Single frames) and Attributes.

View: View consists of Show Labels Text, Show Labels Attributes, Show selected Label only, Show Bounding Boxes, Show Polygons, Show Ground Mesh, Uncolor Ground Points, Show Lidar Cube, Show Grid, Enable Secondary Views, Enable Camera Views, Perspective, Orthographic, Light Theme, Dark Theme, and Reset View.

LiDAR: LiDAR consists of Enable Point Intensity, Color By Z, Show Colored Points inside Box, Show Black Points(Only for coloured lidars), Ground, Cloud Radius and Lidar FOV(Horizontal and Vertical).

Secondary Menu bar: It consists of a Label Set, Labelling Pipeline, Label category, point size, Labelling Instructions, Reset View. Additionally, for sequence annotations, we have Fused cloud and Z-threshold along with these.

Panel Content: It is on the right of image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these we also have Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular image. If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on Dataset the tool navigates the user to the dataset page in the tool.

• User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

Buttons in the lower navigation bar and their usage:

• Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.

• Preload Data: It loads the frames in advance before opening them.

• User can hide the bottom bar using the Hide bottom bar option.

• This is only for sequence datasets.

Calibrations Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration Selection

Select the Camera Intrinsic Calibration button to create a new dataset.

Calibration Instructions Page

Upon selecting Camera Intrinsic Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Configuration for Checkerboard

• Target configuration = Checkerboard

• Enable use EXIF metadata to use of EXIF metadata from images to optimize calibration. Disable this if using an External Lens for the Camera is recommended.

• Camera lens model: For wide-angle cameras, use Fish-eye and Standard for the rest

  • Standard is the Brown-Conrady camera model

  • Fish-eye is the Kannala-Brandt camera model

• Horizontal corners: Number of horizontal inner corners in the checkerboard

• Vertical corners: Number of vertical inner corners in the checkerboard

Configuration for Charucoboard

• Target configuration = Charuco Board

• Enable use EXIF metadata to use of EXIF metadata from images to optimize calibration. Disable this if using an External Lens for the Camera is recommended.

• Camera lens model: For wide-angle cameras, use Fish-eye and Standard for the rest

• Charucoboard Dictionary: There are multiple types of aruco markers from which the charuco board can be made. Please visit for supported charuco dictionaries.

• Horizontal corners: Number of chessboard squares in the horizontal direction in charucoboard

• Vertical corners: Number of chessboard squares in the vertical direction in charucoboard

• Square size: The size of the square in the board in meters

• Marker size: The size of the ArUco marker present inside the charucoboard in meters.

Calibration Pipeline

Add images

Upload the images taken from the camera for which intrinsics need to be calculated.

Settings (optional)

Alpha is the new scaling parameter in Camera Intrinsic Calibration. Alpha can have values between 0 and 1.

1. Alpha=0, it returns an undistorted image with minimum unwanted pixels

2. Alpha=1, all pixels are retained with some extra black pixels.

Default: Includes k1, k2, k3, p1, p2 in distortion coefficients

Extended intrinsics: Includes k1, k2, p1, p2, k3, k4, k5, k6 in distortion coefficients

Minimal(K1,K2): excludes k3 from distortion coefficients and includes only k1, k2, p1, p2

Run calibration

Click the Calibrate button at the bottom to trigger calibration. You can see the intrinsic parameters and error statistics in the right panel upon completion.

Verify Results

Error Stats

The Reprojection error is in pixels. It is the mean of the Euclidean distance between the auto-detected checkerboard corners and reprojected checkerboard corners. The closer the Reprojection error is to zero, the better the intrinsics are.

Uncertainties (only for fisheye camera model)

The uncertainties represent the potential variability in the optimized parameters. They reflect how precisely each parameter is determined during calibration, based on the provided input data.

• Lower uncertainty: Indicates higher confidence in the parameter's accuracy and reliability.

• Higher uncertainty: Suggests the parameter is less well-determined, which could result from insufficient data, poor data distribution, or sensitivity to noise and errors.

By analyzing these uncertainties, you can assess the quality and robustness of the calibration results.

Undistorted Image

• Users can visualize the Undistorted image to check the quality of the intrinsics.

• The side-by-side view can be used to check both the distorted and undistorted images simultaneously

Checkerboard Coverage

• Checkerboard coverage shows the area covered by the checkerboard corners from all uploaded images. The higher the coverage, the better the intrinsic parameters.

  • 0 - 50% is low coverage

  • 51 - 70% is moderate coverage

  • 71 - 100% is Good coverage

• Users can see the individual reprojection error of all the checkerboard corner points. A color ramp is used to depict the reprojection error. A light red color shows a lower reprojection error, and a darker red indicates a higher reprojection error.

Save to profile

Camera Intrinsic parameters can be saved to the profile for easier import in other calibrations.

1. Calibration list page, where the users can load an existing dataset or create a new one.

2. New calibration selection modal.

3. Get started page of vehicle-camera setup.

4. Calibration settings modal.

• Dataset name can be added here.

• The user has to select the shape of the vehicle. Either rectangle or trapezoid.

For rectangle-shaped vehicles, users can input the measured values

For trapezoid-shaped vehicles, users can input the following measured values.

Description for vehicle details:

Configure checkerboard and Aruco:

AruCo markers are used for automatic wheel detection. Add the measurement of the marker.

Similarly, checkerboard configurations need to be updated.

5. Enter the intrinsic parameters for the mounted camera.

• Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the intrinsic calibration tool and calibrate the results. Save them to profile and then load them here.

• Or users can also load the JSON file.

6. Add images related to the mounted camera. One for the left view and the other for the right view.

7. Detect the checkerboard corners for both the mounted camera images.

• Click on detect corners to get the checkerboard corners auto-detected.

• Or else manually add the border corners to get all checkerboard corners.

8. Enter the intrinsic parameters for the external camera.

• Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the intrinsic calibration tool and calibrate the results. Save them to profile and then load them here.

• Or users can also load the JSON file.

9. Upload images are taken from the external camera for the left view.

10. Detect the checkerboard corners for all the left view external camera images.

• Click on detect corners to get the checkerboard corners auto-detected.

• Or else manually add the border corners to get all checkerboard corners.

11. Front and rear wheels are auto-detected.

• Wheel points are auto-detected, users can view the markers by selecting undistorted images.

12. Upload images taken from the external camera for the right view.

13. Detect the checkerboard corners for both the right view external camera images.

• Click on detect corners to get the checkerboard corners auto-detected.

• Or else manually add the border corners to get all checkerboard corners.

14. Front and rear wheels are auto-detected.

• Wheel points are auto-detected, users can view the markers by selecting undistorted images.

15. Click on the run calibration button. This takes all the input configuration and the file data to get the calibrated results.

16. The Top right bar shows the extrinsic parameters.

17. Visualize button shows the 3d representation of the car and its wheels. Along with the camera center and its frustum.

18. Export option helps the user to export the calibrated data for the mounted camera with the vehicle.

19. Users can check the error stats and add more images to see the change in error stats.

• Reprojection Error: Its value is the mean delta of the marked wheel point and the reprojection of the calibrated wheel position. It's measured in pixels.

• Translation Error: Its value is the mean delta of the distance between the ray produced from marked wheel points and the calibrated wheel position in 3d space. It's measured in meters.

Save calibration dataset:

We have a save option in the top right corner. A user can click on the Save button to save the calibration dataset at any time during the calibration process.

Extrinsic Calibration Output:

• roll, pitch, and yaw are in degrees and px, py, pz are in meters.

• roll, pitch, and yaw, px, py, pz are the extrinsic parameters downloaded from the calibration tool.

• vehiclePoint3D is the 3d coordinates of a point in the vehicle coordinate system.

• imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Overview: Given the initial state (bounding box) of a target in the first frame of a video sequence, the aim of Visual Object Tracking is to automatically obtain the states of the object in the subsequent video frames.

How is this used for making labeling easy?

2d VOT uses computer vision algorithms to predict the bounding box of an object in all frames given the initial manual box in a particular frame. The following action path is used to leverage this capability for 2d labeling.

1. Draw a bounding box on the required target object.

2. Apply 2d VOT to predict bounding box in subsequent and previous frames and draw them in the canvas.

3. User iterates through each frame to verify the predictions and make adjustments as necessary.

Steps to track an object:

1. Draw bounding on the required target object

2. Uncheck the “Fill new labels” option and click on the “Track label” button

3. Labels will be automatically visible on the timeline once Tracking is complete.

4. If tracking is not complete and the user will see a warning message stating the same, try clicking on “Track Label” again to resume progress.

Tips on using Track label

• For each “Track label” click, tracking is done for the selected label from the active frame in both directions.

• Predicted labels do not overwrite manual labels and stop updating frames in both directions once a manual frame is encountered in the timeline.

• Recommended using for objects that are visible across many frames to avoid the overhead of re-adjusting in that many frames.

• Predicted labels take into consideration the scale change of the object and not the orientation changes of the object. Hence, if an object changes its orientation drastically from the initial state, then it is recommended to re-adjust the frame at the beginning of the new orientation and apply it to track again.

Calibration Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration selection

Select Vehicle-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting Vehicle-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

Select the Terrain as Flat and approach as Target

Configuration

Vehicle configuration

For more details, click here

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Target configuration

• Horizontal corners: Total number of inner corners from left to right. The blue dots shown in the above preview correspond to the horizontal corners.

• Vertical corners: Total number of inner corners from top to bottom. The red dots shown in the above preview correspond to the vertical corners.

• Square size: It is the length of the arm of the square in meters. The square size corresponds to the length of the yellow square highlighted in the preview.

• Left padding: The distance from the leftmost side of the board to the left-most corner point in meters. Corresponds to the left blue line in the preview.

• Right padding: The distance from the rightmost side of the board to the rightmost corner point in meters. Corresponds to the right blue line in the preview.

• Top padding: The distance from the topmost side of the board to the topmost corner point in meters. Corresponds to the top red line in the preview.

• Bottom padding: The distance from the bottom-most side of the board to the bottom-most corner point in meters. Corresponds to the bottom red line in the preview.

Upload files from the mounted camera

Detect target corners in images

Users can click on Detect corners to detect the corners in the target. This is an automated process, and our algorithm usually detects the corners in the image accurately.

Add target configuration for images

For each image, enter the following target configuration

1. Distance from Vehicle Reference Point (VRP) to Intersection Reference Point (IRP).

2. If the board is placed perpendicular to the ground and on the ground directly, the target height should be 0. If the the board is placed perpendicular to the ground, but above ground level, the target height should be the distance from the bottom edge of the board to the ground level. If the board is parallel to the ground, the target height is the thickness of the board itself.

3. Distance from Intersection Reference Point (IRP) to Target Reference Point (TRP).

Run Calibration

Users must click Run calibration to calculate the extrinsic parameters and error stats.

Visualizer

Visualize button shows the 3d representation of the car and its wheels, along with the camera center and its frustum.

Error stats

Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. The closer the error stats to zero, the better the extrinsic parameters.

• Translation Error: The distance between the centroids of the 3d projections of target corners and the target configuration in the Vehicle coordinate system.

• Rotation Error: The angle between the planes formed from 3d projections of target corners and the target configuration in the Vehicle coordinate system.

Extrinsic Calibration Output

• roll, pitch, and yaw are in degrees, and px, py, and pz are in meters.

• vehiclePoint3D is the 3d coordinates of a point in the vehicle coordinate system.

• imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Camera coordinates system

We currently show three different types of camera coordinate systems. The extrinsic parameters change according to the selected Camera coordinate system. The export option exports the extrinsic parameters based on the selected camera coordinate system.

• Optical coordinate system: It's the default coordinate system that we follow.

• ROS REP 103: It is the coordinate system followed by ROS. On changing to this, you can see the change in the visualization and the extrinsic parameters.

• NED: This follows the north-east-down coordinate system.

Vehicle Coordinate Frame

The origin of the vehicle coordinate frame is the midpoint of the line joining the rear wheel centers on the ground.

1. X-axis points in the vehicle's forward direction

2. Y-axis towards the left of the vehicle and

3. Z-axis pointing upward

Data collection

• For the targetless calibration, users must record a scene in the camera and the LiDAR sensor.

• No target is required for this calibration, but the scene should have vehicles (cars and vans) in the camera and the LiDAR data.

• For better calibration, vehicles should be close to the LiDAR (10m - 50m) with a good number of points in LiDAR and present on both the left and right sides of the image; Having too many vehicles may result in calibration errors.

• Ensure that the vehicles (including the ego) stop or move slowly. This reduces the effect of the time difference between LiDAR and the camera.

• Select 3-4 frames from the collected data that have vehicles on both sides of the images and vehicles visible in the lidar pcd.

Calibration Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration selection

Select LiDAR-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting LiDAR-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

Users can choose either the target-based or the targetless calibration. The target-based calibration uses the checkerboard/charucoboard as the calibration target, and the targetless calibration uses the scene captured in both LiDAR and the camera sensor data.

Configuration

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Upload files from LiDAR and Camera

Add point cloud files from the LiDAR and images from the camera sensor. After adding, pair the point cloud files with the matching image files before continuing.

Sample CSV format

X, Y, Z
0,-0,-0
62.545,-3.5064,-3.5911
62.07,-3.5133,-4.1565
32.773,-1.8602,-3.4055

Estimated extrinsic parameters

Mapping of corresponding points

To get the initial estimates, users can map any four corresponding points from the image and the point cloud data.

Manually enter extrinsic parameters

Alternatively, users can add the initial estimates if they know them. In such a case, users can skip manually adding the markers. Users can click Add estimated extrinsic parameters to add the initial estimates.

Verifying the accuracy of the estimated extrinsic parameters

Once the estimated extrinsic parameters are in the tool, users can visualize the parameters by clicking on the visualize button. In the visualization, we have a few sensor fusion techniques through which the accuracy of the extrinsic parameters can be visualized. For more details, visit Sensor fusion techniques.

Estimated extrinsic parameters are crucial in generating accurate extrinsic parameters.

To get good initial estimates, users must clear the markers and redo the markings if the estimated parameters are way off.

Segmentation

There are two types of segmentation approaches available for the user to select:

Auto segmentation

1. This approach automates the segmentation of vehicles in point clouds and images using a deep learning model trained on various datasets.

Manual segmentation:

1. Lidar: In this approach, the user needs to add bounding boxes in the lidar frame and fit the boxes to vehicles in the point cloud. The bounding boxes must be added for all the point clouds uploaded for calibration. This can be done by selecting the Bounding box mode, adding bounding boxes, and clicking Save Labels.

2. Image: There are two ways to do manual segmentation

   1. Semantic Painting: Users can use the brush to paint the vehicles in the image and click on Save Labels.

   2. Segment anything: In this approach, Users place a cluster of points on each vehicle. The same vehicle points should be placed under the same category. Please place at least one point on each surface of the car, such as the windshield, sides, roof, etc., so that when the model runs, it doesn't miss any part of the vehicle. After placing the points in each image, please click on the save label to save the data.

Note: Auto segmentation is suggested initially. Based on the segmented vehicles in the point clouds and images, the user can decide whether to proceed with auto-segmentation or perform the segmentation manually.