Our powerful and user-friendly data annotation platform is available as a managed service, in the cloud, or on-premise. SaaS Offering: Host your data on Deepen servers and receive automatic updates and access to all features. Hybrid On-premise: Host your data on your own servers and provide Deepen a URL to an asset that is behind a VPN. All source data will be unreachable by Deepen but generated assets(annotation data) will be stored on Deepen servers. Full On-premise: Work with our engineering team to deploy

Our data management workflow is designed to help managers manage a team of annotators in a seamless manner, yet produce high-quality results. It includes features such as task assignments on a user level, multi-stage quality assurance down to each label, and exporting your labeled data with just a click.

Overview: Create, configure, and upload your dataset to your workspace.

Steps to create a dataset:

1. In your workspace, click on “Create dataset” in the top right corner.

2. Enter a dataset name (use alphanumeric characters only. Special characters are not allowed).

3. Select a dataset type (Images/Video/3D Point Cloud).

4. Select Dataset Format (JSON) when the 3D point cloud is selected.

5. Select Frames per second when Video (only .mp4 files are supported) is selected from the Dataset type.

6. Select a labeling mode (Individual/Sequence).

7. Select a dataset (Only Zip files are allowed) by clicking on browse from the computer, or drag and drop the dataset.

8. Import dataset configuration from profile (if any profile is already created).

9. Categories and Attributes can be configured if there is no Dataset Profile

10. Enter Labelling Instructions if required.

11. You can also manage your team by assigning them certain permissions like Labeller, Labels Admin, or Reader for Local Users or Local Groups.

12. Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of the pipeline as you want.

13. Enable 2D Tools in 3D Editor (only for 3D datasets) to label 2D annotations for the 3D datasets.

14. Users can enable customer review/Enable auto-task assignments while creating the dataset if needed.

15. Add a new tag or select an existing tag to find the dataset easily.

16. Click on Create to create a dataset.

17. Once created, you can see the dataset with the “Pending upload” status and the file count after finishing the upload process.

18. Refresh the browser once the data is uploaded.

Here is the 3D input format for uploading a dataset:

How to add a user:

1. When a workspace is opened there is the Users option on the left side panel.

2. Click on the Users option and at the top right click on Add user.

3. Enter valid email and select the scope for the user and click on Add User.

Permission rights of each user: Deepen AI annotation tools have different user access levels for different tasks. Different roles and their description are as follows:

1. Admin: This role can view/add/edit/delete a project, add/delete a dataset, add users, add payment details, sign up for different pricing plans, assign users, add files and add labels.

2. Reader: This role can view all the labels, reports and profiles but cannot make any changes.

3. Datasets Admin: This role has access to create and delete datasets. They can assign frames to already added users. They cannot label or add any users.

4. Datasets Creator: This role has access to add datasets, but cannot delete datasets.

5. Labels Admin: This role can add and delete labels.

6. Labeller: This role has access to annotate the allocated frames (Labelling and QA)

7. Customer: This role has permission to accept or reject labels in the workspace.

Apart from these, we have additional scopes for Access Tokens. They are:

1. Dataset Labels Admin: This role can add and delete the labels on the dataset level only.

2. Dataset Labeller: This role has access to annotate the allocated frames(Labelling and QA) on the dataset level.

3. Dataset Calibration Edit: This role can view, create, manage and delete calibration datasets at the dataset level.

4. Dataset Calibration Read: This role can view all at the dataset level for calibration dataset.

5. Dataset Reader: This role can view all the labels, reports and profiles but cannot make any changes at the dataset level.

6. Dataset Customer: This role can accept labels at the dataset level.

Overview: Creating a group allows you to create a group within a workspace.

Creating a Group: Users can create a group by clicking on the create group on the Users page and add other users to it.

User Permissions: Permissions can be changed at the datasets levels for different Users by using user permissions.

User Group: Permissions can be set for a group of users by selecting a group and every user has the same permissions for the selected datasets.

Overview: Users can export Labels/Dataset/Profile using the export button.

Overview: Share your labeled dataset on your own site

Steps to embed link:

1. To enter the embed labeling screen, go to “Share” on the dataset page.

2. Enter the title and number of frames you wish to display. A unique embedding link wrapped around <iframe> tag will be automatically generated.

3. Paste the link in your HTML or JavaScript editor.

Overview: Users can add different type categories and add attributes at label and category levels.

Steps to create a dataset profile:

1. Click on the “dataset profile” at the left panel.

2. Create a profile name e.g.: My Image profile.

3. When configuring categories, choose a label type. Then, create a category by typing the name and click enter the box below.

4. For 3D bounding boxes, you can configure your default bounding box size by clicking on the category.

5. You can also add the label attributes specific to the category when you select a category.

6. Common attributes are attributes that apply to all categories.

7. Frame Attributes are attributes that apply at a frame level.

8. Dataset Attributes are attributes that apply at the dataset level.

9. Auto-task assignments can be enabled by clicking on Enable auto-task assignments.

10. Next, enter the labeling instructions (optional).

11. Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of pipelines as you want.

12. In the Image Editor user can give the attributes to the labeled objects by pressing shortcut keys or from the tool itself.

Overview: Import Profile allows you to import an existing dataset configuration from a dataset to another dataset

You can find the import label feature at the top right under the 3 dot dropdown in a dataset.

Steps for Importing Labels:

1. Click on Select Profile and select an existing profile that is already created.

2. Click Next and Customise profile.

3. Once the above step is done click on Next: Finish and Click on Import config.

4. You can view your imported labels in the label set you selected on the labeling screen.

General FAQ:

1. What is Deepen AI? Deepen is an industry-leading computer vision training data solution with fast AI enabled labeling tools, labeling automation, human workforce, and data management.

2. Does Deepen only sell labeling services? Deepen sells both AI-enabled labeling tools and labeling services, which includes: -An image and video annotation tool and services -Multi-sensor (Lidar, Radar, Sonar, Camera and more) fusion annotation tool and services -Quality and performance management tools.

3. How do I set up a Deepen project? Once you have created a Deepen account, follow the simple steps on this page to set up a project.

4. Do I need to be an AI engineer to use Deepen tools? No, we designed our user-friendly platform to require minimal technical knowledge and expertise such that anyone can annotate data using our out-of-the-box web application. However, it may be beneficial to have someone experienced in data science on your team to help ensure that your machine learning project is successfully developed and deployed.

5. What is the difference between Deepen 2D and Deepen 3D? Deepen 2D includes annotation capability for camera data such as images and videos. Deepen 3D supports multi-data annotation which includes the camera, LiDAR, radar, and more.

6. Which internet browser should I use? Google Chrome will be recommended when using Deepen tools.

Overview: Access token is created for admin to access different APIs in your specific workspace.

How to create an access token:

1. Go to the left panel, under "Developer settings" click on "Access Token".

2. Click on "Generate new token" on the top right of the screen to start generating a token.

3. Fill in your note and select the scope of users allowed to access the token.

4. Click on "Generate token" to create token access.

5. This token will not expire, you have to revoke the token access manually.

How to view/revoke created token:

1. Once the token is created, go to "Developer Tokens" to view the token.

2. You can click on the token to copy it and click on the "revoke" button to revoke the token manually.

We're introducing an improved lifecycle for the tasks in the dataset to streamline your dataset workflow experience. Here are the new task states we support:

Waiting: Indicates the task is in waiting state and needs to be submitted from a previous stage

Ready: Indicates the task is ready for you to start working on

In progress: Indicates the user is working on the task

Submitted: Indicates the task was submitted for review, and the next stage user has to verify and accept it. Please note that you will not be able to modify a task once submitted.

Accepted: Indicates the task was verified by the next stage user and has approved it and no changes would be further required in it. Please note that you will not be able to reject a task once accepted.

Rejected: Indicates the task was rejected by the next stage user and needs to be fixed (users can refer to issues if any were created for the task before rejecting the task). Rejected tasks need re-work and have to be submitted again for further review.

Note: Once a task is submitted user can't edit the task in that stage.

Task options you see for different possibilities:

• If you are in a stage after the first one (e.g. second, third)

  • If the previous stage was not accepted yet, you see Accept & modify, Accept & submit and Reject
• If you are in the last stage of the pipeline

  • Once you've accepted the previous stage task, you see Finish.

New tasks page inside a dataset

Users with admin scopes and customers will now have a new tasks screen visualizing the flow of the work in the dataset across all stages. Only users with dataset admin or above scopes will be able to assign users to the tasks.

When clicked on a task in the Task column (left-most column), the task is launched in the editor with all the required options set automatically, so you can start working without worrying about which stage you have to go to or which label set you to need to set to.

The Deepen reports offer a complete view of the dataset management environment. They track important aspects such as data quality, user contributions, labeling progress, and real-time activity within the workspace. This information is vital for making informed decisions and optimizing the workflow for dataset-related tasks.

Dataset Reports:

The dataset report provides information about the datasets in the workspace. It includes details such as the number of frames, labels, time spent on the dataset, average time across all stages, and completed tasks.

User Reports:

The user report gives information about the users working on the dataset. It provides details such as the total number of labels created and the average daily time spent.

Quality Reports:

The Quality report relates to issues that arise within the dataset, providing information on the status of each issue.

Labelling Reports:

The Labelling report provides information about the labeling categories, label type, label count, and stage ID.

Real-Time Reports:

The Real-time report provides real-time updates about the current workspace. It allows users to track the number of users who are working on the workspace.

Productivity Reports:

The productivity report provides information about the user's productivity across all label types and stages.

How to enable auto-task assignments for datasets?

Users with admin permission can go to the tasks screen in the dataset details page, and turn on the auto-task assignment feature using the Auto-assign option. Users can also set an assignment config for each stage by clicking on the menu settings icon beside the Auto-assign option.

How does this work? When auto-assign is enabled. Users having labelling permission can go to the tasks page at the workspace level and send a request for work by clicking on the Request work button. once the user requests work, tasks will be assigned to the user.

Overview: Users can export labels using export labels by using different methods. Steps to export data:

1. Labels that are done can be downloaded from the tool in JSON format.

2. Only labels which are marked as done are exported as JSON files.

3. Labels: Labels in the last pipeline stages will be downloaded. (Files should be marked as done)

4. Labels by file names: Download labels from the last pipeline stage for all label types, with the file name as the key in JSON format.

5. Customize your export: Files can also be downloaded by selecting particular frames, particular annotations, categories, attributes, types etc. (Files need not be marked as done if labels are downloaded using these).

6. 2D Semantic labels can be downloaded by clicking on the 2D Semantic Segmentation option.
7. Video for downloading paint labels https://drive.google.com/file/d/1EUzTqbRlyW85ZMgOJS74jCPTXk70vhH0/view?usp=sharing

Note: 2D Semantic segmentation output can be found here

Overview: Users can now perform instance segmentation tasks in polygon mode for single-frame 2D image datasets with AI assistance. In order to use the feature, the user needs to follow the steps below.

Creating the Segmentation Mask

• Go to Polygon mode

• Choose the “2D Segmentation” button

• Draw a tight box enclosing the object to be segmented

• AI server will return an output polygon mask covering the object

Refining the Segmentation Mask

Users have an option to edit/refine the masks using the pre-existing tools in the polygon labelling mode or they can use the AI assistance to refine the polygons automatically. To use AI assistance, the user needs to follow the steps below:

• To activate AI refinement mode, select the label which was created using the “2D Segmentation” button.

• Choose the “Refine Polygons” button (pencil) besides the “2D Segmentation” button.

• In the refinement mode, the user can plot two kinds of points as below.

  • Green indicates the regions which the user wants the model to include in the segmentation mask.

  • Red indicates the regions which the user wants the model to exclude from the segmentation mask.

• The user can plot Green points on the image using “Left Mouse Click”.

• Points To add the Red points, the user can use the “Left Mouse Click” with “Alt” hotkey.

• As soon as any of the above points are added, an API call will be made to refine the points.

• The segmentation will be refined using the input provided and the polygon will be updated accordingly. The refinement mode will be deactivated once the polygon is refined. You need to activate the refinement mode again using the “Refine Polygons” button in the secondary menu bar to make further edits.

• To add multiple points at once before refining the polygon, the user can use the "Shift" hotkey along with the process described above. The last point in this scenario needs to be added with “Shift” hotkey released for the polygon to be refined.

• The user can delete existing points using “Ctrl” + “Shift” hotkeys along with "Left Mouse Click" on the point to be deleted.

Assessments are designed to automatically evaluate the proficiency of users, generate a score, and produce a comprehensive report. At present, we provide support for automatic evaluation of 2D bounding boxes. However, we are committed to expanding our support for additional label types soon. To create an assessment, follow the below steps:

1. Open the datasets page and click on “More”.
2. Click on “Assessments” and then “New” to create a new assessment.
3. Enter the assessment name, attendees, passmark, IoU matching threshold, time limit, and maximum attempts.
4. Select a benchmark dataset that has labels and is marked as “Done”.
5. Once the assessment is created, Users can launch the dataset and start the assessment.
6. Annotate the bounding boxes and submit the assessment once all the labels are annotated.
7. You can go back to the assessment page to view the results. If you fail the assessment, you can reattempt it.
8. To compare the assessment results with the benchmark dataset, the assessment creator must click on the view results button.
9. The results show missing objects and false positives.

2D Label Types:

3D Label Types:

Overview: Polyline or Line annotations are continuous lines made of one or more line segments. The entire line is treated as one object. Polylines are typically used to annotate open shapes since they don’t have to be closed as polygons do. This tool is often used to label shapes like road lane marks or other similar shapes.

Steps to Draw a Line:

1. Click on Label type and select Line from the top left menu of the editor.

2. Select a category from the drop-down box.

Semantic Segmentation

Overview: Semantic segmentation is the task of classifying each and every pixel in an image into a class or category. 1. Select an instance or semantic segmentation by clicking “Label” at the top panel

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the semantic if the objects are touching each other than their drawn polygons are merged with each other with a single id. 4. In the below image, you can see the merged polygons with a single id

Instance Segmentation:

Overview: Identify each object instance of each pixel for every known object within an image.

1. Select an instance or semantic segmentation by clicking “Label” at the top panel.

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the Instance segmentation, each object can be labelled distinctly with a unique ID. 4. In the image below, you can see the object’s polygons with different IDs like Car1, Car2 and Car3.

Overview: Painting foreground and background allows you to segment an object occluded in the background

Steps to segment foreground or background:

1. Knowing how to use the foreground and background are very useful in data labeling.

2. Start labeling by using left-click.

3. When we label the 1st object which was occluded by another 2nd object. It should be labeled with an extra region of the object.

4. By using the foreground option for the 2nd object which was in front of the 1st object. It will cut the 1st object polygon accurately.

5. By using the background option we can label an occluded object without affecting the polygon in front of the occluded objects

6. Click “control + shift” to draw a Polygon within a polygon.

Overview: Labels are generated automatically based on the image, this is also known as pre-labeling.

Steps for using AI Sense: 1.Go to the Editor Page and Click on the AI Sense button from the right side menu. 2.Click on Auto Detect. 3.Click on Preview to view the labels or Add All to add the labels. 4.Users can add individual labels or use Confidence Score to detect labels. 5.Make sure the category is given before using AI Sense.

Categories supported for AI Sense: Single Frame Detection categories:(2D Bounding Boxes)

1. person: A person walking in the scene (usually on sidewalks, crosswalks, outside driveable space

2. bicycle: Human or electric-powered 2-wheeled vehicle designed to travel at lower speeds either on the edge of the road surface, sidewalks, or bike paths. A driving bicycle is also considered a bicycle.

3. car: All Sedan, Suv, minivans, and sports cars are marked as cars

4. motorcycle: Gasoline or electric-powered 2-wheeled vehicle designed to move rapidly (at the speed of standard cars) on the road surface.

5. airplane

6. bus: A road vehicle designed to carry many passengers.

7. train

8. truck: Motor vehicles designed to transport cargo, goods, merchandise, and a wide variety of objects.

9. boat

10. traffic light: A road signal for directing vehicular traffic by means of coloured lights, typically red for stop, green for go, and yellow for proceeding with caution.

11. fire hydrant: a fitting in a street or other public place with a nozzle by which a fire hose may be attached to the water main.

12. stop sign: A stop sign is a traffic sign designed to notify drivers that they must come to a complete stop

13. parking meter

14. bench

15. bird

16. cat

17. dog

18. horse

19. sheep

20. cow

21. elephant

22. bear

23. zebra

24. giraffe

25. backpack

26. umbrella

27. handbag

28. tie

29. suitcase

30. frisbee

31. skis

32. snowboard

33. sports ball

34. kite

35. baseball bat

36. baseball glove

37. skateboard

38. surfboard

39. tennis racket

40. bottle

41. wine glass

42. cup

43. fork

44. knife

45. spoon

46. bowl

47. banana

48. apple

49. sandwich

50. orange

51. broccoli

52. carrot

53. hot dog

54. pizza

55. donut

56. cake

57. chair

58. couch

59. potted plant

60. bed

61. dining table

62. toilet

63. tv

64. laptop

65. mouse

66. remote

67. keyboard

68. cell phone

69. microwave

70. oven

71. toaster

72. sink

73. refrigerator

74. book

75. clock

76. vase

77. scissors

78. teddy bear

79. hair drier

80. toothbrush

Sequence tracking model categories:(2D Bounding Boxes)

1. car

2. truck

3. person

4. bus

5. bike

6. rider: A person driving a bicycle or motorcycle is considered as the rider

7. motor

8. train

Categories for 3d Bounding box (Sequence and Single Frame):

1. vehicle: All Sedan, Suv, minivans, and sports cars are marked as cars

2. pedestrian: A person walking in the scene(usually on sidewalks, crosswalks, outside driveable space)

3. cyclist: A Person who is riding a Cycle

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

1. Click on Label type and select Semantic painting from the top left menu of the editor.

2. Select a category/class from the dropdown.

3. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon) to start painting.

4. Selecting Brush/Polygon from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon”.

5. Use mouse left click to paint the image.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You need to save by clicking on Paint at the top panel or click on save or use Enable Autosave.

8. To erase, click on the eraser icon to erase the paint.

Note: The output file can be downloaded by clicking Export ⇾ Labels ⇾ 2D Semantic Segmentation.

Overview: By using 2D Editor users can annotate 2D Bounding Boxes,2D Polygons, 2D Lines, 2D Points, 3D bounding boxes and Semantic painting for the camera images.

Primary Toolbar: It has Home, Select and Move Tool(V), Pan Tool(2), Select and Move, Zoom In, Zoom Out and Create Issue.

• Home Button: Home button is used to navigate to the Homepage.

• Select and move tools(V): Users can move or edit labels

• Pan(2): This Option is used for moving images. Users can’t label images if this option is selected.

• Zoom In: Users can Zoom in by selecting this option.

• Zoom Out: Users can Zoom Out by selecting this option.

• Create Issue: Users can create issues at the frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar. More information on issues can be found here.

• The Top toolbar consists of Frame, Label, View and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete. Polygon annotation additionally has Attributes, Polygon Segmentation Mode and Polygon Drawing Mode.

View: View consists of Canvas view, Show Labels Text, Show Labels Attributes, Grid Size, Show Grid, and Show Merge Options for Polygons.

Secondary Menu bar: It consists of Label Set, Stage, Label type, Point size, Labelling Instructions, View Editor, View Original Image, View Masked Image(only for Polygons) and Reset View.

Panel Content: It is on the right of the image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these it also has Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular file.If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on the workspace the tool navigates the user to the workspace page in the tool.

Buttons in the lower navigation bar and their usage:

• Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.

• Preload Data: It loads the frames in advance before opening them.

• User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

• User can hide the bottom bar using the Hide bottom bar option for sequence annotations.

Overview: A scene label is a behaviour or an event or an action by an object or a scene in the sequence.

Steps to enable scene labeling:

1. To start enabling scene labeling, upload a dataset as a sequence datasets or video that is not split into frames. By checking the "use video directly instead of splitting into frames" option, it will not the video into frames and rather upload the entire video.

2. Create a category under scene labeling. Ex: cut in

3. Clicking on the New Scene Label button at the bottom of the editor creates a new label from the user's current frame with the selected category. Ex: cut in:1

Steps to create a scene label:

1. Click on the label name or track selects the label and will enable the "Start Scene" and "End Scene" buttons.

2. Navigating to the desired frame and clicking on the "End Scene" buttons will extend the track from the already existing start point to the selected frame.

3. Upon selecting the frame number below the already existing frame and click on the "Start Scene" button extends the track to the left towards the user's new start frame.

4. Click on label or track selects the labels which enable the user to click on the edges of the track and drag towards the user's desired frame.

Segment Anything Model (SAM): A new AI model that can “cut out” any object, in any image, with a single click.

Steps to perform Segment Anything in semantic segmentation:

1. Click on Label type and select Semantic Painting from the top left menu of the editor.
2. Click on Paint and select Paint using Segment Anything.
3. Users can label an object by selecting either Points or Box. The more points that are added, the higher the accuracy of the output. For better accuracy, it is advisable to place more points in the center of the object.
4. After selecting the point/box, click on the Generate label to preview the results. The user can either accept or reject the label.
5. Click on Paint and select Save to save all the changes made. It is important to note that for Semantic painting, users must manually label by clicking on Save.

6. Users can overwrite the painted region by selecting Overwrite painted region.

7. Users can unselect the Paint using the Segment Anything option and make edits using brush/polygon.

Overview: The Polygon tool is used to create geometric annotations by drawing lines between points.

Steps to Draw a Polygon

1. Click on Label type and select Polygon from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a polygon, keeping points on the image with a left mouse click. You can label the same polygon ID by holding an “Alt” key for the first polygon point or by selecting a label and clicking on Draw selected polygon. Then continue the polygon labeling for the object with the same ID.

4. You can select the label by clicking on the label with a left mouse click. Double-click on a polygon to edit it by dragging the highlighted points of the line with a left mouse click and also you can delete the selected highlighted point by pressing the “Delete” key. To add a point on the line, click on the line using the left mouse click.

Polygon (Foreground/Background): Knowing how to use the foreground and background can come in handy in data labelling for polygons. If there is an object at the back of another object, that is when this feature comes into play. Draw a polygon as Foreground:

1. Selecting a foreground option from the image editor page

2. For example, if you want to label a “Car” in the below image without interrupting the “Building” polygon, you want to select a Foreground option before saving the “Car” label as the Building is the Background and car is the foreground object.

3. If you use Background instead of Foreground option, it will not be cut by the Background polygon label(Building).

Draw a polygon as Background:

1. Selecting a Background option from the image editor page.

2. For example, if you want to label a building in the below image without interrupting the car polygon, you want to select a background option before saving the building label because the Building is the Background and the car is the foreground object.

3. When you label a background object with a background option, it will get a polygon accurately to the object without disturbing the foreground label (Car). 4. If you use the foreground instead of the background option, it will interrupt the foreground polygon label (Car).

Overview: Given the initial state (bounding box) of a target in the first frame of a video sequence, the aim of Visual Object Tracking is to automatically obtain the states of the object in the subsequent video frames.

How is this used for making labeling easy?

2d VOT uses computer vision algorithms to predict the bounding box of an object in all frames given the initial manual box in a particular frame. The following action path is used to leverage this capability for 2d labeling.

1. Draw a bounding box on the required target object.

2. Apply 2d VOT to predict bounding box in subsequent and previous frames and draw them in the canvas.

3. User iterates through each frame to verify the predictions and make adjustments as necessary.

Steps to track an object:

1. Draw bounding on the required target object

2. Uncheck the “Fill new labels” option and click on the “Track label” button

3. Labels will be automatically visible on the timeline once Tracking is complete.

4. If tracking is not complete and the user will see a warning message stating the same, try clicking on “Track Label” again to resume progress.

Tips on using Track label

• For each “Track label” click, tracking is done for the selected label from the active frame in both directions.

• Predicted labels do not overwrite manual labels and stop updating frames in both directions once a manual frame is encountered in the timeline.

• Recommended using for objects that are visible across many frames to avoid the overhead of re-adjusting in that many frames.

• Predicted labels take into consideration the scale change of the object and not the orientation changes of the object. Hence, if an object changes its orientation drastically from the initial state, then it is recommended to re-adjust the frame at the beginning of the new orientation and apply it to track again.

Overview: The key point annotation is used to detect shape variations and small objects, it helps to understand each point motion in the targeted object.key points can help in facial recognition and also is used to detect human body parts and estimate their poses with the right accuracy.

Steps to Draw a Point:

1. Click on Label type and select Point from the top left menu of the editor.

2. Select a category from the drop-down box.

3. You can label the points by left-clicking the mouse.

4. You can select the point by clicking on the point with a left mouse click. Then you can edit the point, by dragging the highlighted point with a left mouse click.

5. You can draw another point by pressing the “Esc” key twice.

6. Labels are saved automatically.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

User can add a 3D bounding box to the image by selecting the 3D bounding box label Type.

Steps to Draw a 3D Bounding Box on image:

1. Click on Label type and select the 3D bounding box from the top left menu of the editor.

2. Select a category from the drop-down box.

3. You can label the 3D bounding box by left-clicking the mouse. On mouse left click and drag user can see a 2D box which will be the front face of and another click will generate a corresponding 3D bounding box.

4. Hovering on the box reflects which face/vertex will be moved and you can use Shift + Left Click-drag to move the box.

5. You can select the 3D bounding box by clicking on the box with a left mouse click. Then you can edit the bounding box by dragging the highlighted box with a left mouse click.

6. Labels are saved automatically.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

Users can now stream data from their cloud buckets directly into the app without having to upload it. Whatever state of files users have in the bucket is automatically synced with the corresponding files in users datasets. With this now, users can also add or delete files from datasets by simply adding/deleting them in your bucket.

Users can create a Data stream by clicking on create Data stream button on the data streams page.

Images and Point clouds(JSON Format) are supported as of now, and we are working on adding more file type supports.

User needs to choose which cloud provider they need and then provide few details which are bucket name(bucket where your data is stored), folder name(location of the folder that will be synced to create a dataset present within the bucket), credential file(a json file with the private key of one user that can enable us to authenticate to your bucket location) and optionally user can provide a dataset configuration profile which will be applied to all the datasets that are created using this data stream configuration.

Once a data stream configuration is created, you can click on the "Sync bucket" option to sync your datasets with your bucket instantly. Post that you can use this dataset like normal datasets and it has all the native functionalities that are already supported by Datasets.

Instructions for creating a 3D bounding box within a sequence dataset:

1. Click on Label type and select “3D bounding box” from the top left menu.

2. To create a Bounding Box: Click on Create Tool (C) from the left-hand sidebar to start labelling. Note that this mode only adds bounding boxes.

3. To Edit Bounding Box: Users are only able to edit boxes when using the “Select and Move Tool(V).”

4. To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the space of the box to move the bounding box to the desired direction.

5. To Rotate the Bounding Box: To rotate the bounding box, hold down the “R” key and click the left mouse button. Then, click on the “Rotated arrows” icon.

6. To Increase or Decrease Bounding Box: To enable scaling (increasing or decreasing box size) action, hold “S+Left Click”. Now click “Circle” to adjust the bounding box in various directions.

7. To Delete the Bounding Box: Select the label and press Shift + Delete to remove the label from the sequence, or click on the trash can icon of the label in the right panel to delete a specific key frame or a manual frame.

8. Refer to understand the Timeline feature in sequence labelling.

Overview: 3D Editor is used to label 3D Bounding Boxes, Polygons, Instance & Semantic paint and 3D Lines for Point Cloud.

Primary Toolbar: It is on the left side of the image editor which has Home, Create Tool(C), Select and Move Tool (V), Select and Scale Tool(S), Select and Rotate Tool(R), Orbit Tool(1), Pan Tool(2), Dolly Tool(3) and Create Issue.

• Home Button: Home button is used to navigate to the Homepage.

• Create Tool: The user needs to click on Create Tool to add any new labels. Labels can't be created without selecting this option.

• Select and Move: The user can label frames using this option. The user can’t label if this option is not selected.

• Select and Scale Tool: Selects labels and activates a widget to scale them in 3D space.

• Select and Rotate (R): Selects labels and activates a widget to rotate them in 3D space.

• Orbit Tool(1): Click and drag on it to rotate the view around the lidar.

• Pan Tool(2): Click and drag to pan the lidar up, down, left and right.

• Dolly Tool(3): Click and drag to move the lidar forward and backwards.

• Create issues: Users can create issues at frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar.

Primary Menu bar: It is on the top of the image editor. It consists of Frame, Label, View, LiDAR, and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete, Fit Label to Points, Set box on ground by default, Set box on ground, Enable one click placement, Rotate, Rotate 180 in all frames, Propagate Labels(Only for Single frames) and Attributes.

View: View consists of Show Labels Text, Show Labels Attributes, Show selected Label only, Show Bounding Boxes, Show Polygons, Show Ground Mesh, Uncolor Ground Points, Show Lidar Cube, Show Grid, Enable Secondary Views, Enable Camera Views, Perspective, Orthographic, Light Theme, Dark Theme, and Reset View.

LiDAR: LiDAR consists of Enable Point Intensity, Color By Z, Show Colored Points inside Box, Show Black Points(Only for coloured lidars), Ground, Cloud Radius and Lidar FOV(Horizontal and Vertical).

Secondary Menu bar: It consists of a Label Set, Labelling Pipeline, Label category, point size, Labelling Instructions, Reset View. Additionally, for sequence annotations, we have Fused cloud and Z-threshold along with these.

Panel Content: It is on the right of image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these we also have Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular image. If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on Dataset the tool navigates the user to the dataset page in the tool.

• User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

Buttons in the lower navigation bar and their usage:

• Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.

• Preload Data: It loads the frames in advance before opening them.

• User can hide the bottom bar using the Hide bottom bar option.

• This is only for sequence datasets.

A timeline feature is a useful tool for visualizing your labeling activity and improving labeling speed and efficiency for sequence datasets. By providing information about the labels across datasets, it allows you to easily identify, navigate, and edit manual frames. With the timeline, you can perform these operations much faster and with greater accuracy. For more detailed information on how to use the timeline, please refer to the instructions below.

Frame Navigation

Using Frame Number

To navigate across frames using the frame number, you can click on the frame number and input the desired frame. Alternatively, you can hover over the frame number and drag left or right to move to the previous or next frame.

Using Frame Slider

You can navigate across frames in the timeline view by dragging the blue slider left or right.

Using Navigation Buttons

You can navigate across frames using the play, previous, and next buttons located on the timeline view. See the button interactions below.

You can use the ctrl/cmd key with mouse scroll to zoom in or zoom out the scale present in the timeline view.

Extend/Clip Labels

To extend or clip labels to particular frames, hover on the right or left edges of the label track (coloured bars) to see a black bar. Drag this black bar in the left/right directions to extend or clip the label to some particular frames from the sequence.

Delete label in specific frame:

You can delete the interpolated label in a specific frame by holding the command (for Mac) or Ctrl (for others) and left-clicking to a specific frame of the labelling timeline.

Note: By disabling “fill new labels”, new labels created will not be interpolated.

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to create a 3D bounding box:

1. Click on Label type and select 3D bounding box from the top left menu of the editor.

2. To Create a Bounding Box: Start labelling by clicking on Create Tool(C) using Left-click on the left-hand sidebar. Note that it only adds bounding boxes in this mode.
3. To Edit Bounding Box: Users can edit boxes only in “Select and Move Tool(V)”.
4. To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the empty space of the box to move the bounding box to the desired direction.
5. To Rotate Bounding Box: Hold “R+Left Click”. Now Click “Rotated arrows” to rotate the bounding box.
6. To Increase or Decrease Bounding Box: To Enable only Scale(Increase or Decrease Box Size)action, Hold “S+Left Click”. Now Click “Circle” to adjust the bounding box in different directions.
7. To Delete Bounding Box: Select the label and click delete(shift+delete for sequence) or click on the trash can icon of the label in the right panel.

Overview: One-click bounding box allows you to create a perfectly labeled bounding box with just one click.

Steps to create a one-click bounding box:

2. Once selected, place the bounding box at a dynamic object. Ex: Vehicle. Placing at concentrated points will yield better results.

3. Look at the secondary view to check the quality of the placement. Adjust the label accordingly if needed.

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

How to enable customer review for datasets? While creating a new dataset, turn on the Enable customer Review option. If you would like to use this feature for an existing dataset, go to the Settings screen in the dataset you would like to enable this, and turn on Enable customer review option.

How does this work? When customer review is enabled, users can submit the dataset for review once all the tasks are complete in all the pipeline stages. It is recommended for users to complete their existing tasks and resolve their pending issues across all stages before submitting them to review. Please note that once submitted, users will not be able to modify labels and tasks in the editor.

Users will not be able to access the review app when the work is in progress in any of the pipeline stages. When the review app is launched for the first time, you're greeted with a screen to set your review configuration. Users are provided with two types of review options: Sampling review, Full review.

In the sampling review, you can review a sample size of frames to review which are randomly chosen instead of all the frames like in the Full review. Once you have set the configuration, you are ready to get started!

When reviewing, if you notice any labels where re-work might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Issue Creation:

• Users can create issues if they find any errors in the frame/dataset.

• To create an issue, click on the Issues icon.

• Click on the area and Name the issue and select priority and click on Create.

• On the right side menu, all the created issues can be viewed and managed.

Overview: In the 3D coordinate system, lines can be described using vector equations or parametric equations. Lines in 3D have equations similar to lines in 2D and can be found given two points on the line.

Steps to draw lines:

4. Draw a line using the mouse left click.

5. Press the ESC key for drawing another line.

6. Lines can be edited only in Select and Edit Tool(V).

7. Use mouse left click to select a line.

8. User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.

9. Labels will be saved automatically.

10. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Before:

Accessing Show Ground Mesh:

After:

User can Hide/Unhide the lidar points from top and bottom in the editor.

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

2. To Create a Paint Label, Select a class and use left-click to paint on Lidar.
4. Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”.

5. To paint another category change category type and paint using left-click.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P”.

8. You need to save by clicking on Paint at the top panel and then click on save or use Enable Autosave.
9. To erase, hold “e” or click on the eraser icon to erase the paint.

What is Deepen AI Suite? Deepen AI is the fastest way to annotate data to build and ship artificial intelligence applications. Deepen AI Suite provides enterprise-grade AI-accelerated tools focusing on image, video. and sensor fusion (camera, LiDAR, radar, and more) training data. Who is it for?

1. Believe commercial solutions will be a far better choice than building and maintaining internal tools and infrastructure.

2. Want the highest quality standards for creating training data.

3. Want to standardize creating and managing training data across various teams within the organization.

4. Want to leverage labeling automation and reduce costs in human labeling.

5. Want the ability to work with internal labeling teams (usually experts) as well as fully managed external labeling service.

6. Want to leverage state of the art labeling tools for creating training data.

Overview: Boxes can be adjusted in all frames at a single place.

Steps for using Label View:

1. In Label View the selected bounding box can be viewed and adjusted in all the preloaded frames.

2. The box will be displayed in Top, Front and Side view.

3. Select a bounding box and Click on ’Label view’ to enable it.

Overview: Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: Click on Paint at the top panel and select 'Paint with Polygon' or use right-click at the type selection. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: Ground Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: To select the ground polygon, Use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5.Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: Smart Brush is best used to paint a large clustered point quickly.

Steps to paint with Smart Brush:

1. Select a category from the editor.

2. Selecting Smart Brush: Click on Paint at the top panel and select 'Paint with Smart Brush' or use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points.

4. To paint another category change category type and paint using left-click.

5. By increasing the distance, It would increase the search radius to expand the smart brush.

6. By increasing the number of points, it would increase the difficulty for the cluster to grow as more number of points are required to be considered as the core point of the cluster.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P”

8. You need to save by click on Paint at the top panel and then click on save.

9. To erase, hold “e” or click on the eraser icon to erase the paint

Overview: Polygons are straight-sided shapes (3 or more sides), defined by three-dimensional points (vertices) and the straight lines that connect them (edges).

Steps to draw a polygon:

1. Click on Label type and select 3D polygon from the top left menu of the editor.

2. Select a category from the dropdown.

3. Click on Create Tool(C) to create a new Polygon.

4. Draw a Polygon using the mouse left click.

5. Press the ESC key to end the Polygon.

6. Polygons can be edited only in Select and Edit Tool(V).

7. Use the mouse left click to select a Polygon.

8. User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.

9. Labels will be saved automatically.

10. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: Instance Painting allowed to paint each instance of each object featured in the point cloud instead of categorizing each pixel like in semantic segmentation.

1. Click on Label type and select 3D Instance painting from the top left menu of the editor.

2. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon or Cube) to start painting.

3. Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”

4. Instance Painting: To Create a Paint Label, Select a class and hold C on the keyboard using left-click on the Lidar.

5. To create another instance of the same or another category, hold “C” while painting. It will automatically generate a new instance.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You need to save by click on Paint at the top panel and then click on save.

8. To erase, hold “e” or click on the eraser icon to erase the paint.

9. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: Tasks can be assigned in both labeling and configured pipeline stages for added users. Steps for Task Assignments: You can assign the tasks to the user in the dataset level under “Tasks”. 1. Click on Manage tasks for assignments. 2. Here we can assign the files to both the labeling and QA and other configured pipeline stages.

3. Admin needs to select the user email from the drop-down boxes and select the files from the unassigned files or enter the range(E.g: 1-4,5,6-11). 4. Labelling users can check the assigned files from the editor by clicking on Your work at the centre of the editor.

5.There are some filters by which the user can sort based on their preferences.

6. Users can move to the assigned frames/labels in the editor by clicking on the next task.

Steps to paint with Smart Brush: 1. Select a category from the editor. 2. Selecting 3D Brush/Cube: Click on Paint at the top panel and select 'Paint with 3D Brush' or use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points. 4. To paint another category change the category type and paint using the cube. 5. Use the 'F' key to start painting. 6. The size of the cube can be adjusted.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P” 8. You need to save by click on Paint at the top panel and then click on save.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Overview: The brush is best used to paint dynamic objects like vehicles.

Steps to paint using a brush: 1. Select a category from the editor.

2. Selecting a Brush: Click on Paint at the top panel and select 'Paint with Brush'. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. To paint another category change category type and paint using left-click. 6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers. 7. Brush depth: This option lets you adjust how far the brush has to paint the points in the LiDAR. Setting a lower depth value, lets you paint points closest to the brush and a higher depth value lets you paint all the points that fall (including the closest up to the farthest) under the brush. The Default value is 5 which can be adjusted. 8. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 9. You need to save by click on Paint at the top panel and then click on save.

10. To erase, hold “e” or click on the eraser icon to erase the paint.

Following are some shortcut keys which can be used in 3D Editor

Overview: Secondary views enable the user to look at the label from three different views - Top, Front, and Side View.

To Disable secondary views, click on "view" in the top menu and Unselect Enable Secondary Views

Labels for 3D instance segmentation can be downloaded as combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The dpn file is a binary file which contains point level information for label cateogry and label id. For a dataset with n number of points, first n bytes of dpn file represent the label category, the next n bytes represent the lower bytes of the label id and the subsequent n bytes represent the higher bytes of the label id the point belongs to.

For example, consider a dataset containing 3 files each with 10,000 points, 11,000 points and 9000 points respectively. Assuming that the point with index 50 in second file of the dataset is annotated as below :

label_category : "paint_category_2" label_id : 1024

For the above scenario, dpn file will contain 90000 bytes ((10,000 + 11,000 + 9000) * 3). The annotation information for the point will be present at 10,050th (10,000 + 50), 40,050th (30,000 + 10,050) and 70,050th (60,000 + 10,050) indexes. The values of bytes at these indexes are as below:

10050 : 2 (1 based index of "paint_category_2" in metadata) 40050 : 0 (1024 % (2^8)) 70050 : 4 (1024 / (2^8)) The value 0 for label category is reserved for unpainted points.

Overview: Issue Creation allows you to create an issue for a label, notify the annotator about the issue, and then discuss the issue with our comment feature.

Steps to create an issue:

1. You can create an issue using by clicking on the "create issue" icon in the left panel on the labeling screen.

2. Once you've created the issue using the "create issue" feature. You can go into the QC view by clicking on "issues" in the right panel.

3. If the current stage is not the first stage, then the assignee email is the email to which this task is assigned in the previous stage.

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

How to enable customer review for datasets? While creating a new dataset, turn on the Enable customer Review option. If you would like to use this feature for an existing dataset, go to the Settings screen in the dataset you would like to enable this, and turn on Enable customer review option.

How does this work? When customer review is enabled, users can submit the dataset for review once all the tasks are complete in all the pipeline stages. It is recommended for users to complete their existing tasks and resolve their pending issues across all stages before submitting them to review. Please note that once submitted, users will not be able to modify labels and tasks in the editor.

Users will not be able to access the review app when the work is in progress in any of the pipeline stages. When the review app is launched for the first time, you're greeted with a screen to set your review configuration. Users are provided with two types of review options: Sampling review, Full review.

In the sampling review, you can review a sample size of frames to review which are randomly chosen instead of all the frames like in the Full review. Once you have set the configuration, you are ready to get started!

When reviewing, if you notice any labels where rework might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good, you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Issue Creation:

• Users can create issues if they find any errors in the frame/dataset.

• To create an issue, click on the Issues icon.

• Click on the area or the label and name the issue. select the priority and click on Create.

• On the right side menu, all the created issues can be viewed and managed.

Labels for 3D semantic segmentation can be downloaded as the combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in the metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The DPN file is a binary file which contains point level information for label category and label id. For a dataset with n number of points, dpn file contains n bytes that represent the label category.

For example, consider a dataset containing 3 files each with 10,000 points, 11,000 points and 9000 points respectively. Assuming that the point with index 50 in second file of the dataset is annotated as below :

label_category : "paint_category_2"

For the above scenario, dpn file will contain 30000 bytes (10,000 + 11,000 + 9000). The annotation information for the point will be present at 10,050th (10,000 + 50) index. The byte value at this index would be 2 which is the 1 based index of "paint_category_2" in the paint categories provided in the metadata. The value 0 for label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header:

< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": ["Drivable region", "Uneven terrain", "Hard Vegetation", "Soft vegetation", "Construction object", "Other Traversable Static", "Ghost", "Static Object", "car", "Vehicle _ Truck", "Pedestrian with Object _ Adult", "Pedestrian _ Adult", "Vehicle _ Trailer", "Construction Object _ Construction Cones and Poles", "Ghost _ Sensor Artifact", "Vehicle _ Van", "Vehicle _ Car", "Ego Car", "Ground", "dynamic_buffer"]}

You can get the paint_categories and compression format from the above data. Here the compression format is pako_compressed and you can use pako decompression to get the paint labels for each point.

For the paint categories, your understanding is correct. We reserve 0 for unlabelled points and use the index from the paint_categories field in the paint labels.

We Support 7 different dictionaries. 4x4, 5x5, 6x6 and 7x7 can have a maximum of 1000 markers, The original dictionary supports upto 1024 markers, apriltag_36h11 can have upto 587 markers, and apriltag_25h9 can have upto 35 markers.

• Original

• 4X4

• 5x5

• 6x6

• 7x7

• apriltag_36h11

• apriltag_25h9

How to choose the bits of charuco dictionary?

The more the number of markers/corners, the better the accuracy of your calibration, but your camera should be able to identify your board correctly . So consider the camera's resolution, the target's distance from the camera, and the environment around you while choosing the dictionary to get a clear image and more corners.

while having multiple target avoid using 4x4 as it has been observed to provide ghost points due to its relatively simple pattern.

Overview: Import Labels allows users to import existing dataset labels from a dataset to another dataset in JSON and DPN(3D Segmentation) format for both 2D and 3D.

• Users can find the import labels feature beside the launch button(If customer review is enabled, it is beside Launch review) on the dataset page.

• Once import labels is selected, the user can select the format and click on Import Labels. Labels will be uploaded in a few minutes based on the file size.

• Users can select Import into label set if they want to import labels into a different Label Set.

• If web paint is selected, a metadata file should also be uploaded in JSON format. The segmentation format can be found at Data Output Format - 3D Semantic Segmentation

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to draw a Bounding Box:

1. Click on Label type and select the Bounding box from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a bounding box by clicking on the image with the left mouse click and Drag the 2D-box up to the corners of the object.

4. You can select the label by clicking on the label with a left mouse click. Then you can edit the 2D-Box, by dragging the highlighted points of the box with a left mouse click.

5. Labels are saved automatically.

6. Click “control + shift” to draw a bounding box within a box.

7. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Overview: Importing a dataset configuration via JSON

1. Go to the Datasets tab, find the import profile feature at the top right under the 3 dot dropdown

2. Click on import profile, there you can click on import profile from JSON and select a JSON file.

3. Customize what you wish to import

4. Once you've confirmed, click on "import config". All the configurations in the dataset will then be changed according to the JSON file you've chosen"

Overview: Background feature allows you to paint an object that is occluded

Foreground:

1. Click on Paint and select “Paint Foreground”

2. Once “Paint Foreground” is selected, the user can paint on any area of Lidar.

3. In the image editor, there is an option for selecting the foreground as “Paint LiDAR as Foreground”.

4. You need to save by click on Paint at the top panel and then click on save.

Steps to paint background:

1. Click on Paint and select “Paint Background” at the top panel 2. Once “Paint background” is selected, paint that is already labeled won’t be affected.

3. Paint Background is mostly used when there is some unpainted area in Lidar

Calibrations Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration Selection

Select the Camera Intrinsic Calibration button to create a new dataset.

Calibration Instructions Page

Upon selecting Camera Intrinsic Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Configuration for Checkerboard

• Target configuration = Checkerboard

• Enable use EXIF metadata to use of EXIF metadata from images to optimize calibration. Disable this if using an External Lens for the Camera is recommended.

• Camera lens model: For wide-angle cameras, use Fish-eye and Standard for the rest

  • Standard is the Brown-Conrady camera model

  • Fish-eye is the Kannala-Brandt camera model

• Horizontal corners: Number of horizontal inner corners in the checkerboard

• Vertical corners: Number of vertical inner corners in the checkerboard

Configuration for Charucoboard

• Target configuration = Charuco Board

• Enable use EXIF metadata to use of EXIF metadata from images to optimize calibration. Disable this if using an External Lens for the Camera is recommended.

• Camera lens model: For wide-angle cameras, use Fish-eye and Standard for the rest

• Horizontal corners: Number of chessboard squares in the horizontal direction in charucoboard

• Vertical corners: Number of chessboard squares in the vertical direction in charucoboard

• Square size: The size of the square in the board in meters

• Marker size: The size of the ArUco marker present inside the charucoboard in meters.

Calibration Pipeline

Add images

Upload the images taken from the camera for which intrinsics need to be calculated.

Settings (optional)

Alpha is the new scaling parameter in Camera Intrinsic Calibration. Alpha can have values between 0 and 1.

1. Alpha=0, it returns an undistorted image with minimum unwanted pixels

2. Alpha=1, all pixels are retained with some extra black pixels.

Default: Includes k1, k2, k3, p1, p2 in distortion coefficients

Extended intrinsics: Includes k1, k2, p1, p2, k3, k4, k5, k6 in distortion coefficients

Minimal(K1,K2): excludes k3 from distortion coefficients and includes only k1, k2, p1, p2

Run calibration

Click the Calibrate button at the bottom to trigger calibration. You can see the intrinsic parameters and error statistics in the right panel upon completion.

Verify Results

Error Stats

The Reprojection error is in pixels. It is the mean of the Euclidean distance between the auto-detected checkerboard corners and reprojected checkerboard corners. The closer the Reprojection error is to zero, the better the intrinsics are.

Undistorted Image

• Users can visualize the Undistorted image to check the quality of the intrinsics.

• The side-by-side view can be used to check both the distorted and undistorted images simultaneously

Checkerboard Coverage

• Checkerboard coverage shows the area covered by the checkerboard corners from all uploaded images. The higher the coverage, the better the intrinsic parameters.

  • 0 - 50% is low coverage

  • 51 - 70% is moderate coverage

  • 71 - 100% is Good coverage

• Users can see the individual reprojection error of all the checkerboard corner points. A color ramp is used to depict the reprojection error. A light red color shows a lower reprojection error, and a darker red indicates a higher reprojection error.

Save to profile

Camera Intrinsic parameters can be saved to the profile for easier import in other calibrations.

Annotation data is often collected by ROS and stored as ROS bags. In order to use deepen.ai’s system for annotation, the data must be first converted into the JSON format: https://help.deepen.ai/documentation/3d-input-output/3d-upload-json-format

This JSON file specifies the exact data for annotation. It includes point clouds, images, timestamps, intrinsic and extrinsic calibrations, as well as localization information.

Usually, each user writes a script to convert their bag data into the JSON format. We cannot use a common script for conversion for several reasons:

1. Each ROS bag may have many topics. The user needs to specify the exact topics for annotation.

2. Some information such as intrinsic and extrinsic calibrations may not be in the ROS bags.

3. If a special camera model is used, the user may wish to only send the rectified images for annotation.

4. The JSON format links the point cloud and images shown to the annotators. It does not sync using timestamps like RVIZ.

5. If accurate localization is available, the point cloud is transformed into a fixed world coordinate.

6. Advanced processing such as LiDAR ego-motion compensation is typically performed in the script.

Users not familiar with the JSON format may find the conversion script difficult to develop. At deepen, we have worked with many clients on their data conversion scripts. Although each script is different, we have found many common themes in these scripts. If you are developing a new script, this tutorial will help you to get started and get your data ready for annotation. ROS has good Python support. Most conversion scripts are written in Python, so this tutorial also assumes that. We will walk you through the various steps of the conversion script. We will try to describe the simplest processing. There are many advanced techniques which can improve the data, but we will not cover them here.

Reading the Bag file(s)

The first step is very obvious. You need to specify the path of the files and ROS topics to annotate. The topics should include point clouds and images. Images are very important for annotation speed and accuracy.

Synchronizing the point clouds and images

In the 3D annotation tool, the annotator is shown a point cloud and one image from each camera. Thus, the JSON format specifies the connection between the point cloud and images. To calculate this, our script needs to explicitly synchronize the data. Often, the synchronization is done by ROS timestamps. Let us assume that there is only one LiDAR. Thus, there is only a single point cloud sequence. In this stage, we make a single pass through the point cloud data and collect its timestamps. This is our primary timestamp sequence.

We then make a pass through each image topic that we can show to the annotators. As we go through the images, we find the closest timestamp to each LiDAR timestamp. This will be the image attached to each point cloud. Thus, there is one image from each camera for each point cloud. The image sequence is synchronized to the LiDAR sequence.

Temporal Calibration (Advanced Topic)

Technically, each timestamp occurs when the LiDAR finishes a spin or a camera closes its shutter and ROS records its data. The sensors, network, and ROS itself all add latency to this process. Thus, the timestamps usually occur a few milliseconds after the actual events. This latency is typically different for each sensor and introduces inconsistency in timestamps. Temporal calibration is the method to adjust the timestamps, so all sensors have consistent timestamps. We will not cover it here, and you may skip it for your initial conversion script.

Multi-LiDAR

In the case when we have multiple LiDARs that we need to annotate, we can pick one LiDAR as the primary one and synchronize all other LiDARs to it. Note that there is the “d” field for each point in deepen’s JSON format, to which we can assign the LiDAR ID. If you use the “d” field, our annotation UI allows you to distinguish between point clouds from different LiDARs.

Localization

The JSON format specifies all points in a single world coordinate because it would make annotation much easier and more accurate: All static objects would be in a fixed location. All dynamic objects would have their simple motions, without complications from the motion of the ego vehicle itself. In order to achieve this, we would need accurate localization data. The best localization usually comes from an expensive INS/GNSS. Otherwise, LiDAR SLAM combined with other sensors such as IMU and odometers can also give accurate localization in most situations.

If an accurate localization is available as a ROS topic, we just need to find the LiDAR pose with a timestamp closest to that of the LiDAR point cloud. We can use the pose to transform the point cloud into the world coordinate. Note that accurate LiDAR poses require LiDAR to INS calibration, which we will not cover here.

If localization is unavailable, we suggest that you try one of the LiDAR SLAM algorithms. If you skip this step and use the LiDAR point cloud as-is, it is still possible to annotate the data, but the cost and accuracy would both suffer.

Ego Motion Compensation (Advanced topic)

Most LiDARs have a slow “rolling shutter” where each scan or spin can take tens or hundreds of milliseconds. During this time interval, the LiDAR itself may be going through a complicated motion. If we treat the LiDAR as stationary for the entire time interval, the point cloud would be inaccurate. Ego motion compensation is the technique to solve this problem, but we will not cover it here. You can ignore this issue unless your vehicle was moving at a high speed such as on a highway.

Intrinsic and Extrinsic Calibrations

Accurate intrinsic calibration is required for each camera. We support the common plumb-bob (Brown & Conrady) and equidistant fisheye (Kannaly & Brandt) camera models. If you choose a model we don’t currently support, you can submit rectified images instead.

Extrinsic calibration is for specifying the camera pose for each image. If localization and ROS tf transforms are available, you just need to obtain the camera pose at the timestamp of the image. Note that we are using the camera coordinate system in OpenCV. It is identical to the ROS optical coordinate system, not the standard ROS coordinate system.

If the tf transforms cannot give you correct camera poses, but you have the extrinsic LiDAR-camera calibration, you can apply the extrinsic calibration to obtain the camera pose from the LiDAR pose. It is just a simple matrix multiplication. Since the point cloud and image usually have different timestamps, it would be more accurate to interpolate the LiDAR or Camera poses, but we will skip it for this tutorial.

Output

After obtaining the information above, we just need to make a pass through the point clouds and images to output the JSON and image files. Note that the JSON format supports BASE64 encoding, which can make your JSON files much smaller and the script run much faster.

Debugging

Debugging your first conversion script often takes a long time. We have some tips to find the most common bugs.

The first tip to debug is to output a small number of frames. Your script will finish quickly. You can upload this subset into deepen’s tool quickly and visualize the data.

Localization

To verify your localization accuracy, the easiest way is to load a moving point cloud sequence into deepen’s annotation tool. You can use the “fuse” option to accumulate multiple point clouds together. If the static objects are sharp in the fused point cloud, you are likely to have the correct localization. If not, there are many common mistakes:

1. Low quality or misconfigured INS/GNSS

2. Wrong usage of INS output

3. Wrong coordinate frame is used

4. Bad output from LiDAR SLAM algorithm

Calibration

To validate your camera calibrations, you can load a sequence into the annotation tool. Add a 3D box to an object. If the projection of the box on the image looks correct, you probably have the correct calibrations. Otherwise, there are many common mistakes:

1. Giving rectified images and also non-zero distortion coefficients

2. Using the ROS coordinate system instead of the optical coordinate system

3. Scaled the images but did not change the intrinsic parameters

4. Bad LiDAR-camera calibration. You can use deepen’s calibration tool to redo or correct the calibration. https://help.deepen.ai/documentation/calibration/camera-lidar

5. The pose of the camera may be off due to localization error.

Synchronization

In order to validate the synchronization of your data, you can load a sequence into the annotation tool. Add a box to a moving object. If the projections on all camera images look correct, you should have a proper synchronization between point cloud and camera. For a high-precision check, the object should have a high angular velocity relative to the sensors. Therefore, you can use an object moving quickly from left to right or use a sequence where the ego vehicle is turning rapidly. If you found a synchronization error, there are several common mistakes:

1. For big errors, there is likely to be some bugs in the synchronization logic in the script.

2. For small errors, temporal calibration may be needed.

3. For very small errors, we may need to consider the exact time the LiDAR scans the object which is different from the timestamp we assigned to the entire point cloud. This is an advanced topic.

Visualization code

We have developed an offline visualization script to visualize the JSON files. Please find the tool and documentation for it here.

Sample Code

We will release a sample conversion script soon.

Please contact us if you run into problems with your conversion script. We will gladly help you debug your script.

Calibration Target

1. Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically. https://drive.google.com/file/d/1mTR8HTpvROE1Pv0rmXEBVLSxs_yMDnvf/view?usp=sharing

2. Charucoboard of more than two horizontal and vertical squares. Click here for supported Charuco dictionaries.

Target-based calibration

Data collection

Place the target roughly 3m to 5m from the camera. For the closest target position, the closer should be far enough so that all the board's edges are visible from the camera and lidar. So, it is highly recommended to do this capture inside a building rather than outside. No target position should be occluded in the camera or lidar view.

The same target should be used in all camera and lidar frames.

For example, please take images with a single board at various positions like the following.

The boards and all sensors should be static while collecting the data. To avoid time-synchronization problems, please keep the boards and the sensors stationary for at least 10 seconds while collecting each data pair.

For example, these are the steps to collect one set of calibration data:

1. Orient the camera toward the target. Start recording. Wait for 10 seconds (Don't move/rotate your car/robot/sensors). Stop recording. You must have a recording of images and lidar data for 10 seconds. Extract one image from the camera, and one frame of lidar data captured 5 seconds after the recording has started (e.g., if you start recording at 3:00:00, you stop recording at 3:00:10. We need a frame captured at 3:00:05) and save them.

2. Change the target's location and orientation. Start recording. Wait 10 seconds (Don't move/rotate your car/robot). Stop recording. Again, you must have a recording of images and lidar data for 10 seconds. Extract one image from the camera, and one frame of lidar data captured 5 seconds after the recording starts and save them.

3. Repeat the process for at least 5 data pairs.

Optional Board Position

on-ground: The target is placed on the ground (touching the ground). In such a case, enable the on ground flag in the target configuration. Also, make sure that the lidar data captures the ground points. By doing so, we can optimize the extrinsic parameters using the ground points.

Tilted: A holder holds the target up in the air and tilts right by around 45 degrees. In such a case, enable the Tilted flag in the target configuration. This approach enables deep optimization, and the extrinsic parameters are further optimized using the edge points of the board.

Targetless calibration

Data collection

For the targetless calibration, users must record a scene in the camera and the LiDAR sensor. No target is required for this calibration, but the scene should have vehicles (cars and vans) in the camera and the LiDAR data. For better calibration, vehicles should be close to the LiDAR (3m - 10m) with a good number of points in LiDAR and present on both the left and right sides of the image; having too many vehicles may result in calibration errors. Ensure that the vehicles (including the ego) stop or move slowly. This reduces the effect of the time difference between LiDAR and the camera. Select 3-4 frames from the collected data that have vehicles on both sides of the images and vehicles close to the lidar.

Calibration Target

1. Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically.

2. Charucoboard of more than two horizontal and vertical squares. for supported Charuco dictionaries.

Data for camera intrinsic and distortion calibration

Checkerboard

Place the checkerboard in the field of view of the camera and make sure the checkerboard is in the camera's focus. Move the checkerboard or the camera to different positions and take images so that the coverage of the checkerboard in the camera's field of view is as high as possible. While moving the checkerboard or camera, it is necessary that the entire checkerboard is present in the camera's field of view. In addition, the checkerboard should be tilted slightly in different directions for each image. Do not move the board or the camera while taking the images. Also, do not hold the checkerboard by hand to minimize blur due to shaking. You can also use a computer monitor instead of a physical checkerboard for intrinsic calibration. Monitors are very accurate and flat. You can show this on the monitor for calibration.

Charucoboard

Place the charucoboard in the field of view of the camera and make sure the charucoboard is in the camera's focus. Move the charucoboard or the camera to different positions and take images so that the coverage of the charucoboard in the camera's field of view is as high as possible. While moving the charucoboard or camera, it is not necessary that the entire charucoboard is present in the camera's field of view. In addition, the charucoboard should be tilted slightly in different directions for each image. Do not move the board or the camera while taking the images. Also, do not hold the charucoboard by hand to minimize blur due to shaking. You can also use a computer monitor instead of a physical charucoboard for intrinsic calibration. Monitors are very accurate and flat. Please visit our website to download charucoboard:

Checkerboard sample data collection

Fisheye and wide angle cameras

For cameras with high distortions, such as fisheye and wide-angle cameras, we need to pay special attention to the field of distortion. If each board only covers a small portion of the image, it is likely to lead to overfitting of the field of distortion. i.e. Locally correct distortion but globally incorrect distortion. We recommend to have at least one image where the board covers most of the field of view. It can give useful information on the distortion across the entire field of view.

Long focal length cameras

For cameras with long focal length and narrow (<= 30 degree) field-of-view, it is difficult to obtain the focal length and principal point accurately. If all boards face the camera almost directly, there may be large errors in the vanishing points and focal length. We need some images where the board is very tilted to obtain accurate vanish points, focal length, and principal point.

Charucoboard sample data collection

Following are some shortcut keys that can be used in 2D Editor. These can be found from the editor under the Help⇾Keyboard shortcuts menu or on the bottom right of the editor.

1. What are the Recommended Browsers? Chrome (preferably the latest or at least version 97+)

2. What is the required Screen Resolution? Recommended 1920x1080 pixels or more (at least 1366x768 pixels)

3. What is the list of domains which need to be whitelisted? Domain Whitelist: The following domains and any requests from its child URLs have to be whitelisted: https://tools.calibrate.deepen.ai https://storage.googleapis.com/deepenai-prod-rannotateclientdata/ https://storage.googleapis.com/deepen_calibrate_assets https://www.googleapis.com https://fonts.googleapis.com https://use.fontawesome.com https://fonts.gstatic.com https://help.deepen.ai/documentation/

`The zip file should contain a set of JSON files and camera images. Each JSON file will correspond to one point cloud frame of the dataset. JSON files should be at the root directory level. The order of the frames is decided by the order of JSON filename sorted in ascending order. For instance, filenames can be 0001.json, 0002.json, 0003.json, ... Filenames can also be 0.json, 1.json, 2.json, …

Each JSON file should be an object with the following 5 fields:

1. Images

2. Timestamp

3. Points

4. Device position

5. Device heading

1. Images

An image (array) - of all camera images corresponding to one point cloud frame. Usually, the number of images corresponds to the number of cameras in the system. If there are no images, please use an empty array. Each element is a JSON object. Fields in the image object are as follows:

1. fx (float) - focal length in x direction.

2. fy (float) - focal length in the y direction.

3. cx (float) - x coordinate of principal point.

4. cy (float) - y coordinate of principal point.

5. timestamp (float) - time in seconds when the image was captured.

6. image_url (string) - corresponds to an image path inside the zip file, e.g. “images/0001.png”. It can also be an external URL. Image types supported are .jpeg and .png.

7. position (object) - position of the camera with respect to the world frame. Details of JSON objects can be found below.

8. heading (object) - orientation of the camera with respect to the world frame. Please find details of the JSON object below.

9. camera_model (string) - the camera model to be used for undistorting the image. Supported values for camera_model : pinhole (default) - uses k1, k2, p1, p2, k3, k4 distortion coefficients fisheye - uses k1, k2, k3, k4 distortion coefficients mod_kannala - uses k1, k2, k3, k4 distortion coefficients

10. k1 (float) - distortion coefficient.

11. k2 (float) - distortion coefficient.

12. p1 (float) - distortion coefficient.

13. p2 (float) - distortion coefficient.

14. k3 (float) - distortion coefficient.

15. k4 (float) - distortion coefficient.

16. camera_name - this is optional, but if given in JSON file, tool will use the same name to refer to this camera instead of using camera_0, camera_1, etc.

If images are already undistorted, k1, k2, p1, p2, etc. should be all 0's. You can find more details on the camera parameters here.

A sample image JSON is as follows:

2. Timestamp

Timestamp (float) – time in seconds at which the point cloud frame was captured.

3. Points

Points can be given in 2 formats-the first format is an array of JSON objects, and the second format is base64 encoded strings of points and intensities. Points in JSON object array format: A Points array of JSON objects of all LiDAR points having their x, y, z, i, r, g, b,d values. x, y and z values are mandatory and i, r, g, b and d values are optional for each point. In general, the “up” direction towards the sky should be in the positive z direction for the visualization to work correctly. Each element of the array is a JSON object, as shown in this section. rgb value in xyzrgb type point will be supported in a future release. Each point can have other values like velocity, as well, for which we can add custom support. Fields in point object are as follows:

1. x (float) – x coordinate of the point, in meters.

2. y (float) – y coordinate of the point, in meters.

3. z (float) – z coordinate of the point, in meters.

4. i (float) - intensity value between 0 and 1, this is an optional field

5. d (integer) - non-negative device id to represent points from multiple sensors, this is an optional field

x, y and z values are in world coordinates. If you are unable to put the point cloud in the world coordinate, you can fall back to the local LiDAR coordinate and let us know. We will contact you about the issue.

For Multi Lidar points, add the field 'd' in the points array to represent lidar id, it should be a non-negative integer value. A sample point JSON object is as follows:

"points": [
        {
            "i": 4.00,
            "x": -0.10,
            "y": 6.22,
            "z": 1.66,
            "d": 1
        },
        {
            "i": 11.00,
            "x": -0.14,
            "y": 9.20,
            "z": 1.80,
            "d": 2
        },
        {
            "i": 14.00,
            "x": -0.17,
            "y": 10.69,
            "z": 1.52,
            "d": 3
        }
]

If you want to add a name for each lidar id, then you need to add another field “multi_lidar_keys”, please note this is an optional field.

"multi_lidar_keys" : {
        "1" : "Lidar_1",
        "2" : "Lidar_2",
        "3" : "Lidar_3"
    }

4. Device position

A device_position (object) – position of LiDAR or camera with respect to world frame. Similar to the point cloud, if you are unable to put the device position in the world coordinate, you can fall back to the local LiDAR coordinate and let us know. We will contact you about the issue. For camera, if you do not have any position information, please use (0, 0, 0) and let us know. Fields in position object are as follows:

1. x (float) – x coordinate of device/camera position, in meters.

2. y (float) – y coordinate of device/camera position, in meters.

3. z (float) – z coordinate of device/camera position, in meters.

Sample position JSON object:

5. Device heading

A device_heading (object) – orientation parameters of LiDAR or camera with respect to world frame. If you are unable to put LiDAR heading in world coordinate, please use the identity quaternion (x = 0, y = 0, z = 0, w = 1). If you can not obtain extrinsic camera calibration parameters, please also use the identity quaternion. We will contact you about this issue. Fields in the heading object are as follows, the 4 components are quaternions:

1. x (float) – x component of device/camera orientation.

2. y (float) – y component of device/camera orientation.

3. z (float) – z component of device/camera orientation.

4. w (float) – w component of device/camera orientation.

A sample heading JSON object is as follows:

Please note that in JSON, the order of the dictionary values of quaternion doesn't matter. Following two JSONs will give exact same result:

A sample JSON can be found below,

{
  "images": [
    {
      "fx": 561.997914,
      "timestamp": 1541186225.8394644,
      "p2": 0.00251,
      "k1": -0.142792,
      "p1": 0.001203,
      "k3": 0,
      "k2": 0.022846,
      "cy": 361.97667,
      "cx": 664.16411,
      "image_url": "0.png",
      "fy": 561.585651,
      "position": {
        "y": -152.77584902657554,
        "x": 311.21505956090624,
        "z": -10.854137529636024
      },
      "heading": {
        "y": -0.7046155108831117,
        "x": 0.034278837280808494,
        "z": 0.7070617895701465,
        "w": -0.04904659893885366
      }
      "camera_model": "pinhole",
      "camera_name": "front"
    },
    {
      "fx": 537.74122,
      "timestamp": 1541186225.8499014,
      "p2": -0.000507,
      "k1": -0.133161,
      "p1": -0.0007,
      "k3": 0,
      "k2": 0.020764,
      "cy": 353.596887,
      "cx": 687.798477,
      "image_url": "1.png",
      "fy": 541.411032,
      "position": {
        "y": -152.7458074214421,
        "x": 311.168923367011,
        "z": -10.855340458227541
      },
      "heading": {
        "y": -0.571381519522144,
        "x": -0.4283386878183726,
        "z": 0.5635977900941452,
        "w": 0.4152188081814165
      },
      "camera_name": "rear"
    }
  ],
  "timestamp": 1541186225.848686,
  "device_heading": {
    "y": -0.014390929501214435,
    "x": -0.006511549504752948,
    "z": -0.8798010889637369,
    "w": 0.4750795141124911
  },
  "points": [
    {
      "y": -147.6858459726749,
      "x": 319.51523557465174,
      "z": -11.55716049374703,
      "i": 0.32
    },
    {
      "y": -147.709804574419,
      "x": 319.5387083489352,
      "z": -11.559980704176585,
      "i": 0.32
    },
    {
      "y": -147.8861052361809,
      "x": 319.3026396838094,
      "z": -11.536266484496409,
      "i": 0.32
    }
  ],
  "device_position": {
    "y": -152.4309172401979,
    "x": 311.42759643080274,
    "z": -11.704321251954227
  }
}

Another sample (Multi Lidar):

{
  "images": [
    {
      "fx": 561.997914,
      "timestamp": 1541186225.8394644,
      "p2": 0.00251,
      "k1": -0.142792,
      "p1": 0.001203,
      "k3": 0,
      "k2": 0.022846,
      "cy": 361.97667,
      "cx": 664.16411,
      "image_url": "camera_1.jpg",
      "fy": 561.585651,
      "position": {
        "y": -152.77584902657554,
        "x": 311.21505956090624,
        "z": -10.854137529636024
      },
      "heading": {
        "y": -0.7046155108831117,
        "x": 0.034278837280808494,
        "z": 0.7070617895701465,
        "w": -0.04904659893885366
      },
      "camera_name": "front"
    },
{
      Another 5 cameras will have similar settings then followed by 
        "heading": {
        "y": -0.3485952869673183,
        "x": -0.7973743859088446,
        "z": 0.1897018750524244,
        "w": 0.45463019389638054
      }
    },
    {
      "fx": 354.848025,
      "timestamp": 1541186225.832587,
      "cy": 331.448458,
      "cx": 812.600868,
      "image_url": "another_camera_1.jpg",
      "fy": 356.521296,
      "position": {
        "y": -152.11720409699208,
        "x": 311.6143516201334,
        "z": -10.845064295267472
      },
      "heading": {
        "y": 0.2603210255938851,
        "x": -0.8843903286926823,
        "z": -0.11886794907638565,
        "w": 0.36872363747252623
      }
    }
  ],
  "timestamp": 1541186225.848686,
  "device_heading": {
    "y": -0.014390929501214435,
    "x": -0.006511549504752948,
    "z": -0.8798010889637369,
    "w": 0.4750795141124911
  },
  "points": [
        {
            "i": 4.00,
            "x": -0.10,
            "y": 6.22,
            "z": 1.66,
            "d": 1
        },
        {
            "i": 11.00,
            "x": -0.14,
            "y": 9.20,
            "z": 1.80,
            "d": 2
        },
        {
            "i": 14.00,
            "x": -0.17,
            "y": 10.69,
            "z": 1.52,
            "d": 3
        }
  ],
  "multi_lidar_keys" : {
        "1" : "Lidar_1",
        "2" : "Lidar_2",
        "3" : "Lidar_3"
  },
  "device_position": {
    "y": -152.4309172401979,
    "x": 311.42759643080274,
    "z": -11.704321251954227
  }
}

Calibration Target

Checkerboard of more than two horizontal inner corners and vertical inner corners. You can use the attached pdf. It has seven internal corners horizontally and nine internal corners vertically. https://drive.google.com/file/d/1mTR8HTpvROE1Pv0rmXEBVLSxs_yMDnvf/view?usp=sharing

Prerequisites

Reference Points for Data Collection

Paste the adhesive tape on the ground along the vehicle's edges based on the camera-facing direction. Assume calibrating the front-facing camera sensor as an example for this collection process. Paste the adhesive tape shown in the image below for the front-facing camera. (See How to paste adhesive tape along the vehicle for different camera-facing directions?)

To maintain a fixed distance between the target and the vehicle, paste the adhesive tape parallel to the vehicle with a 1-10m distance between the target and the vehicle, as shown in the below image. It acts as a Reference Line (RL) for the target.

Definitions

Vehicle Reference Point (VRP)

The adhesive tape's left intersection point (from the camera perspective) along the vehicle's edges, as shown in the below image.

Intersection Reference Point (IRP)

This is the left intersection point of the adhesive tape from the vehicle with the adhesive tape from the reference line, as shown in the below image.

Target Reference Point (TRP)

This is the bottom-left edge corner of the target, as shown in the below image.

Vehicle dimensions

Left wheelbase

Distance between the center of the front left wheel to the center of the rear left wheel.

Right wheelbase

Distance between the center of the front right wheel to the center of the rear right wheel.

Note: For a rectangular vehicle, Left wheelbase = Right wheelbase

Front track

Distance between the outer edge of the front left wheel to the outer edge of the front right wheel.

Rear track

Distance between the outer edge of the rear left wheel to the outer edge of the rear right wheel.

Note: For a rectangular vehicle, Front track = Rear track

Front overhang

Distance between the center of the front left/right wheel to the vehicle's front.

Rear overhang

Distance between the center of the rear left/right wheel to the vehicle's rear.

Data collection

1. Mount the camera onto the vehicle and note the distance between VRP and IRP.

2. Place the board at the Reference Line and ensure the board is parallel or perpendicular to the car.

3. Take an image from the mounted camera and note the distance between IRP and TRP.

4. Move the board along the Reference Line, and at every placement, take an image from the mounted camera and note the distance between IRP and TRP.

Note: We recommend capturing at least three images with the target perpendicular to the ground and at least three images parallel to the ground in different positions on the Reference Line to maximize accuracy.

How to paste adhesive tape along the vehicle for different camera-facing directions?

Calibration Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration selection

Select LiDAR-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting LiDAR-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

Users can choose either the target-based or the targetless calibration. The target-based calibration uses the checkerboard/charucoboard as the calibration target, and the targetless calibration uses the scene captured in both LiDAR and the camera sensor data.

Configuration

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Target Configuration

Charucoboard target configuration

• Rows: Total number of squares in the horizontal direction.

• Columns: Total number of squares in the vertical direction.

• Square size: It is the length of the arm of the square in meters.

• Marker size: It is the length of the arm of the aruco marker in meters. This is usually 0.8 times the Square size.

• Left padding: The distance from the board's left edge to the left of the first square in the row.

• Right padding: The distance from the board's right edge to the right of the last square in the row.

• Top padding: The distance from the board's bottom edge to the bottom of the last square in the column.

• Bottom padding: The distance from the board's top edge to the top of the first square in the column.

• On ground: Enable this if the checkerboard is placed on the ground and the point cloud has the ground points in the scene around the checkerboard placement.

• Tilted: Enable this if the charcoboard is tilted.

Upload files from LiDAR and Camera

Add point cloud files from the LiDAR and images from the camera sensor. After adding, pair the point cloud files with the matching image files before continuing.

sample CSV format

X, Y, Z
0,-0,-0
62.545,-3.5064,-3.5911
62.07,-3.5133,-4.1565
32.773,-1.8602,-3.4055

Detect target corners in images

Our algorithms automatically detect corners in the charucoboards

Estimated extrinsic parameters

LiDAR frame and Target selection for mapping

1. Click on Change next to the lidar file to select the file for mapping

2. Click on Change next to the Target to select the target for mapping

Map target in point cloud

The extrinsic parameter space is vast, so we need an estimated entry point for optimization. The user can provide estimated extrinsic parameters in three ways.

Mapping of target corner points

Users can map the target corner points in the point cloud and get the initial estimates of the extrinsic parameters. Only one point cloud mapping is sufficient to get the initial estimates.

Auto detect target

Our algorithms can automatically detect targets in the point cloud if the lidar channel data is provided on the configuration page. Please note that the auto-detection might not work properly if there are many flat surfaces, like walls, ceilings, etc., in the scene.

Add estimated extrinsic parameters

Users can manually enter estimated extrinsic parameters.

Verifying the accuracy of the estimated extrinsic parameters

Once the estimated extrinsic parameters are in the tool, users can visualize the parameters by clicking on the visualize button. In the visualization, we have a few sensor fusion techniques through which the accuracy of the extrinsic parameters can be visualized. For more details, visit Sensor fusion techniques.

Estimated extrinsic parameters are crucial in generating accurate extrinsic parameters.

To get good initial estimates, users must clear the markers and redo the markings if the estimated parameters are way off.

Run calibration

Users need to click on Calibrate to optimize the estimated extrinsic parameters further. All the uploaded pairs are used in the optimization process.

Additional options in the run calibration

Deep Optimization: Users can select deep optimization to optimize the extrinsic further for datasets with the Tilted option enabled on the configuration page.

Max correspondence: This value is used as input for the algorithm. Users can tweak the value by analyzing the fused point cloud LiDAR files. Suppose the difference between the input and the generated cloud is more significant; the user can try to increase the value of the max correspondence and retry to improve the calibration results.

Error stats

Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. The closer the error stats to zero, the better the extrinsic parameters.

• Translation Error: Mean of difference between the centroid of points of checkerboard in the LiDAR and the projected corners in 3D from an image. Values are shown in meters. This calculation happens in the LiDAR coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

• Plane Translation Error: Mean of the Euclidean distance between the centroid of projected corners in 3D from an image and plane of the target in the LiDAR. Values are shown in meters. Note: If the board is only partially covered by the LiDAR or the LiDAR scan lines are non-uniformly distributed, translation and reprojection errors are inaccurate, but this plane translation error is accurate even in these scenarios.

• Rotation Error: Mean difference between the normals of the target in the point cloud and the projected corners in 3D from an image. Values are shown in degree. This calculation happens in the LiDAR coordinate system. Note: All LiDARs have noise when measuring distance. This will, in turn, cause noise in the target's point clouds and the target's normals. Usually, this metric cannot measure accurately below 1 degree. For an accurate rotation error, we suggest using a faraway straight edge such as a building edge, roofline, or straight pole and projecting the point cloud onto the image. The rotation error can be calculated from the number of pixels between the image edges and the projected points.

• Reprojection Error: Mean difference between the centroid of the target corners from the image and the centroid of the projected target from the LiDAR space onto the image. This is calculated in the image coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

• Individual error stats for each image/LiDAR pair can be seen. The average shows the mean of the errors of all the eligible image/LiDAR pairs.

Download calibration parameters

Once the entire calibration is done, users can download all intrinsic and extrinsic parameters by clicking the Export button in the header.

Analyzing the extrinsic parameters in Visualization Mode:

Sensor fusion techniques

Users can use the following techniques to visualize the extrinsic parameters.

Frustum: Users can see the image's field of view in the LiDAR frame. This uses both the camera matrix and the extrinsic parameters. Image axes are also displayed according to the extrinsic parameters.

LiDAR points in image: Users can see the LiDAR points projected in the camera image using extrinsic parameters.

color points from camera: Users can see the camera's color points in the lidar space using the extrinsic parameters.

• Show target in LiDAR: Users can see the checkerboard points projected in the LiDAR frame using the extrinsic params.

Image: Target Identification

The target in the image is filled with points. If the target configuration the user provides is correct, there will be no overflow or underflow.

LiDAR: Extracted target

This shows the extracted target from the original lidar file. We use this to calculate the error statistics. We compare the extracted target with the projected target.

Fused Point Cloud:

Targets from all the point clouds are cropped and fused into a single point cloud.

• Input cloud: This contains the fuse of all input clouds filtering the target area. If the target is not in the LiDAR file, the user has to fix the extrinsic parameters by going back to the mapping step or manually updating them.

• Generated target: This contains the fuse of all generated targets. If the target is inaccurate, the user has to fix the target configuration or the inner corner detection.

• Input and generated target: This contains the fused output of the Input cloud and Generated target. This helps us to analyze the difference between the input and the generated output before optimization.

• Target begin vs after optimization: This helps to know the difference between the generated target, using the extrinsic values before and after the optimization step.

Extrinsic Calibration Output:

• roll, pitch, and yaw are in degrees, and px, py, and pz are in meters.

• lidarPoint3D is the 3d coordinates of a point in the lidar coordinate system.

• imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Camera coordinates system

We currently show three different types of camera coordinate systems. The extrinsic parameters change according to the selected Camera coordinate system. The export option exports the extrinsic parameters based on the selected camera coordinate system.

• Optical coordinate system: It's the default coordinate system that we follow.

• ROS REP 103: This is the coordinate system followed by ROS. When you change to this, you can see the change in the visualization and the extrinsic parameters.

• NED: This follows the North-East-Down coordinate system.

Sample Script

This is a sample Python script to project lidar points on an image using extrinsic parameters. It uses the open3d and opencv libraries.

Tool usage guide for old UX

Overview: Deepen Calibrate is a software tool that makes the critical task of sensor data calibration simple and quick.

Calibration List:

• This page contains the list of calibrations. Users can launch an existing dataset, delete and even manage user’s access to these dataset.

Calibration Launch:

• Users can click on ‘Get Started’ to go to the launch page.

• Users can calibrate multiple cameras to LiDAR within the same dataset. But calibration needs to be performed individually for each camera/LiDAR combination.

Start calibration:

Camera Intrinsic Parameters:

1. Intrinsic parameters for the camera are to be added here. Users have three options.

2. Users can use the intrinsic calibration tool and calibrate the results. Save them to profile and then load them here.

3. Or users can also load the JSON file.

Choice for manual or auto extrinsic parameters:

• If the user has known extrinsic parameters and they can directly enter the values or else he can choose to calculate them using the tool.

Manual extrinsic parameters:

• Users can update the extrinsic values manually.

• They can choose to verify the values by going to ‘Visualization Mode’.

• They can also further fine-tune these values.

Add Images and LiDAR Pair:

• Users need to upload images and LiDAR pairs in case of extrinsic Calibration.

• Each pair must have a checkerboard in their view. Please make sure that the checkerboard is in a different position in each pair.

• Users can click on the image/LiDAR on the left side panel to get the image viewer or the LiDAR viewer.

• Users can also delete/add image/LiDAR from the left side panel as well.

Checkerboard Configuration:

• Users need to fill up the config of the checkerboard (Please refer to the Checkerboard Configuration Description section for more details).

Map Pair:

• Users can click on ‘Start Mapping’ to go to the mapping mode. Here the user will have an image viewer on the left side and a LiDAR viewer on the right side.

• Users have an option to add points in the image. And he has to map each point from the image to the corresponding area in the LiDAR viewer.

• Users have the option to paint an area in the LiDAR for each selected point in the image.

• The centroid of the painted area is taken into consideration.

• The calibration results will depend on this step, if the selected area is small then the result would be better.

• So the user has all the options to zoom in, zoom out, pan and rotate.

• Users even have the option to erase a particular painted area and improve the correspondence relation.

• In most cases, four points are preselected in the image. (All preselected four points are checkerboard borders). Just have to select and map each image point to LiDAR points.

• Mapping can be done on any pair.

• Users can navigate from one pair to another using the buttons ‘Map previous file’ and ‘Map next file’.

• Once mapping is done, the user can move out of mapping mode by clicking on ‘Finish Mapping’.

• It's sufficient if the user's map a single pair. There is no requirement to map all the image/LiDAR pairs.

• User can click on ‘Run extrinsic Calibration’, to get the extrinsic parameters

• 'Run extrinsic calibration' button is visible on selecting the image/LiDAR for which the mapping is done.

Visualization Mode:

• Users can toggle ‘Enable Visualization Mode’ to go to visualization mode.

• In this mode, the user can verify the extrinsic parameters by either checking frustum or lidar points on the image.

• Users can project the generated checkerboard on the LiDAR viewer from the image.

• Also, users can add a bounding box and look at its projection in the image.

• Users can manually modify extrinsic parameters to improve those values by simultaneously looking at frustum and lidar points.

Detect corners in all images:

• Once the users confirm the extrinsic parameters. They can fine tune the extrinsic parameters and improve them.

• But the users must make sure that the extrinsic parameters are decent enough by using options provided in the visualization mode.

• For this step, users have to identify the checkerboard corners in all images.

• Auto-detect corners will work for most cases.

• If auto-detect fails, users have to fallback to Manual corner detection. (Please refer to Manual Corner Detection Section)

Improve extrinsic calibration:

• Finally, users can click on ‘Improve extrinsic calibration’. Once the user runs this the algorithm will try to improve the extrinsic parameters.

Analysing the improvement of extrinsic parameters:

• Users can verify the extrinsic parameters in visualization mode as mentioned earlier.

• But after improvising the extrinsic parameters, the user has an option to check and verify the algorithm behaviour as well. (Please refer to Analysing the improved results in Visualization Mode for more details.)

Error stats:

Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. We extract the checkerboard from the raw point cloud of the LiDAR frame and compare it with the checkerboard corners in the 2-D image. The extracted checkerboard can be viewed from the visualizer. The three extrinsic error metrics along with their description are as follows.

• Translation Error: Mean of difference between the centroid of points of checkerboard in the LiDAR and the projected corners in 3-D from an image. Values shown in meters. This calculation happens in the LiDAR coordinate system.

• Rotation Error: Mean of difference between the normals of checkerboard in the point cloud and the projected corners in 3-D from an image. Values are shown both in degree. This calculation happens in the LiDAR coordinate system.

• Reprojection Error: Mean of difference between the centroid of image corners and projected lidar checkerboard points on the image in 3-D. Values shown in meters. This calculation happens in the image coordinate system.

• Individual error stats for each image/LiDAR pair can be seen. Average shows the mean of the errors of all the eligible image/LiDAR pairs.

• If the errors are closer to zero then they are better.

Download calibration parameters:

• Once the entire calibration is done, users can download all intrinsic and extrinsic parameters.

Save calibration dataset:

• We have a save option on the top left corner. A user can click on the Save button to save the calibration dataset at any time during the calibration process.

Checkerboard Configuration Description:

1. Horizontal Corner Count: These are the count of corners in the top row from first to the last. (left to right).

2. Vertical Corner Count: These are the count of corners in the left column from the first to the last. (top to bottom).

3. Square Size: It is the dimension of the square size in meters.

4. Distance from left Corner: The distance from the leftmost side of the board to the left most corner point in meters.

5. Distance from right Corner: The distance from the rightmost side of the board to the rightmost corner point in meters.

6. Distance from top corner : The distance from the topmost side of the board to the topmost corner point in meters.

7. Distance from bottom corner: The distance from the bottom-most side of the board to the bottom-most corner point in meters.

8. Is checkerboard on the ground: Enable this if the checkerboard is on the ground.

Analyzing the improved results in Visualization Mode:

1. Image ‘Checkerboard Identification’:

• This can be used to verify whether the checkerboard area is being properly identified or not.

• Users can change the configuration of the checkerboard or can also retry detecting corners in order to fix the checkerboard identification.

• This step displays the undistorted images. So users can verify if the un distortion is correct or not.

2. Image ‘Raw File’:

• The raw image files are displayed.

3. LiDAR ‘Raw File’ :

• The raw LiDAR files are displayed.

4. LiDAR ‘Extracted checkerboard’:

• This shows the extracted checkerboard from the original lidar file. Used for the error stats calculation. We compare extracted checkerboard with the projected checkerboard.

5. Fused Point Cloud: When a user enables the ‘Fused point cloud’, he can select a fused file among the following.

• Input Cloud: This contains the fuse of all input clouds filtering the checkerboard area. If the checkerboard is not in the LiDAR file, then the user has to fix the extrinsic parameters by going back to the mapping step or manually updating the extrinsic parameters.

• Generated Checkerboard: This contains the fuse of all generated checkerboards. If the checkerboard is not accurate, then the user has to fix the checkerboard configuration or the inner corner detection.

• Input and Generated Checkerboard: This contains the fused output of above two files. This helps us to analyze the difference between the input and the generated output before optimization.

• Checkerboard begin vs after optimization: This helps to know the difference between the generated checkerboard, using the extrinsic values before and after the optimization step.

• Input and Generated Checkerboard after optimization: This contains the fused lidar data of input cloud and generated checkerboard after optimization. If they are overlapped. Then the user can make sure that the extrinsic values are accurate. Or else he can choose to retry improving the calibration results.

Manual Controls to move the generated checkerboard on the actual checkerboard:

• Rotation and axis movement controls are added for the projected checkerboard in the visualization stage. The users can drag the projected checkerboard to align the actual checkerboard in the lidar viewer, extrinsic params are recalculated according to the change that was made. This is an additional way to get the initial estimates of the extrinsic params.

Max correspondence:

This value is used as an input to the algorithm. Users can tweak the value by analyzing the fused point cloud LiDAR files. If the difference between the input and the generated cloud is more, then the user can try to increase the value of the max correspondence and retry improving the calibration results.

Toolbar Options:

• Users have an option to disable the tool tips.

• Users have an option to reset the view of the image/LiDAR to default.

• Users have an option to clear the points/corners added in the image/LiDAR.

Manual Corner Detection:

If the checkerboard corners are not auto-detected. Users can select four boundary points in the order (top-left, top-right , bottom-left, bottom-right). And then click on retry corner detection, to get the remaining inner corners of the checkerboard.

Extrinsic Calibration Output:

• roll, pitch, yaw are in degrees and px, py, pz are in meters.

• roll, pitch, yaw, px, py, pz are the extrinsic parameters downloaded from the calibration tool.

• lidarPoint3D is the 3d coordinates of a point in the lidar coordinate system.

• imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Deep optimisation:

• A new feature has been added which is a deep optimisation. Now the calibration can be further improved with deep optimisation which uses the edge lines of the checkerboard in the optimisation process.

• In the visualization mode, users can use LiDAR drop down and select the Edge points checkerboard to visualize the extracted edges of the checkerboard from the raw LiDAR.

• Users can also use the 2D Line Reprojection Error to verify the individual error value of each pair. This shows the combined reprojection error of all the four lines to the 2D scene.

• Checkerboard should be tilted for enabling deep optimisation. Users also has to check the option that the 'Is checkerboard tilted' to see the deep optimise button on the improve calibration accuracy mode. Please check the Deep optimisation option on the improve calibration accuracy mode and then click on the Improve calibration accuracy for the deep optimisation to run.

Camera sensor coordinates:

We currently show three different types of the camera sensor coordinate system. On selecting the camera coordinate system, the extrinsic parameters change accordingly. The export option exports the extrinsic parameters based on the selected camera coordinate system.

• Optical coordinate system: Its the default coordinate system which we follow.

• ROS REP 103: It is the coordinate system followed by ROS. On changing to this, you can see the change in the visualization and the extrinsic parameters.

• NED : This follows the north-east-down coordinate system.

FAQ:

How do I get the controls to rotate and move the projected checkerboard?

Users can enable the checkbox ‘checkerboard in LiDAR’, the checkerboard will be projected in red color. Select the ‘Bounding Box Select’ option from the tool options of the LiDAR viewer. On hovering over the checkerboard the color of it changes to blue, now select the checkerboard to see the controls. All three rotations and movements are enabled.

This calibration will be deprecated soon. Please use the lane-based targetless vehicle camera calibration instead.

Data collection for targetless vehicle-camera calibration:

1. Mount camera on the vehicle (direction should at least be partially facing the front ).

2. Input the Camera Intrinsics in the config page.

3. Record a scene with 5 to 60 where the vehicle is moving.

4. Keep the speed of the vehicle such that the images captured are not blurry and there is enough overlap between frames of images for feature extraction. Avoid stationary vehicle clips.

5. Having good edges (like buildings) in the images will result in better calibration.

6. The Processing and the feature extraction for the necessary frames from the video takes time.

7. Select a frame where you can easily draw the horizon line and adjust the vanishing point ( Horizon line can be considered as the line segregating the sky and the earth and vanishing point is point where the parallel lines of the road meet at infinity)

8. Click on calibrate

Example data:

Extrinsic Calibration Output:

• roll, pitch, yaw are in degrees .

• roll, pitch, yaw are the extrinsic parameters downloaded from the calibration tool.

• vehiclePoint3D is the 3d coordinates of a point in the vehicle coordinate system.

Calibration Homepage

• This page lets users view, create, launch, and delete calibration datasets. Admins can manage users’ access to these datasets on this page.

• Click on New Calibration to create a new calibration dataset.

Calibration selection

Select LiDAR-Camera Calibration to create a new dataset.

Calibration Instructions Page

Upon selecting LiDAR-Camera Calibration, the user is welcomed to the instructions page. Click on Get started to start the calibration setup.

Approach selection

Users can choose either the target-based or the targetless calibration. The target-based calibration uses the checkerboard/charucoboard as the calibration target, and the targetless calibration uses the scene captured in both LiDAR and the camera sensor data.

Configuration

Target-based

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Checkerboard target configuration

• Horizontal corners: Total number of inner corners from left to right. The blue dots shown in the above preview correspond to the horizontal corners.

• Vertical corners: Total number of inner corners from top to bottom. The red dots shown in the above preview correspond to the vertical corners.

• Square size: This is the length of the square's arm in meters. It corresponds to the length of the yellow square highlighted in the preview.

• Left padding: The distance from the leftmost side of the board to the left-most corner point in meters. Corresponds to the left blue line in the preview.

• Right padding: The distance from the rightmost side of the board to the rightmost corner point in meters. Corresponds to the right blue line in the preview.

• Top padding: The distance from the topmost side of the board to the topmost corner point in meters. Corresponds to the top red line in the preview.

• Bottom padding: The distance from the bottom-most side of the board to the bottom-most corner point in meters. Corresponds to the bottom red line in the preview.

• On ground: Enable this if the checkerboard is placed on the ground and the point cloud has the ground points in the scene around the checkerboard placement.

• Tilted: Enable this if the checkerboard is tilted.

Charucoboard target configuration

• Rows: Total number of squares in the horizontal direction.

• Columns: Total number of squares in the vertical direction.

• Square size: It is the length of the arm of the square in meters.

• Marker size: It is the length of the arm of the aruco marker in meters. This is usually 0.8 times the Square size.

• Left padding: The distance from the board's left edge to the left of the first square in the row.

• Right padding: The distance from the board's right edge to the right of the last square in the row.

• Top padding: The distance from the board's bottom edge to the bottom of the last square in the column.

• Bottom padding: The distance from the board's top edge to the top of the first square in the column.

• On ground: Enable this if the checkerboard is placed on the ground and the point cloud has the ground points in the scene around the checkerboard placement.

• Tilted: Enable this if the charcoboard is tilted.

Targetless

Camera Intrinsic Parameters

Intrinsic parameters for the camera are to be added here. Users have three options.

• Users can use the Camera Intrinsic calibration tool to calibrate the results, save them to the profile, and then load them here. For more details, click here.

• Users can also load the JSON file.

• Users can manually enter the intrinsic parameters if they already have them.

Upload files from LiDAR and Camera

Add point cloud files from the LiDAR and images from the camera sensor. After adding, pair the point cloud files with the matching image files before continuing.

sample CSV format

X, Y, Z
0,-0,-0
62.545,-3.5064,-3.5911
62.07,-3.5133,-4.1565
32.773,-1.8602,-3.4055

Detect target corners in images (Only for target-based)

Users can click on Detect corners to detect the corners in the target. This is an automated process, and our algorithm usually detects the corners in the image accurately.

Suppose the target corners are not auto-detected. Users can follow the steps below and add the four boundary markers to get the inner checkerboard corners.

Estimated extrinsic parameters

Target-based

The extrinsic parameter space is vast, so we need an estimated entry point for optimization. The user can provide estimated extrinsic parameters in three ways.

Mapping of target corner points

Users can map the target corner points in the point cloud and get the initial estimates of the extrinsic parameters. Only one point cloud mapping is sufficient to get the initial estimates.

Auto detect target

Our algorithms can automatically detect targets in the point cloud if the lidar channel data is provided on the configuration page. Please note that the auto-detection might not work properly if there are many flat surfaces, like walls, ceilings, etc., in the scene.

Add estimated extrinsic parameters

Users can manually enter estimated extrinsic parameters.

Targetless calibration

Mapping of corresponding points

To get the initial estimates, users can map any four corresponding points from the image and the point cloud data.

Manually enter extrinsic parameters

Alternatively, users can add the initial estimates if they know them. In such a case, users can skip manually adding the markers. Users can click Add estimated extrinsic parameters to add the initial estimates.

Verifying the accuracy of the estimated extrinsic parameters

Once the estimated extrinsic parameters are in the tool, users can visualize the parameters by clicking on the visualize button. In the visualization, we have a few sensor fusion techniques through which the accuracy of the extrinsic parameters can be visualized. For more details, visit Sensor fusion techniques.

Estimated extrinsic parameters are crucial in generating accurate extrinsic parameters.

To get good initial estimates, users must clear the markers and redo the markings if the estimated parameters are way off.

Segmentation: (Only for Targetless Calibration)

There are two types of segmentation approaches available for the user to select:

1. Auto segmentation: This approach automates the segmentation of vehicles in point clouds and images using a deep learning model trained on various datasets.

2. Manual segmentation:

   1. Lidar: In this approach, the user needs to add bounding boxes in the lidar frame and fit the boxes to vehicles in the point cloud. The bounding boxes must be added for all the point clouds uploaded for calibration. This can be done by selecting the Bounding box mode, adding bounding boxes, and clicking Save Labels.

   2. Image: There are two ways to do manual segmentation

      1. Semantic Painting: Users can use the brush to paint the vehicles in the image and click on Save Labels.

      2. Segment anything: In this approach, Users place a cluster of points on each vehicle. The same vehicle points should be placed under the same category. Please place at least one point on each surface of the car, such as the windshield, sides, roof, etc., so that when the model runs, it doesn't miss any part of the vehicle. After placing the points in each image, please click on the save label to save the data.

Note: Auto segmentation is suggested initially. Based on the segmented vehicles in the point clouds and images, the user can decide whether to proceed with auto-segmentation or perform the segmentation manually.

Run Calibration

Users need to click on Calibrate to optimize the estimated extrinsic parameters further. All the uploaded pairs are used in the optimization process.

Additional options in the run calibration

Target-based

Deep Optimization: Users can select deep optimization to optimize the extrinsic further for datasets with the Tilted option enabled on the configuration page.

Max correspondence: This value is used as input for the algorithm. Users can tweak the value by analyzing the fused point cloud LiDAR files. Suppose the difference between the input and the generated cloud is more significant; the user can try to increase the value of the max correspondence and retry to improve the calibration results.

Targetless

Users can optimize only angles by selecting the Angles only check box. It is observed that enabling Angles only results in achieving better Sensor angle accuracy (note that sensor position is not optimized in this case).

Error stats (target based)

Users can use these error values to estimate the accuracy of the calibration results alongside visual confirmation. The closer the error stats to zero, the better the extrinsic parameters.

• Translation Error: Mean of difference between the centroid of points of checkerboard in the LiDAR and the projected corners in 3D from an image. Values are shown in meters. This calculation happens in the LiDAR coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

• Plane Translation Error: Mean of the Euclidean distance between the centroid of projected corners in 3D from an image and plane of the target in the LiDAR. Values are shown in meters. Note: If the board is only partially covered by the LiDAR or the LiDAR scan lines are non-uniformly distributed, translation and reprojection errors are inaccurate, but this plane translation error is accurate even in these scenarios.

• Rotation Error: Mean difference between the normals of the target in the point cloud and the projected corners in 3D from an image. Values are shown in degree. This calculation happens in the LiDAR coordinate system. Note: All LiDARs have noise when measuring distance. This will, in turn, cause noise in the target's point clouds and the target's normals. Usually, this metric cannot measure accurately below 1 degree. For an accurate rotation error, we suggest using a faraway straight edge such as a building edge, roofline, or straight pole and projecting the point cloud onto the image. The rotation error can be calculated from the number of pixels between the image edges and the projected points.

• Reprojection Error: Mean difference between the centroid of the target corners from the image and the centroid of the projected target from the LiDAR space onto the image. This is calculated in the image coordinate system. Note: If the board is only partially covered by the LiDAR, this value is inaccurate due to the error in the position of the centroid.

• Individual error stats for each image/LiDAR pair can be seen. The average shows the mean of the errors of all the eligible image/LiDAR pairs.

Download calibration parameters

Once the entire calibration is done, users can download all intrinsic and extrinsic parameters by clicking the Export button in the header.

Analyzing the extrinsic parameters in Visualization Mode:

Sensor fusion techniques

Users can use the following techniques to visualize the extrinsic parameters.

Frustum: Users can see the image's field of view in the LiDAR frame. This uses both the camera matrix and the extrinsic parameters. Image axes are also displayed according to the extrinsic parameters.

LiDAR points in image: Users can see the LiDAR points projected in the camera image using extrinsic parameters.

color points from camera: Users can see the camera's color points in the lidar space using the extrinsic parameters.

• Show target in LiDAR: Users can see the checkerboard points projected in the LiDAR frame using the extrinsic params.

Image: Target Identification

The target in the image is filled with points. If the target configuration the user provides is correct, there will be no overflow or underflow.

LiDAR: Extracted target:

This shows the extracted target from the original lidar file. We use this to calculate the error statistics. We compare the extracted target with the projected target.

Fused Point Cloud:

Targets from all the point clouds are cropped and fused into a single point cloud.

• Input cloud: This contains the fuse of all input clouds filtering the target area. If the target is not in the LiDAR file, the user has to fix the extrinsic parameters by going back to the mapping step or manually updating them.

• Generated target: This contains the fuse of all generated targets. If the target is inaccurate, the user has to fix the target configuration or the inner corner detection.

• Input and generated target: This contains the fused output of the Input cloud and Generated target. This helps us to analyze the difference between the input and the generated output before optimization.

• Target begin vs after optimization: This helps to know the difference between the generated target, using the extrinsic values before and after the optimization step.

Validate Ground Truth:

To verify the extrinsic parameters obtained from the calibration, we have an additional step that shows how close the final extrinsic values are to the actual extrinsic values of the setup.

Steps to be followed to validate ground truth:

1. Select the Validate Ground Truth option displayed at the top panel in the visualizer

2. From the image, select any edge of the board that will be used for error estimation

3. Draw a line that exactly matches the edge selected from the image, which is called the Ground Truth line

1. Draw another line joining the edge of the points that are projected from lidar onto the image (Projected Line)

1. After adding both lines to the image, click the Validate Ground Truth button in the right panel. This generates the ground truth Angle and Pixel errors.

Extrinsic Calibration Output.

• roll, pitch, and yaw are in degrees, and px, py, and pz are in meters.

• lidarPoint3D is the 3d coordinates of a point in the lidar coordinate system.

• imagePoint3D is the 3d coordinates of a point in the camera coordinate system.

Camera coordinates system

We currently show three different types of camera coordinate systems. The extrinsic parameters change according to the selected Camera coordinate system. The export option exports the extrinsic parameters based on the selected camera coordinate system.

• Optical coordinate system: It's the default coordinate system that we follow.

• ROS REP 103: This is the coordinate system followed by ROS. When you change to this, you can see the change in the visualization and the extrinsic parameters.

• NED: This follows the North-East-Down coordinate system.

Sample Script

This is a sample Python script to project lidar points on an image using extrinsic parameters. It uses the open3d and opencv libraries.

Tool usage guide for old UX

Calibration Target:

Checkerboard is the calibration target. For small vehicles, the minimum checkerboard size is 0.6m^2. For large cars, the minimum checkerboard size is 2.4m^2. If calibrating a small car, you can print the below pdf file on a foam board at 1.0m x 0.6m. Most print shops can print this. https://drive.google.com/file/d/1mTR8HTpvROE1Pv0rmXEBVLSxs_yMDnvf?usp=sharing

For car wheels, use the Aruco markers. Click here for Aruco markers. Note that the ArUco markers should be as large and flat as possible with a white border. The white border is necessary for tag detection.

Data collection for vehicle-camera calibration:

1. We need two cameras,

   1. A mounted camera on the car with respect to which calibration is done against the vehicle.

   2. An external camera to take images from both left and right sides of the vehicle. Note: The external camera should have a fixed focal length. Changing focal length / auto-focus will change the camera intrinsic parameters. A DSLR with manual fixed focal length can make a good external camera. Modern cell phone cameras all have auto-focus so should not be used as an external camera.

2. Intrinsic parameters for both mounted and external cameras before proceeding to vehicle-camera calibration. Intrinsics can be obtained from the Camera Intrinsic calibration as well.

3. Have a checkerboard with known horizontal corners, vertical corners, and square size.

4. Stick Aruco markers to the vehicle wheels to auto-detect the wheel center. Click here for Aruco markers. Note that the ArUco markers should be as large and flat as possible with a white border. Note: The ArUco markers must match the wheel position. Mismatched markers will not be recognized in the application.