1 of 100

Deepen AI - Enterprise

Deepen AI Overview

What is Deepen AI Suite? Deepen AI is the fastest way to annotate data to build and ship artificial intelligence applications. Deepen AI Suite provides enterprise-grade AI-accelerated tools focusing on image, video. and sensor fusion (camera, LiDAR, radar, and more) training data. Who is it for?

Believe commercial solutions will be a far better choice than building and maintaining internal tools and infrastructure.
Want the highest quality standards for creating training data.
Want to standardize creating and managing training data across various teams within the organization.
Want to leverage labeling automation and reduce costs in human labeling.
Want the ability to work with internal labeling teams (usually experts) as well as fully managed external labeling service.
Want to leverage state of the art labeling tools for creating training data.

FAQ

General FAQ:

What is Deepen AI? Deepen is an industry-leading computer vision training data solution with fast AI enabled labeling tools, labeling automation, human workforce, and data management.
Does Deepen only sell labeling services? Deepen sells both AI-enabled labeling tools and labeling services, which includes: -An image and video annotation tool and services -Multi-sensor (Lidar, Radar, Sonar, Camera and more) fusion annotation tool and services -Quality and performance management tools.
How do I set up a Deepen project? Once you have created a Deepen account, follow the simple steps on this page to set up a project.
Do I need to be an AI engineer to use Deepen tools? No, we designed our user-friendly platform to require minimal technical knowledge and expertise such that anyone can annotate data using our out-of-the-box web application. However, it may be beneficial to have someone experienced in data science on your team to help ensure that your machine learning project is successfully developed and deployed.
What is the difference between Deepen 2D and Deepen 3D? Deepen 2D includes annotation capability for camera data such as images and videos. Deepen 3D supports multi-data annotation which includes the camera, LiDAR, radar, and more.
Which internet browser should I use? Google Chrome will be recommended when using Deepen tools.

Saas & On Premise

Our powerful and user-friendly data annotation platform is available as a managed service, in the cloud, or on-premise. SaaS Offering: Host your data on Deepen servers and receive automatic updates and access to all features. Hybrid On-premise: Host your data on your own servers and provide Deepen a URL to an asset that is behind a VPN. All source data will be unreachable by Deepen but generated assets(annotation data) will be stored on Deepen servers. Full On-premise: Work with our engineering team to deploy

Data Management

Data Management Overview

Our data management workflow is designed to help managers manage a team of annotators in a seamless manner, yet produce high-quality results. It includes features such as task assignments on a user level, multi-stage quality assurance down to each label, and exporting your labeled data with just a click.

Creating/Uploading a dataset

Overview: Create, configure, and upload your dataset to your workspace.

Steps to create a dataset:

In your workspace, click on “Create dataset” in the top right corner.
Enter a dataset name (use alphanumeric characters only. Special characters are not allowed).
Select a dataset type (Images/Video/3D Point Cloud).
Select Dataset Format (JSON) when the 3D point cloud is selected.
Select Frames per second when Video (only .mp4 files are supported) is selected from the Dataset type.
Select a labeling mode (Individual/Sequence).
Select a dataset (Only Zip files are allowed) by clicking on browse from the computer, or drag and drop the dataset.
Import dataset configuration from profile (if any profile is already created).
Categories and Attributes can be configured if there is no Dataset Profile
Enter Labelling Instructions if required.
You can also manage your team by assigning them certain permissions like Labeller, Labels Admin, or Reader for Local Users or Local Groups.
Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of the pipeline as you want.
Enable 2D Tools in 3D Editor (only for 3D datasets) to label 2D annotations for the 3D datasets.
Users can enable customer review/Enable auto-task assignments while creating the dataset if needed.
Add a new tag or select an existing tag to find the dataset easily.
Click on Create to create a dataset.
Once created, you can see the dataset with the “Pending upload” status and the file count after finishing the upload process.
Refresh the browser once the data is uploaded.

Here is the 3D input format for uploading a dataset:

Create a dataset profile

Overview: Users can add different type categories and add attributes at label and category levels.

Steps to create a dataset profile:

Click on the “dataset profile” at the left panel.
Create a profile name e.g.: My Image profile.
When configuring categories, choose a label type. Then, create a category by typing the name and click enter the box below.
For 3D bounding boxes, you can configure your default bounding box size by clicking on the category.
You can also add the label attributes specific to the category when you select a category.
Common attributes are attributes that apply to all categories.
Frame Attributes are attributes that apply at a frame level.
Dataset Attributes are attributes that apply at the dataset level.
Auto-task assignments can be enabled by clicking on Enable auto-task assignments.
Next, enter the labeling instructions (optional).
Lastly, configure the workflow pipeline by inputting the name and selecting the checkbox to allow labeling activity in that pipeline. You can add as many stages of pipelines as you want.
In the Image Editor user can give the attributes to the labeled objects by pressing shortcut keys or from the tool itself.

Auto Task Assignments

How to enable auto-task assignments for datasets?

Users with admin permission can go to the tasks screen in the dataset details page, and turn on the auto-task assignment feature using the Auto-assign option. Users can also set an assignment config for each stage by clicking on the menu settings icon beside the Auto-assign option.

Admin needs to add users to use the auto assignments feature.

How does this work? When auto-assign is enabled. Users having labelling permission can go to the tasks page at the workspace level and send a request for work by clicking on the Request work button. once the user requests work, tasks will be assigned to the user.

Task Life Cycle

We're introducing an improved lifecycle for the tasks in the dataset to streamline your dataset workflow experience. Here are the new task states we support:

Waiting: Indicates the task is in waiting state and needs to be submitted from a previous stage

Ready: Indicates the task is ready for you to start working on

In progress: Indicates the user is working on the task

Submitted: Indicates the task was submitted for review, and the next stage user has to verify and accept it. Please note that you will not be able to modify a task once submitted.

Accepted: Indicates the task was verified by the next stage user and has approved it and no changes would be further required in it. Please note that you will not be able to reject a task once accepted.

Rejected: Indicates the task was rejected by the next stage user and needs to be fixed (users can refer to issues if any were created for the task before rejecting the task). Rejected tasks need re-work and have to be submitted again for further review.

Note: Once a task is submitted user can't edit the task in that stage.

Task options you see for different possibilities:

If you are in a stage after the first one (e.g. second, third)
- If the previous stage was not accepted yet, you see Accept & modify, Accept & submit and Reject
If you are in the last stage of the pipeline
- Once you've accepted the previous stage task, you see Finish.

New tasks page inside a dataset

Users with admin scopes and customers will now have a new tasks screen visualizing the flow of the work in the dataset across all stages. Only users with dataset admin or above scopes will be able to assign users to the tasks.

When clicked on a task in the Task column (left-most column), the task is launched in the editor with all the required options set automatically, so you can start working without worrying about which stage you have to go to or which label set you to need to set to.

Tasks Assignments

Overview: Tasks can be assigned in both labeling and configured pipeline stages for added users. Steps for Task Assignments: You can assign the tasks to the user in the dataset level under “Tasks”. 1. Click on Manage tasks for assignments. 2. Here we can assign the files to both the labeling and QA and other configured pipeline stages.

3. Admin needs to select the user email from the drop-down boxes and select the files from the unassigned files or enter the range(E.g: 1-4,5,6-11). 4. Labelling users can check the assigned files from the editor by clicking on Your work at the centre of the editor.

5.There are some filters by which the user can sort based on their preferences.

6. Users can move to the assigned frames/labels in the editor by clicking on the next task.

Creating a group

Overview: Creating a group allows you to create a group within a workspace.

Creating a Group: Users can create a group by clicking on the create group on the Users page and add other users to it.

User Permissions: Permissions can be changed at the datasets levels for different Users by using user permissions.

User Group: Permissions can be set for a group of users by selecting a group and every user has the same permissions for the selected datasets.

Adding a user

How to add a user:

When a workspace is opened there is the Users option on the left side panel.
Click on the Users option and at the top right click on Add user.
Enter valid email and select the scope for the user and click on Add User.

Permission rights of each user: Deepen AI annotation tools have different user access levels for different tasks. Different roles and their description are as follows:

Admin: This role can view/add/edit/delete a project, add/delete a dataset, add users, add payment details, sign up for different pricing plans, assign users, add files and add labels.
Reader: This role can view all the labels, reports and profiles but cannot make any changes.
Datasets Admin: This role has access to create and delete datasets. They can assign frames to already added users. They cannot label or add any users.
Datasets Creator: This role has access to add datasets, but cannot delete datasets.
Labels Admin: This role can add and delete labels.
Labeller: This role has access to annotate the allocated frames (Labelling and QA)
Customer: This role has permission to accept or reject labels in the workspace.

Apart from these, we have additional scopes for Access Tokens. They are:

Dataset Labels Admin: This role can add and delete the labels on the dataset level only.
Dataset Labeller: This role has access to annotate the allocated frames(Labelling and QA) on the dataset level.
Dataset Calibration Edit: This role can view, create, manage and delete calibration datasets at the dataset level.
Dataset Calibration Read: This role can view all at the dataset level for calibration dataset.
Dataset Reader: This role can view all the labels, reports and profiles but cannot make any changes at the dataset level.
Dataset Customer: This role can accept labels at the dataset level.

Embed labeled dataset

This feature allows you to share you labeled data on your own site. This feature is only available for point cloud as the moment.

Overview: Share your labeled dataset on your own site

Steps to embed link:

To enter the embed labeling screen, go to “Share” on the dataset page.
Enter the title and number of frames you wish to display. A unique embedding link wrapped around <iframe> tag will be automatically generated.
Paste the link in your HTML or JavaScript editor.

Export

Overview: Users can export Labels/Dataset/Profile using the export button.

Export labels

Overview: Users can export labels using export labels by using different methods. Steps to export data:

Labels that are done can be downloaded from the tool in JSON format.
Only labels which are marked as done are exported as JSON files.
Labels: Labels in the last pipeline stages will be downloaded. (Files should be marked as done)
Labels by file names: Download labels from the last pipeline stage for all label types, with the file name as the key in JSON format.
Customize your export: Files can also be downloaded by selecting particular frames, particular annotations, categories, attributes, types etc. (Files need not be marked as done if labels are downloaded using these).
2D Semantic labels can be downloaded by clicking on the 2D Semantic Segmentation option.
Video for downloading paint labels https://drive.google.com/file/d/1EUzTqbRlyW85ZMgOJS74jCPTXk70vhH0/view?usp=sharing

Note: 2D Semantic segmentation output can be found here

If the labels are not marked as done, an empty JSON file will be downloaded when clicked on Labels.

Import Labels

Overview: Import Labels allows users to import existing dataset labels from a dataset to another dataset in JSON and DPN(3D Segmentation) format for both 2D and 3D.

Users can find the import labels feature beside the launch button(If customer review is enabled, it is beside Launch review) on the dataset page.

Once import labels is selected, the user can select the format and click on Import Labels. Labels will be uploaded in a few minutes based on the file size.
Users can select Import into label set if they want to import labels into a different Label Set.
If web paint is selected, a metadata file should also be uploaded in JSON format. The segmentation format can be found at Data Output Format - 3D Semantic Segmentation

Import profile

Overview: Import Profile allows you to import an existing dataset configuration from a dataset to another dataset

You can find the import label feature at the top right under the 3 dot dropdown in a dataset.

Steps for Importing Labels:

1. Click on Select Profile and select an existing profile that is already created.

2. Click Next and Customise profile.

3. Once the above step is done click on Next: Finish and Click on Import config.

4. You can view your imported labels in the label set you selected on the labeling screen.

Import profile via JSON

Overview: Importing a dataset configuration via JSON

Go to the Datasets tab, find the import profile feature at the top right under the 3 dot dropdown

2. Click on import profile, there you can click on import profile from JSON and select a JSON file.

3. Customize what you wish to import

4. Once you've confirmed, click on "import config". All the configurations in the dataset will then be changed according to the JSON file you've chosen"

Access token for APIs

Overview: Access token is created for admin to access different APIs in your specific workspace.

How to create an access token:

Go to the left panel, under "Developer settings" click on "Access Token".

2. Click on "Generate new token" on the top right of the screen to start generating a token.

3. Fill in your note and select the scope of users allowed to access the token.

4. Click on "Generate token" to create token access.

5. This token will not expire, you have to revoke the token access manually.

How to view/revoke created token:

Once the token is created, go to "Developer Tokens" to view the token.
You can click on the token to copy it and click on the "revoke" button to revoke the token manually.

Data Streaming

Users can now stream data from their cloud buckets directly into the app without having to upload it. Whatever state of files users have in the bucket is automatically synced with the corresponding files in users datasets. With this now, users can also add or delete files from datasets by simply adding/deleting them in your bucket.

Users can create a Data stream by clicking on create Data stream button on the data streams page.

Images and Point clouds(JSON Format) are supported as of now, and we are working on adding more file type supports.

User needs to choose which cloud provider they need and then provide few details which are bucket name(bucket where your data is stored), folder name(location of the folder that will be synced to create a dataset present within the bucket), credential file(a json file with the private key of one user that can enable us to authenticate to your bucket location) and optionally user can provide a dataset configuration profile which will be applied to all the datasets that are created using this data stream configuration.

Once a data stream configuration is created, you can click on the "Sync bucket" option to sync your datasets with your bucket instantly. Post that you can use this dataset like normal datasets and it has all the native functionalities that are already supported by Datasets.

Reports

The Deepen reports offer a complete view of the dataset management environment. They track important aspects such as data quality, user contributions, labeling progress, and real-time activity within the workspace. This information is vital for making informed decisions and optimizing the workflow for dataset-related tasks.

Dataset Reports:

The dataset report provides information about the datasets in the workspace. It includes details such as the number of frames, labels, time spent on the dataset, average time across all stages, and completed tasks.

User Reports:

The user report gives information about the users working on the dataset. It provides details such as the total number of labels created and the average daily time spent.

Quality Reports:

The Quality report relates to issues that arise within the dataset, providing information on the status of each issue.

Labelling Reports:

The Labelling report provides information about the labeling categories, label type, label count, and stage ID.

Real-Time Reports:

The Real-time report provides real-time updates about the current workspace. It allows users to track the number of users who are working on the workspace.

Productivity Reports:

The productivity report provides information about the user's productivity across all label types and stages.

Assessments

Assessments are benchmark exams designed to test annotators labeling and reviewer's skills.

Assessments are designed to automatically evaluate the proficiency of users, generate a score, and produce a comprehensive report. At present, we provide support for automatic evaluation of 2D bounding boxes. However, we are committed to expanding our support for additional label types soon. To create an assessment, follow the below steps:

Open the datasets page and click on “More”.
Click on “Assessments” and then “New” to create a new assessment.
Enter the assessment name, attendees, passmark, IoU matching threshold, time limit, and maximum attempts.
Select a benchmark dataset that has labels and is marked as “Done”.
Once the assessment is created, Users can launch the dataset and start the assessment.
Annotate the bounding boxes and submit the assessment once all the labels are annotated.
You can go back to the assessment page to view the results. If you fail the assessment, you can reattempt it.
To compare the assessment results with the benchmark dataset, the assessment creator must click on the view results button.
The results show missing objects and false positives.

2D/3D editors

Editor Content

2D Label Types: 2D Bounding Boxes 2D Polyline/Line 2D Polygon 2D Points 2D Semantic Painting 3D Bounding boxes on images

3D Label Types: 3D Bounding Boxes 3D Lines 3D Polygons 3D Semantic Segmentation 3D Instance Painting

AI sense (ML-assisted labeling)

Overview: Labels are generated automatically based on the image, this is also known as pre-labeling.

Steps for using AI Sense: 1.Go to the Editor Page and Click on the AI Sense button from the right side menu. 2.Click on Auto Detect. 3.Click on Preview to view the labels or Add All to add the labels. 4.Users can add individual labels or use Confidence Score to detect labels. 5.Make sure the category is given before using AI Sense.

Categories supported for AI Sense: Single Frame Detection categories:(2D Bounding Boxes)

person: A person walking in the scene (usually on sidewalks, crosswalks, outside driveable space
bicycle: Human or electric-powered 2-wheeled vehicle designed to travel at lower speeds either on the edge of the road surface, sidewalks, or bike paths. A driving bicycle is also considered a bicycle.
car: All Sedan, Suv, minivans, and sports cars are marked as cars
motorcycle: Gasoline or electric-powered 2-wheeled vehicle designed to move rapidly (at the speed of standard cars) on the road surface.
airplane
bus: A road vehicle designed to carry many passengers.
train
truck: Motor vehicles designed to transport cargo, goods, merchandise, and a wide variety of objects.
boat
traffic light: A road signal for directing vehicular traffic by means of coloured lights, typically red for stop, green for go, and yellow for proceeding with caution.
fire hydrant: a fitting in a street or other public place with a nozzle by which a fire hose may be attached to the water main.
stop sign: A stop sign is a traffic sign designed to notify drivers that they must come to a complete stop
parking meter
bench
bird
cat
dog
horse
sheep
cow
elephant
bear
zebra
giraffe
backpack
umbrella
handbag
tie
suitcase
frisbee
skis
snowboard
sports ball
kite
baseball bat
baseball glove
skateboard
surfboard
tennis racket
bottle
wine glass
cup
fork
knife
spoon
bowl
banana
apple
sandwich
orange
broccoli
carrot
hot dog
pizza
donut
cake
chair
couch
potted plant
bed
dining table
toilet
tv
laptop
mouse
remote
keyboard
cell phone
microwave
oven
toaster
sink
refrigerator
book
clock
vase
scissors
teddy bear
hair drier
toothbrush

Sequence tracking model categories:(2D Bounding Boxes)

car
truck
person
bus
bike
rider: A person driving a bicycle or motorcycle is considered as the rider
motor
train

Categories for 3d Bounding box (Sequence and Single Frame):

vehicle: All Sedan, Suv, minivans, and sports cars are marked as cars

2. pedestrian: A person walking in the scene(usually on sidewalks, crosswalks, outside driveable space)

3. cyclist: A Person who is riding a Cycle

Assisted 2d Segmentation

Overview: Users can now perform instance segmentation tasks in polygon mode for single-frame 2D image datasets with AI assistance. In order to use the feature, the user needs to follow the steps below.

Creating the Segmentation Mask

Go to Polygon mode
Choose the “2D Segmentation” button

Draw a tight box enclosing the object to be segmented
AI server will return an output polygon mask covering the object

Refining the Segmentation Mask

Users have an option to edit/refine the masks using the pre-existing tools in the polygon labelling mode or they can use the AI assistance to refine the polygons automatically. To use AI assistance, the user needs to follow the steps below:

To activate AI refinement mode, select the label which was created using the “2D Segmentation” button.
Choose the “Refine Polygons” button (pencil) besides the “2D Segmentation” button.
In the refinement mode, the user can plot two kinds of points as below.
- Green indicates the regions which the user wants the model to include in the segmentation mask.
- Red indicates the regions which the user wants the model to exclude from the segmentation mask.
The user can plot Green points on the image using “Left Mouse Click”.
Points To add the Red points, the user can use the “Left Mouse Click” with “Alt” hotkey.
As soon as any of the above points are added, an API call will be made to refine the points.
The segmentation will be refined using the input provided and the polygon will be updated accordingly. The refinement mode will be deactivated once the polygon is refined. You need to activate the refinement mode again using the “Refine Polygons” button in the secondary menu bar to make further edits.
To add multiple points at once before refining the polygon, the user can use the "Shift" hotkey along with the process described above. The last point in this scenario needs to be added with “Shift” hotkey released for the polygon to be refined.
The user can delete existing points using “Ctrl” + “Shift” hotkeys along with "Left Mouse Click" on the point to be deleted.

Scene Labeling

Overview: A scene label is a behaviour or an event or an action by an object or a scene in the sequence.

Steps to enable scene labeling:

To start enabling scene labeling, upload a dataset as a sequence datasets or video that is not split into frames. By checking the "use video directly instead of splitting into frames" option, it will not the video into frames and rather upload the entire video.
Create a category under scene labeling. Ex: cut in
Clicking on the New Scene Label button at the bottom of the editor creates a new label from the user's current frame with the selected category. Ex: cut in:1

Steps to create a scene label:

Click on the label name or track selects the label and will enable the "Start Scene" and "End Scene" buttons.
Navigating to the desired frame and clicking on the "End Scene" buttons will extend the track from the already existing start point to the selected frame.
Upon selecting the frame number below the already existing frame and click on the "Start Scene" button extends the track to the left towards the user's new start frame.
Click on label or track selects the labels which enable the user to click on the edges of the track and drag towards the user's desired frame.

2D Editor

2D Editor Overview

Overview: By using 2D Editor users can annotate 2D Bounding Boxes,2D Polygons, 2D Lines, 2D Points, 3D bounding boxes and Semantic painting for the camera images.

Primary Toolbar: It has Home, Select and Move Tool(V), Pan Tool(2), Select and Move, Zoom In, Zoom Out and Create Issue.

Home Button: Home button is used to navigate to the Homepage.

Select and move tools(V): Users can move or edit labels
Pan(2): This Option is used for moving images. Users can’t label images if this option is selected.
Zoom In: Users can Zoom in by selecting this option.
Zoom Out: Users can Zoom Out by selecting this option.
Create Issue: Users can create issues at the frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar. More information on issues can be found here.

The Top toolbar consists of Frame, Label, View and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete. Polygon annotation additionally has Attributes, Polygon Segmentation Mode and Polygon Drawing Mode.

View: View consists of Canvas view, Show Labels Text, Show Labels Attributes, Grid Size, Show Grid, and Show Merge Options for Polygons.

Secondary Menu bar: It consists of Label Set, Stage, Label type, Point size, Labelling Instructions, View Editor, View Original Image, View Masked Image(only for Polygons) and Reset View.

Panel Content: It is on the right of the image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these it also has Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular file.If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on the workspace the tool navigates the user to the workspace page in the tool.

Buttons in the lower navigation bar and their usage:

Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.
Preload Data: It loads the frames in advance before opening them.

User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

User can hide the bottom bar using the Hide bottom bar option for sequence annotations.

2D Bounding Boxes

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to draw a Bounding Box:

Click on Label type and select the Bounding box from the top left menu of the editor.
Select a category from the drop-down box.
Draw a bounding box by clicking on the image with the left mouse click and Drag the 2D-box up to the corners of the object.
You can select the label by clicking on the label with a left mouse click. Then you can edit the 2D-Box, by dragging the highlighted points of the box with a left mouse click.
Labels are saved automatically.
Click “control + shift” to draw a bounding box within a box.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

2D Polyline/Line

Overview: Polyline or Line annotations are continuous lines made of one or more line segments. The entire line is treated as one object. Polylines are typically used to annotate open shapes since they don’t have to be closed as polygons do. This tool is often used to label shapes like road lane marks or other similar shapes.

Steps to Draw a Line:

Click on Label type and select Line from the top left menu of the editor.
Select a category from the drop-down box.

2D Polygon

Overview: The Polygon tool is used to create geometric annotations by drawing lines between points.

Steps to Draw a Polygon

Click on Label type and select Polygon from the top left menu of the editor.

2. Select a category from the drop-down box.

3. Draw a polygon, keeping points on the image with a left mouse click. You can label the same polygon ID by holding an “Alt” key for the first polygon point or by selecting a label and clicking on Draw selected polygon. Then continue the polygon labeling for the object with the same ID.

4. You can select the label by clicking on the label with a left mouse click. Double-click on a polygon to edit it by dragging the highlighted points of the line with a left mouse click and also you can delete the selected highlighted point by pressing the “Delete” key. To add a point on the line, click on the line using the left mouse click.

5. You can draw another line by pressing the “Esc” key twice. 6.Labels are saved automatically. 7. Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Polygon (Foreground/Background): Knowing how to use the foreground and background can come in handy in data labelling for polygons. If there is an object at the back of another object, that is when this feature comes into play. Draw a polygon as Foreground:

1. Selecting a foreground option from the image editor page

2. For example, if you want to label a “Car” in the below image without interrupting the “Building” polygon, you want to select a Foreground option before saving the “Car” label as the Building is the Background and car is the foreground object.

3. If you use Background instead of Foreground option, it will not be cut by the Background polygon label(Building).

Draw a polygon as Background:

Selecting a Background option from the image editor page.

2. For example, if you want to label a building in the below image without interrupting the car polygon, you want to select a background option before saving the building label because the Building is the Background and the car is the foreground object.

3. When you label a background object with a background option, it will get a polygon accurately to the object without disturbing the foreground label (Car). 4. If you use the foreground instead of the background option, it will interrupt the foreground polygon label (Car).

2D Semantic/Instance Segmentation

Semantic Segmentation

Overview: Semantic segmentation is the task of classifying each and every pixel in an image into a class or category. 1. Select an instance or semantic segmentation by clicking “Label” at the top panel

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the semantic if the objects are touching each other than their drawn polygons are merged with each other with a single id. 4. In the below image, you can see the merged polygons with a single id

Instance Segmentation:

Overview: Identify each object instance of each pixel for every known object within an image.

Select an instance or semantic segmentation by clicking “Label” at the top panel.

2. Draw an object with a polygon using the left mouse click. 3. When you are labeling objects in the Instance segmentation, each object can be labelled distinctly with a unique ID. 4. In the image below, you can see the object’s polygons with different IDs like Car1, Car2 and Car3.

2D Segmentation (foreground/background)

Overview: Painting foreground and background allows you to segment an object occluded in the background

Steps to segment foreground or background:

Knowing how to use the foreground and background are very useful in data labeling.
Start labeling by using left-click.
When we label the 1st object which was occluded by another 2nd object. It should be labeled with an extra region of the object.
By using the foreground option for the 2nd object which was in front of the 1st object. It will cut the 1st object polygon accurately.
By using the background option we can label an occluded object without affecting the polygon in front of the occluded objects
Click “control + shift” to draw a Polygon within a polygon.

2D Points

Overview: The key point annotation is used to detect shape variations and small objects, it helps to understand each point motion in the targeted object.key points can help in facial recognition and also is used to detect human body parts and estimate their poses with the right accuracy.

Steps to Draw a Point:

Click on Label type and select Point from the top left menu of the editor.
Select a category from the drop-down box.
You can label the points by left-clicking the mouse.
You can select the point by clicking on the point with a left mouse click. Then you can edit the point, by dragging the highlighted point with a left mouse click.
You can draw another point by pressing the “Esc” key twice.
Labels are saved automatically.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

2D Semantic Painting

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

1. Click on Label type and select Semantic painting from the top left menu of the editor.

2. Select a category/class from the dropdown.

3. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon) to start painting.

4. Selecting Brush/Polygon from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon”.

5. Use mouse left click to paint the image.

6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.

7. You need to save by clicking on Paint at the top panel or click on save or use Enable Autosave.

8. To erase, click on the eraser icon to erase the paint.

Note: The output file can be downloaded by clicking Export ⇾ Labels ⇾ 2D Semantic Segmentation.

Segment Anything

Segment Anything Model (SAM): A new AI model that can “cut out” any object, in any image, with a single click.

Steps to perform Segment Anything in semantic segmentation:

Click on Label type and select Semantic Painting from the top left menu of the editor.
Click on Paint and select Paint using Segment Anything.
Users can label an object by selecting either Points or Box. The more points that are added, the higher the accuracy of the output. For better accuracy, it is advisable to place more points in the center of the object.
After selecting the point/box, click on the Generate label to preview the results. The user can either accept or reject the label.
Click on Paint and select Save to save all the changes made. It is important to note that for Semantic painting, users must manually label by clicking on Save.
Users can overwrite the painted region by selecting Overwrite painted region.
Users can unselect the Paint using the Segment Anything option and make edits using brush/polygon.

Propagate Labels in Semantic Segementation

Steps to perform propagate labels in semantic segmentation:

1. Click on “Label type” and select “Semantic painting” from the top left menu of the editor.

2. Paint the complete area in the current frame.

3. On the right side, click on “Propagate paints.”

4. Select the next frame to copy the labels and click on “Copy.”

5. Move to the next frame and the copied labels will be displayed.

6. Make any necessary changes to the current frame, and repeat the above steps to copy the labels to the next frame.

2D Semantic Painting/Segmentation Output Format

Details about output format for 2D semantic segmentation paint labels

Labels for 2D semantic segmentation can be downloaded as the combination of a binary (.npy) file for each frame inside the dataset and two supporting metadata (.json), and colors (.json) files per dataset.

The JSON data present in the metadata file is in the below format :

                                      {
                                          "sensor_id": {
                                              "file_id_1": [
                                                    "paint_category_1",
                                                    "paint_category_2",
                                                    "paint_category_3",
                                              ],
                                              "file_id_2": [
                                                  "paint_category_2",
                                                    "paint_category_3",
                                              ]
                                          } 
                                     }

Here, paint_categories is a list of paint categories that are painted in that particular frame, which are configured at the dataset level.

The JSON data present in the colors file is in the below format :

                                   {
                      "format": [
                        "b",
                        "g",
                        "r"
                      ],
                      "paint_category_1": [
                        66,
                        45,
                        115
                      ],
                      "paint_category_2": [
                        70,
                        199,
                        184
                      ]
                }

Here, paint_category_1 is a list of r, g, and b values of the particular category listed in format order, for example here it's blue, green and red.

The npy file is a binary file which contains pixel-level information for the label category. For a file with image_width w pixel and image_height h pixel, npy file contains (h*w) bytes that represent the label category. This npy file will be per frame.

For example, consider a dataset containing a file (file_id_1) with an image_width of 1216 and image_height 2560 pixels, respectively. Assuming that the pixel point at image_width 1200 and image_height 2500 in the file of the dataset is annotated as below :

                                      label_category : "paint_category_2"

For the above scenario, the npy file will contain 3112960 bytes (hw). The annotation information for the point will be present at 3041200th (y * image Width + x) (25001216 + 1200) index. The byte value at this index would be 2 which is the based index of "paint_category_2" in the paint categories provided in the metadata for a particular file(file_id_1). The value 0 for the label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header

Remember the following information:

You can obtain the paint_categories and compression format from the data above.
The compression format is "pako_compressed,” and you can use pako decompression to retrieve the paint labels for each point.
For visualization purposes, you can use colors.json.
Regarding the paint categories, your understanding is correct. We assign 0 for unlabeled points and use the index from the paint_categories field in the paint labels.

3D Bounding boxes on images

User can add a 3D bounding box to the image by selecting the 3D bounding box label Type.

Steps to Draw a 3D Bounding Box on image:

Click on Label type and select the 3D bounding box from the top left menu of the editor.
Select a category from the drop-down box.
You can label the 3D bounding box by left-clicking the mouse. On mouse left click and drag user can see a 2D box which will be the front face of and another click will generate a corresponding 3D bounding box.
Hovering on the box reflects which face/vertex will be moved and you can use Shift + Left Click-drag to move the box.
You can select the 3D bounding box by clicking on the box with a left mouse click. Then you can edit the bounding box by dragging the highlighted box with a left mouse click.
Labels are saved automatically.
Users can submit their work by clicking on the submit button at the bottom. Refer to for more information.

2D ML-powered Visual Object Tracking

Overview: Given the initial state (bounding box) of a target in the first frame of a video sequence, the aim of Visual Object Tracking is to automatically obtain the states of the object in the subsequent video frames.

How is this used for making labeling easy?

2d VOT uses computer vision algorithms to predict the bounding box of an object in all frames given the initial manual box in a particular frame. The following action path is used to leverage this capability for 2d labeling.

Draw a bounding box on the required target object.
Apply 2d VOT to predict bounding box in subsequent and previous frames and draw them in the canvas.
User iterates through each frame to verify the predictions and make adjustments as necessary.

Steps to track an object:

Draw bounding on the required target object

2. Uncheck the “Fill new labels” option and click on the “Track label” button

3. Labels will be automatically visible on the timeline once Tracking is complete.

4. If tracking is not complete and the user will see a warning message stating the same, try clicking on “Track Label” again to resume progress.

Tips on using Track label

For each “Track label” click, tracking is done for the selected label from the active frame in both directions.
Predicted labels do not overwrite manual labels and stop updating frames in both directions once a manual frame is encountered in the timeline.
Recommended using for objects that are visible across many frames to avoid the overhead of re-adjusting in that many frames.
Predicted labels take into consideration the scale change of the object and not the orientation changes of the object. Hence, if an object changes its orientation drastically from the initial state, then it is recommended to re-adjust the frame at the beginning of the new orientation and apply it to track again.

2D Shortcut Keys

Following are some shortcut keys that can be used in 2D Editor. These can be found from the editor under the Help⇾Keyboard shortcuts menu or on the bottom right of the editor.

2D Customer Review

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

How to enable customer review for datasets? While creating a new dataset, turn on the Enable customer Review option. If you would like to use this feature for an existing dataset, go to the Settings screen in the dataset you would like to enable this, and turn on Enable customer review option.

How does this work? When customer review is enabled, users can submit the dataset for review once all the tasks are complete in all the pipeline stages. It is recommended for users to complete their existing tasks and resolve their pending issues across all stages before submitting them to review. Please note that once submitted, users will not be able to modify labels and tasks in the editor.

Users will not be able to access the review app when the work is in progress in any of the pipeline stages. When the review app is launched for the first time, you're greeted with a screen to set your review configuration. Users are provided with two types of review options: Sampling review, Full review.

In the sampling review, you can review a sample size of frames to review which are randomly chosen instead of all the frames like in the Full review. Once you have set the configuration, you are ready to get started!

When reviewing, if you notice any labels where re-work might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Issue Creation:

Users can create issues if they find any errors in the frame/dataset.
To create an issue, click on the Issues icon.

Click on the area and Name the issue and select priority and click on Create.
On the right side menu, all the created issues can be viewed and managed.

3D Editor

3D Editor Overview

Overview: 3D Editor is used to label 3D Bounding Boxes, Polygons, Instance & Semantic paint and 3D Lines for Point Cloud.

Primary Toolbar: It is on the left side of the image editor which has Home, Create Tool(C), Select and Move Tool (V), Select and Scale Tool(S), Select and Rotate Tool(R), Orbit Tool(1), Pan Tool(2), Dolly Tool(3) and Create Issue.

Home Button: Home button is used to navigate to the Homepage.

Create Tool: The user needs to click on Create Tool to add any new labels. Labels can't be created without selecting this option.
Select and Move: The user can label frames using this option. The user can’t label if this option is not selected.
Select and Scale Tool: Selects labels and activates a widget to scale them in 3D space.
Select and Rotate (R): Selects labels and activates a widget to rotate them in 3D space.
Orbit Tool(1): Click and drag on it to rotate the view around the lidar.
Pan Tool(2): Click and drag to pan the lidar up, down, left and right.
Dolly Tool(3): Click and drag to move the lidar forward and backwards.
Create issues: Users can create issues at frame or label level by clicking on the Create Issue icon and selecting a label in the editor from the primary toolbar.

Primary Menu bar: It is on the top of the image editor. It consists of Frame, Label, View, LiDAR, and Help.

Frame: Frame has sub-options which are Previous Frame and Next Frame.

Label: Label has options like Unselect, Hide, Lock, Delete, Fit Label to Points, Set box on ground by default, Set box on ground, Enable one click placement, Rotate, Rotate 180 in all frames, Propagate Labels(Only for Single frames) and Attributes.

View: View consists of Show Labels Text, Show Labels Attributes, Show selected Label only, Show Bounding Boxes, Show Polygons, Show Ground Mesh, Uncolor Ground Points, Show Lidar Cube, Show Grid, Enable Secondary Views, Enable Camera Views, Perspective, Orthographic, Light Theme, Dark Theme, and Reset View.

LiDAR: LiDAR consists of Enable Point Intensity, Color By Z, Show Colored Points inside Box, Show Black Points(Only for coloured lidars), Ground, Cloud Radius and Lidar FOV(Horizontal and Vertical).

Secondary Menu bar: It consists of a Label Set, Labelling Pipeline, Label category, point size, Labelling Instructions, Reset View. Additionally, for sequence annotations, we have Fused cloud and Z-threshold along with these.

Panel Content: It is on the right of image editor. The available options in panel content are Info, Labels, Issues and AI Sense. Apart from these we also have Dataset Attributes, Frame Attributes, Label Properties, Label Attributes and Image Enhancement which are used at different levels while labelling.

Assignments: Here all the files and assigned files are shown. Tasks assignments can be found here Tasks Assignments.

Info: Info displays the file properties of that particular image. If info is selected the user can view the image name, the path of the image and assignments of the particular frame. User can directly move to Dataset or Workspace Page based on selection. For example, if the user clicks on Dataset the tool navigates the user to the dataset page in the tool.

User can submit the work using the submit option at the bottom. Please refer to Task Life Cycle for more information.

Buttons in the lower navigation bar and their usage:

Previous Frame, Play, Next Frame and Frame Numbers. There is a slider to move to the next or previous frames.
Preload Data: It loads the frames in advance before opening them.

User can hide the bottom bar using the Hide bottom bar option.
This is only for sequence datasets.

3D Bounding Boxes — Single Frame/Individual Frame

Overview: Bounding box annotation is used to train autonomous vehicles to detect various objects in a real-world scenario. The objects are detected in the point cloud with high accuracy by using bounding boxes.

Steps to create a 3D bounding box:

Click on Label type and select 3D bounding box from the top left menu of the editor.
To Create a Bounding Box: Start labelling by clicking on Create Tool(C) using Left-click on the left-hand sidebar. Note that it only adds bounding boxes in this mode.
To Edit Bounding Box: Users can edit boxes only in “Select and Move Tool(V)”.
To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the empty space of the box to move the bounding box to the desired direction.
To Rotate Bounding Box: Hold “R+Left Click”. Now Click “Rotated arrows” to rotate the bounding box.
To Increase or Decrease Bounding Box: To Enable only Scale(Increase or Decrease Box Size)action, Hold “S+Left Click”. Now Click “Circle” to adjust the bounding box in different directions.
To Delete Bounding Box: Select the label and click delete(shift+delete for sequence) or click on the trash can icon of the label in the right panel.

3D Bounding Boxes_Sequence

Instructions for creating a 3D bounding box within a sequence dataset:

Click on Label type and select “3D bounding box” from the top left menu.
To create a Bounding Box: Click on Create Tool (C) from the left-hand sidebar to start labelling. Note that this mode only adds bounding boxes.
To Edit Bounding Box: Users are only able to edit boxes when using the “Select and Move Tool(V).”
To move the Bounding Boxes: To Enable only move action, Hold “V+Left Click”. Now click “Arrows” or place the pointer in the space of the box to move the bounding box to the desired direction.
To Rotate the Bounding Box: To rotate the bounding box, hold down the “R” key and click the left mouse button. Then, click on the “Rotated arrows” icon.
To Increase or Decrease Bounding Box: To enable scaling (increasing or decreasing box size) action, hold “S+Left Click”. Now click “Circle” to adjust the bounding box in various directions.
To Delete the Bounding Box: Select the label and press Shift + Delete to remove the label from the sequence, or click on the trash can icon of the label in the right panel to delete a specific key frame or a manual frame.
Refer Sequence Timeline to understand the Timeline feature in sequence labelling.

3D Bounding Boxes Features

Label View

Overview: Boxes can be adjusted in all frames at a single place.

Steps for using Label View:

In Label View the selected bounding box can be viewed and adjusted in all the preloaded frames.
The box will be displayed in Top, Front and Side view.
Select a bounding box and Click on ’Label view’ to enable it.

One-Click Bounding Box

Overview: One-click bounding box allows you to create a perfectly labeled bounding box with just one click.

Steps to create a one-click bounding box:

Once selected, place the bounding box at a dynamic object. Ex: Vehicle. Placing at concentrated points will yield better results.
Look at the secondary view to check the quality of the placement. Adjust the label accordingly if needed.

Sequence Timeline

A timeline feature is a useful tool for visualizing your labeling activity and improving labeling speed and efficiency for sequence datasets. By providing information about the labels across datasets, it allows you to easily identify, navigate, and edit manual frames. With the timeline, you can perform these operations much faster and with greater accuracy. For more detailed information on how to use the timeline, please refer to the instructions below.

Using Frame Number

To navigate across frames using the frame number, you can click on the frame number and input the desired frame. Alternatively, you can hover over the frame number and drag left or right to move to the previous or next frame.

Using Frame Slider

You can navigate across frames in the timeline view by dragging the blue slider left or right.

Using Navigation Buttons

You can navigate across frames using the play, previous, and next buttons located on the timeline view. See the button interactions below.

You can use the ctrl/cmd key with mouse scroll to zoom in or zoom out the scale present in the timeline view.

Extend/Clip Labels

To extend or clip labels to particular frames, hover on the right or left edges of the label track (coloured bars) to see a black bar. Drag this black bar in the left/right directions to extend or clip the label to some particular frames from the sequence.

Delete label in specific frame:

You can delete the interpolated label in a specific frame by holding the command (for Mac) or Ctrl (for others) and left-clicking to a specific frame of the labelling timeline.

Note: By disabling “fill new labels”, new labels created will not be interpolated.

Show Ground Mesh

Before:

Accessing Show Ground Mesh:

After:

Secondary Views

Overview: Secondary views enable the user to look at the label from three different views - Top, Front, and Side View.

To Disable secondary views, click on "view" in the top menu and Unselect Enable Secondary Views

Camera Views

Overview: The Camera View Feature allows the user to check 3D Lidar Images to view as an image on the side. This provides user clarity on each of the Lidar Images.

Hide/UnHide Points in 3D Lidar

User can Hide/Unhide the lidar points from top and bottom in the editor.

3D Lines

Overview: In the 3D coordinate system, lines can be described using vector equations or parametric equations. Lines in 3D have equations similar to lines in 2D and can be found given two points on the line.

Steps to draw lines:

Draw a line using the mouse left click.
Press the ESC key for drawing another line.
Lines can be edited only in Select and Edit Tool(V).
Use mouse left click to select a line.
User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.
Labels will be saved automatically.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

3D Polygons

Overview: Polygons are straight-sided shapes (3 or more sides), defined by three-dimensional points (vertices) and the straight lines that connect them (edges).

Steps to draw a polygon:

Click on Label type and select 3D polygon from the top left menu of the editor.
Select a category from the dropdown.
Click on Create Tool(C) to create a new Polygon.
Draw a Polygon using the mouse left click.
Press the ESC key to end the Polygon.
Polygons can be edited only in Select and Edit Tool(V).
Use the mouse left click to select a Polygon.
User can add a point by using the mouse left-click on the line or delete a point on the line by clicking on the point and pressing the Delete key on the keyboard.
Labels will be saved automatically.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

3D Semantic Segmentation/Painting

Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.

Steps to perform semantic segmentation:

To Create a Paint Label, Select a class and use left-click to paint on Lidar.
Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”.
To paint another category change category type and paint using left-click.
You can also adjust the brush size. The bigger the brush size, the more surface area it covers.
You can also view the point cloud that is painted in the camera view by clicking on the “P”.
You need to save by clicking on Paint at the top panel and then click on save or use Enable Autosave.
To erase, hold “e” or click on the eraser icon to erase the paint.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

3D Instance Segmentation/Painting

Overview: Instance Painting allowed to paint each instance of each object featured in the point cloud instead of categorizing each pixel like in semantic segmentation.

Click on Label type and select 3D Instance painting from the top left menu of the editor.
Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon or Cube) to start painting.
Selecting Brush/Polygon/Smart Brush/3D brush from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon/Smart Brush/3D brush”
Instance Painting: To Create a Paint Label, Select a class and hold C on the keyboard using left-click on the Lidar.
To create another instance of the same or another category, hold “C” while painting. It will automatically generate a new instance.
You can also adjust the brush size. The bigger the brush size, the more surface area it covers.
You need to save by click on Paint at the top panel and then click on save.
To erase, hold “e” or click on the eraser icon to erase the paint.
Users can submit their work by clicking on the submit button at the bottom. Refer to Task Life Cycle for more information.

Fused Cloud

Steps to show fused cloud:

Click on Fused cloud settings or click 'Alt+f' and select “Show Fused Cloud”.
Once the Fused cloud is selected, the accumulated points in the lidar will be shown in the current frame.
The points will be shown only for the frames which are preloaded.

Fused Tile Size: Click on Fused cloud settings and change X and Y values under Fused Tile Size for increasing or decreasing tile size or click on Edit Fused Tile and increase the size of the tile in any direction.

3D Segmentation (Smart Brush)

Overview: Smart Brush is best used to paint a large clustered point quickly.

Steps to paint with Smart Brush:

Select a category from the editor.
Selecting Smart Brush: Click on Paint at the top panel and select 'Paint with Smart Brush' or use right-click at the type selection.
Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points.
To paint another category change category type and paint using left-click.
By increasing the distance, It would increase the search radius to expand the smart brush.
By increasing the number of points, it would increase the difficulty for the cluster to grow as more number of points are required to be considered as the core point of the cluster.
You can also view the point cloud that is painted in the camera view by clicking on the “P”
You need to save by click on Paint at the top panel and then click on save.
To erase, hold “e” or click on the eraser icon to erase the paint

3D Segmentation (Polygon)

Overview: Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: Click on Paint at the top panel and select 'Paint with Polygon' or use right-click at the type selection. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

3D Segmentation (Brush)

Overview: The brush is best used to paint dynamic objects like vehicles.

Steps to paint using a brush: 1. Select a category from the editor.

2. Selecting a Brush: Click on Paint at the top panel and select 'Paint with Brush'. 3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5. To paint another category change category type and paint using left-click. 6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers. 7. Brush depth: This option lets you adjust how far the brush has to paint the points in the LiDAR. Setting a lower depth value, lets you paint points closest to the brush and a higher depth value lets you paint all the points that fall (including the closest up to the farthest) under the brush. The Default value is 5 which can be adjusted. 8. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 9. You need to save by click on Paint at the top panel and then click on save.

10. To erase, hold “e” or click on the eraser icon to erase the paint.

3D Segmentation (Ground Polygon)

Overview: Ground Polygon is best used to paint a large surface area.

Steps to create a Semantic Painting: 1. Select a category from the editor.

2. Selecting Polygon: To select the ground polygon, Use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint on Lidar. 4. Select the surface area in which you want to paint by clicking on the edges of the coverage. 5.Press Enter to close a polygon or press Esc to cancel a polygon. 6. To paint another category change category type and paint using left-click. 7. You can also view the point cloud that is painted in the camera view by clicking on the “P”. 8. You need to save by click on Paint at the top panel and then click on save or use 'Enable Auto Save'.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

3D Painting (Foreground/Background)

Overview: Background feature allows you to paint an object that is occluded

Foreground:

Click on Paint and select “Paint Foreground”
Once “Paint Foreground” is selected, the user can paint on any area of Lidar.
In the image editor, there is an option for selecting the foreground as “Paint LiDAR as Foreground”.
You need to save by click on Paint at the top panel and then click on save.

Steps to paint background:

1. Click on Paint and select “Paint Background” at the top panel 2. Once “Paint background” is selected, paint that is already labeled won’t be affected.

3. Paint Background is mostly used when there is some unpainted area in Lidar

3D Segmentation(3D Brush/Cube)

Steps to paint with Smart Brush: 1. Select a category from the editor. 2. Selecting 3D Brush/Cube: Click on Paint at the top panel and select 'Paint with 3D Brush' or use right-click at the type selection.

3. Semantic Paint: To Create Paint Label, Select a class and use left-click to paint a cluster of points. 4. To paint another category change the category type and paint using the cube. 5. Use the 'F' key to start painting. 6. The size of the cube can be adjusted.

7. You can also view the point cloud that is painted in the camera view by clicking on the “P” 8. You need to save by click on Paint at the top panel and then click on save.

9. To erase, hold “e” or click on the eraser icon to erase the paint.

Label Set

What are the label sets?

A label set is a property that is set on every label and task. Label sets help multiple users to label the same object using their label set id. This will help admins to export labels from a dataset for a particular label set and compare it with the labels from a different label set. When a dataset is created, a “default” label set id is created along with it.

How do I create a label set?

Go to the editor page
Click on the dropdown “Default”

Click on “Create new label set”

Enter the name of the new label set
Select an existing label set from “Include labels from the existing label set” if the user wants to copy labels from an existing label set to the new label set.
Clicking on “create” will create a new label set.

How do I delete a label set?

Go to the editor page.
Click on the Label set.
Click on the delete icon beside the label set name which the user wants to delete.

How do I copy labels between two label set?

Go to the editor page
Click on the Label set
Select the option “Copy labels to label set”
Select the label set from which the labels would be copied.
Select a label set from the dropdown “Select label set”
Clicking on “copy” will copy all the labels

How do I export labels from a label set?

Go to dataset export page
Select a label set from the dropdown box
Click on “Download” to export labels.

Label set Comparison:

How do I compare two label sets?

User can only compare label sets for 3d bounding box labels

Go to the editor page
Click on the Label set from the top left menu of the editor.
Select “Compare”.
Select a label set to compare with
Clicking “compare” will show labels from both the label sets in the lidar viewer and a table as shown below.

3D Shortcut Keys

Following are some shortcut keys which can be used in 3D Editor

3D Customer Review

This feature lets you choose a sample of files or all files in your dataset, and with a simple user interface for you to review the labels, accept the data and file issues if any re-work is needed.

When reviewing, if you notice any labels where rework might be needed, you can quickly create an issue for the same and click Mark for rework to mark the file for rework. If the frame looks good with all the data in it, you can click Mark as done to mark the file as done.

Once you've reviewed the samples, if the data looks good, you can accept and mark the dataset as done by clicking the Accept dataset. If you've marked any files for rework, you can request for rework by clicking Reject dataset.

Customer Review is the same for all the label types (3D bounding box, 3D line, 3d semantic painting, Instance painting and 3D polygon).

Issue Creation:

Users can create issues if they find any errors in the frame/dataset.
To create an issue, click on the Issues icon.

Click on the area or the label and name the issue. select the priority and click on Create.
On the right side menu, all the created issues can be viewed and managed.

3D input/output

JSON input format for uploading a dataset in a point cloud project.

Details on JSON format dataset for point cloud type projects

`The zip file should contain a set of JSON files and camera images. Each JSON file will correspond to one point cloud frame of the dataset. JSON files should be at the root directory level. The order of the frames is decided by the order of JSON filename sorted in ascending order. For instance, filenames can be 0001.json, 0002.json, 0003.json, ... Filenames can also be 0.json, 1.json, 2.json, …

Each JSON file should be an object with the following 5 fields:

Images
Timestamp
Points
Device position
Device heading

1. Images

An image (array) - of all camera images corresponding to one point cloud frame. Usually, the number of images corresponds to the number of cameras in the system. If there are no images, please use an empty array. Each element is a JSON object. Fields in the image object are as follows:

fx (float) - focal length in x direction.
fy (float) - focal length in the y direction.
cx (float) - x coordinate of principal point.
cy (float) - y coordinate of principal point.
timestamp (float) - time in seconds when the image was captured.
image_url (string) - corresponds to an image path inside the zip file, e.g. “images/0001.png”. It can also be an external URL. Image types supported are .jpeg and .png.
position (object) - position of the camera with respect to the world frame. Details of JSON objects can be found below.
heading (object) - orientation of the camera with respect to the world frame. Please find details of the JSON object below.
camera_model (string) - the camera model to be used for undistorting the image. Supported values for camera_model : pinhole (default) - uses k1, k2, p1, p2, k3, k4 distortion coefficients fisheye - uses k1, k2, k3, k4 distortion coefficients mod_kannala - uses k1, k2, k3, k4 distortion coefficients
k1 (float) - distortion coefficient.
k2 (float) - distortion coefficient.
p1 (float) - distortion coefficient.
p2 (float) - distortion coefficient.
k3 (float) - distortion coefficient.
k4 (float) - distortion coefficient.
camera_name - this is optional, but if given in JSON file, tool will use the same name to refer to this camera instead of using camera_0, camera_1, etc.

A sample image JSON is as follows:

2. Timestamp

Timestamp (float) – time in seconds at which the point cloud frame was captured.

3. Points

Points can be given in 2 formats-the first format is an array of JSON objects, and the second format is base64 encoded strings of points and intensities. Points in JSON object array format: A Points array of JSON objects of all LiDAR points having their x, y, z, i, r, g, b,d values. x, y and z values are mandatory and i, r, g, b and d values are optional for each point. In general, the “up” direction towards the sky should be in the positive z direction for the visualization to work correctly. Each element of the array is a JSON object, as shown in this section. rgb value in xyzrgb type point will be supported in a future release. Each point can have other values like velocity, as well, for which we can add custom support. Fields in point object are as follows:

x (float) – x coordinate of the point, in meters.
y (float) – y coordinate of the point, in meters.
z (float) – z coordinate of the point, in meters.
i (float) - intensity value between 0 and 1, this is an optional field
d (integer) - non-negative device id to represent points from multiple sensors, this is an optional field

x, y and z values are in world coordinates. If you are unable to put the point cloud in the world coordinate, you can fall back to the local LiDAR coordinate and let us know. We will contact you about the issue.

For Multi Lidar points, add the field 'd' in the points array to represent lidar id, it should be a non-negative integer value. A sample point JSON object is as follows:

If you want to add a name for each lidar id, then you need to add another field “multi_lidar_keys”, please note this is an optional field.

4. Device position

A device_position (object) – position of LiDAR or camera with respect to world frame. Similar to the point cloud, if you are unable to put the device position in the world coordinate, you can fall back to the local LiDAR coordinate and let us know. We will contact you about the issue. For camera, if you do not have any position information, please use (0, 0, 0) and let us know. Fields in position object are as follows:

x (float) – x coordinate of device/camera position, in meters.
y (float) – y coordinate of device/camera position, in meters.
z (float) – z coordinate of device/camera position, in meters.

Sample position JSON object:

5. Device heading

A device_heading (object) – orientation parameters of LiDAR or camera with respect to world frame. If you are unable to put LiDAR heading in world coordinate, please use the identity quaternion (x = 0, y = 0, z = 0, w = 1). If you can not obtain extrinsic camera calibration parameters, please also use the identity quaternion. We will contact you about this issue. Fields in the heading object are as follows, the 4 components are quaternions:

x (float) – x component of device/camera orientation.
y (float) – y component of device/camera orientation.
z (float) – z component of device/camera orientation.
w (float) – w component of device/camera orientation.

A sample heading JSON object is as follows:

Please note that in JSON, the order of the dictionary values of quaternion doesn't matter. Following two JSONs will give exact same result:

A sample JSON can be found below,

Another sample (Multi Lidar):

How to convert ROS bag into JSON data for annotation

How to convert ROS bag into JSON data for 3D annotation

Annotation data is often collected by ROS and stored as ROS bags. In order to use deepen.ai’s system for annotation, the data must be first converted into the JSON format:

This JSON file specifies the exact data for annotation. It includes point clouds, images, timestamps, intrinsic and extrinsic calibrations, as well as localization information.

Usually, each user writes a script to convert their bag data into the JSON format. We cannot use a common script for conversion for several reasons:

Each ROS bag may have many topics. The user needs to specify the exact topics for annotation.
Some information such as intrinsic and extrinsic calibrations may not be in the ROS bags.
If a special camera model is used, the user may wish to only send the rectified images for annotation.
The JSON format links the point cloud and images shown to the annotators. It does not sync using timestamps like RVIZ.
If accurate localization is available, the point cloud is transformed into a fixed world coordinate.
Advanced processing such as LiDAR ego-motion compensation is typically performed in the script.

Users not familiar with the JSON format may find the conversion script difficult to develop. At deepen, we have worked with many clients on their data conversion scripts. Although each script is different, we have found many common themes in these scripts. If you are developing a new script, this tutorial will help you to get started and get your data ready for annotation. ROS has good Python support. Most conversion scripts are written in Python, so this tutorial also assumes that. We will walk you through the various steps of the conversion script. We will try to describe the simplest processing. There are many advanced techniques which can improve the data, but we will not cover them here.

Reading the Bag file(s)

The first step is very obvious. You need to specify the path of the files and ROS topics to annotate. The topics should include point clouds and images. Images are very important for annotation speed and accuracy.

Synchronizing the point clouds and images

In the 3D annotation tool, the annotator is shown a point cloud and one image from each camera. Thus, the JSON format specifies the connection between the point cloud and images. To calculate this, our script needs to explicitly synchronize the data. Often, the synchronization is done by ROS timestamps. Let us assume that there is only one LiDAR. Thus, there is only a single point cloud sequence. In this stage, we make a single pass through the point cloud data and collect its timestamps. This is our primary timestamp sequence.

We then make a pass through each image topic that we can show to the annotators. As we go through the images, we find the closest timestamp to each LiDAR timestamp. This will be the image attached to each point cloud. Thus, there is one image from each camera for each point cloud. The image sequence is synchronized to the LiDAR sequence.

Temporal Calibration (Advanced Topic)

Technically, each timestamp occurs when the LiDAR finishes a spin or a camera closes its shutter and ROS records its data. The sensors, network, and ROS itself all add latency to this process. Thus, the timestamps usually occur a few milliseconds after the actual events. This latency is typically different for each sensor and introduces inconsistency in timestamps. Temporal calibration is the method to adjust the timestamps, so all sensors have consistent timestamps. We will not cover it here, and you may skip it for your initial conversion script.

Multi-LiDAR

In the case when we have multiple LiDARs that we need to annotate, we can pick one LiDAR as the primary one and synchronize all other LiDARs to it. Note that there is the “d” field for each point in deepen’s JSON format, to which we can assign the LiDAR ID. If you use the “d” field, our annotation UI allows you to distinguish between point clouds from different LiDARs.

Localization

The JSON format specifies all points in a single world coordinate because it would make annotation much easier and more accurate: All static objects would be in a fixed location. All dynamic objects would have their simple motions, without complications from the motion of the ego vehicle itself. In order to achieve this, we would need accurate localization data. The best localization usually comes from an expensive INS/GNSS. Otherwise, LiDAR SLAM combined with other sensors such as IMU and odometers can also give accurate localization in most situations.

If an accurate localization is available as a ROS topic, we just need to find the LiDAR pose with a timestamp closest to that of the LiDAR point cloud. We can use the pose to transform the point cloud into the world coordinate. Note that accurate LiDAR poses require LiDAR to INS calibration, which we will not cover here.

If localization is unavailable, we suggest that you try one of the LiDAR SLAM algorithms. If you skip this step and use the LiDAR point cloud as-is, it is still possible to annotate the data, but the cost and accuracy would both suffer.

Ego Motion Compensation (Advanced topic)

Most LiDARs have a slow “rolling shutter” where each scan or spin can take tens or hundreds of milliseconds. During this time interval, the LiDAR itself may be going through a complicated motion. If we treat the LiDAR as stationary for the entire time interval, the point cloud would be inaccurate. Ego motion compensation is the technique to solve this problem, but we will not cover it here. You can ignore this issue unless your vehicle was moving at a high speed such as on a highway.

Intrinsic and Extrinsic Calibrations

Accurate intrinsic calibration is required for each camera. We support the common plumb-bob (Brown & Conrady) and equidistant fisheye (Kannaly & Brandt) camera models. If you choose a model we don’t currently support, you can submit rectified images instead.

Extrinsic calibration is for specifying the camera pose for each image. If localization and ROS tf transforms are available, you just need to obtain the camera pose at the timestamp of the image. Note that we are using the camera coordinate system in OpenCV. It is identical to the ROS optical coordinate system, not the standard ROS coordinate system.

If the tf transforms cannot give you correct camera poses, but you have the extrinsic LiDAR-camera calibration, you can apply the extrinsic calibration to obtain the camera pose from the LiDAR pose. It is just a simple matrix multiplication. Since the point cloud and image usually have different timestamps, it would be more accurate to interpolate the LiDAR or Camera poses, but we will skip it for this tutorial.

Output

After obtaining the information above, we just need to make a pass through the point clouds and images to output the JSON and image files. Note that the JSON format supports BASE64 encoding, which can make your JSON files much smaller and the script run much faster.

Debugging

Debugging your first conversion script often takes a long time. We have some tips to find the most common bugs.

The first tip to debug is to output a small number of frames. Your script will finish quickly. You can upload this subset into deepen’s tool quickly and visualize the data.

Localization

To verify your localization accuracy, the easiest way is to load a moving point cloud sequence into deepen’s annotation tool. You can use the “fuse” option to accumulate multiple point clouds together. If the static objects are sharp in the fused point cloud, you are likely to have the correct localization. If not, there are many common mistakes:

Low quality or misconfigured INS/GNSS
Wrong usage of INS output
Wrong coordinate frame is used
Bad output from LiDAR SLAM algorithm

Calibration

To validate your camera calibrations, you can load a sequence into the annotation tool. Add a 3D box to an object. If the projection of the box on the image looks correct, you probably have the correct calibrations. Otherwise, there are many common mistakes:

Giving rectified images and also non-zero distortion coefficients
Using the ROS coordinate system instead of the optical coordinate system
Scaled the images but did not change the intrinsic parameters
The pose of the camera may be off due to localization error.

Synchronization

In order to validate the synchronization of your data, you can load a sequence into the annotation tool. Add a box to a moving object. If the projections on all camera images look correct, you should have a proper synchronization between point cloud and camera. For a high-precision check, the object should have a high angular velocity relative to the sensors. Therefore, you can use an object moving quickly from left to right or use a sequence where the ego vehicle is turning rapidly. If you found a synchronization error, there are several common mistakes:

For big errors, there is likely to be some bugs in the synchronization logic in the script.
For small errors, temporal calibration may be needed.
For very small errors, we may need to consider the exact time the LiDAR scans the object which is different from the timestamp we assigned to the entire point cloud. This is an advanced topic.

Visualization code

Sample Code

We will release a sample conversion script soon.

Please contact us if you run into problems with your conversion script. We will gladly help you debug your script.

Data Output Format - 3D Semantic Segmentation

Details about output format for semantic segmentation labels

Labels for 3D semantic segmentation can be downloaded as the combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in the metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The DPN file is a binary file which contains point level information for label category and label id. For a dataset with n number of points, dpn file contains n bytes that represent the label category.

For example, consider a dataset containing 3 files each with 10,000 points, 11,000 points and 9000 points respectively. Assuming that the point with index 50 in second file of the dataset is annotated as below :

label_category : "paint_category_2"

For the above scenario, dpn file will contain 30000 bytes (10,000 + 11,000 + 9000). The annotation information for the point will be present at 10,050th (10,000 + 50) index. The byte value at this index would be 2 which is the 1 based index of "paint_category_2" in the paint categories provided in the metadata. The value 0 for label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header:

< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": ["Drivable region", "Uneven terrain", "Hard Vegetation", "Soft vegetation", "Construction object", "Other Traversable Static", "Ghost", "Static Object", "car", "Vehicle _ Truck", "Pedestrian with Object _ Adult", "Pedestrian _ Adult", "Vehicle _ Trailer", "Construction Object _ Construction Cones and Poles", "Ghost _ Sensor Artifact", "Vehicle _ Van", "Vehicle _ Car", "Ego Car", "Ground", "dynamic_buffer"]}

You can get the paint_categories and compression format from the above data. Here the compression format is pako_compressed and you can use pako decompression to get the paint labels for each point.

For the paint categories, your understanding is correct. We reserve 0 for unlabelled points and use the index from the paint_categories field in the paint labels.

Data Output Format - 3D Instance Segmentation

Details about output format for instance segmentation labels

Labels for 3D instance segmentation can be downloaded as combination of a binary (.dpn) file and a supporting metadata (.json) file per dataset.

The json data present in metadata file is in the below format : { "paint_categories": [ "paint_category_1", "paint_category_2", "paint_category_13, ] }

Here, paint_categories is a list of paint categories configured for the project the dataset belongs to.

The dpn file is a binary file which contains point level information for label cateogry and label id. For a dataset with n number of points, first n bytes of dpn file represent the label category, the next n bytes represent the lower bytes of the label id and the subsequent n bytes represent the higher bytes of the label id the point belongs to.

label_category : "paint_category_2" label_id : 1024

For the above scenario, dpn file will contain 90000 bytes ((10,000 + 11,000 + 9000) * 3). The annotation information for the point will be present at 10,050th (10,000 + 50), 40,050th (30,000 + 10,050) and 70,050th (60,000 + 10,050) indexes. The values of bytes at these indexes are as below:

10050 : 2 (1 based index of "paint_category_2" in metadata) 40050 : 0 (1024 % (2^8)) 70050 : 4 (1024 / (2^8)) The value 0 for label category is reserved for unpainted points.

Quality Assurance

Issue Creation

Overview: Issue Creation allows you to create an issue for a label, notify the annotator about the issue, and then discuss the issue with our comment feature.

Steps to create an issue:

You can create an issue using by clicking on the "create issue" icon in the left panel on the labeling screen.

2. Once you've created the issue using the "create issue" feature. You can go into the QC view by clicking on "issues" in the right panel.

3. If the current stage is not the first stage, then the assignee email is the email to which this task is assigned in the previous stage.

Calibration

A complete guide for calibration. At the moment, our calibration tool only supports the Google Chrome browser.

Camera Intrinsic calibration:

Get precise intrinsic parameters for your camera using the checkerboard technique in just a few steps.

Data collection guide

Tool usage guide

Note: Camera intrinsic calibration can greatly affect the quality of other extrinsic calibrations. Please obtain a high-quality intrinsic calibration before performing extrinsic calibration.

LiDAR-Camera Calibration:

Calibrate your LiDAR-Camera set-up with a simple flow built using our world-class algorithms.

Data collection guide

Tool usage guide

Vehicle-Camera Calibration:

Calibrate your vehicle and camera set up in just a few minutes using our advanced algorithms.

Calibrate your vehicle and LiDAR set up in just a few minutes using our advanced algorithms.

Data collection guide

Tool usage guide

Optimise the calibration parameters of multiple sensors which form a closed loop.

Tool usage guide

Calibration Profile:

Guide to edit/delete/update a calibration profile

Tool usage guide

General calibration options:

Usage guide for the other options in the tool.

Calibration Toolbar options

Calibration List page options

Calibration FAQ: Calibration FAQ

Charuco Dictionary

Explaining the different charuco dictionaries

We Support 7 different dictionaries. 4x4, 5x5, 6x6 and 7x7 can have a maximum of 1000 markers, The original dictionary supports upto 1024 markers, apriltag_36h11 can have upto 587 markers, and apriltag_25h9 can have upto 35 markers.

Original
4X4
5x5
6x6
7x7
apriltag_36h11
apriltag_25h9

How to choose the bits of charuco dictionary?

The more the number of markers/corners, the better the accuracy of your calibration, but your camera should be able to identify your board correctly . So consider the camera's resolution, the target's distance from the camera, and the environment around you while choosing the dictionary to get a clear image and more corners.

while having multiple target avoid using 4x4 as it has been observed to provide ghost points due to its relatively simple pattern.

Calibration FAQ

What are the Recommended Browsers? Chrome (preferably the latest or at least version 97+)
What is the required Screen Resolution? Recommended 1920x1080 pixels or more (at least 1366x768 pixels)
What is the list of domains which need to be whitelisted? Domain Whitelist: The following domains and any requests from its child URLs have to be whitelisted: https://tools.calibrate.deepen.ai https://storage.googleapis.com/deepenai-prod-rannotateclientdata/ https://storage.googleapis.com/deepen_calibrate_assets https://www.googleapis.com https://fonts.googleapis.com https://use.fontawesome.com https://fonts.gstatic.com https://help.deepen.ai/documentation/

Data Collection for Multi Target Lidar-Camera Calibration

Calibration Target

Charucoboard is the calibration target. Charucoboard of the following dictionaries of any size and squares can be used as the target. Targets should be as flat as possible and the surface of the target should be anti-glare (matte finish)

Click here for supported charuco dictionaries.

Note: When multiple targets are used, make sure targets are of different dictionaries.

We suggest that users make at least 1.0m*1.0m charcoboards for good corner detection.

Data for camera-to-lidar extrinsic calibration

Place the targets in the focus of the camera and the LiDAR's field of view. The targets should be placed on the ground or on a stand. Avoid holding the targets with your hands.

The targets and all sensors should be static while collecting the data. To avoid time-synchronization problems, please keep the boards and the sensors stationary for at least 10 seconds while collecting each calibration data set.

For example, these are the steps to collect one set of calibration data:

Orient the camera toward the targets. Start recording. Wait for 10 seconds (Don't move/rotate your car/robot/sensors). Stop recording. You must have a recording of images and lidar data for 10 seconds. Extract one image from the camera and one frame of lidar data captured 5 seconds after the recording has started (e.g., if you start recording at 3:00:00, you stop recording at 3:00:10. We need a frame captured at 3:00:05) and save them.
Change the targets' location and orientation. Start recording. Wait 10 seconds (Don't move/rotate your car/robot). Stop recording. Again, you must have a recording of images and lidar data for 10 seconds. Extract one image from the camera and one frame of lidar data captured 5 seconds after the recording starts and save them. (Dataset front_2)
In the same manner, save Dataset front_3, front_4, and front_5.

Multi-target LiDAR Camera calibration works even with a single pair of images and a lidar frame. The more pairs, the better the accuracy will be.

Optional board position

on ground

The target is placed on the ground (touching the ground). In such a case, enable the on ground flag in the configuration page. Also, make sure that the lidar data captures the ground points. By doing so, we can optimize the extrinsic parameters using the ground points.

Tilted

A holder holds the target up in the air and tilts right by around 45 degrees. In such a case, enable the Tilted flag in the target configuration. This approach enables deep optimization, and the extrinsic parameters are further optimized using the edge points of the board.