Segment Anything Model (SAM): A new AI model that can “cut out” any object, in any image, with a single click.
Steps to perform Segment Anything in semantic segmentation:
Click on Label type and select Semantic Painting from the top left menu of the editor.
Click on Paint and select Paint using Segment Anything.
Users can label an object by selecting either Points or Box. The more points that are added, the higher the accuracy of the output. For better accuracy, it is advisable to place more points in the center of the object.
After selecting the point/box, click on the Generate label to preview the results. The user can either accept or reject the label.
Click on Paint and select Save to save all the changes made. It is important to note that for Semantic painting, users must manually label by clicking on Save.
Users can overwrite the painted region by selecting Overwrite painted region.
Users can unselect the Paint using the Segment Anything option and make edits using brush/polygon.
Steps to perform propagate labels in semantic segmentation:
1. Click on “Label type” and select “Semantic painting” from the top left menu of the editor.
2. Paint the complete area in the current frame.
3. On the right side, click on “Propagate paints.”
4. Select the next frame to copy the labels and click on “Copy.”
5. Move to the next frame and the copied labels will be displayed.
6. Make any necessary changes to the current frame, and repeat the above steps to copy the labels to the next frame.
Overview: Semantic segmentation allows you to label each pixel of an image with a corresponding class of what is being represented.
Steps to perform semantic segmentation:
1. Click on Label type and select Semantic painting from the top left menu of the editor.
2. Select a category/class from the dropdown.
3. Paint Selection: Click on Paint with Brush using Left-click on the left-hand sidebar. Use Right-Click on the mouse for selecting Painting Mode (Brush or Polygon) to start painting.
4. Selecting Brush/Polygon from Top Panel: Click on Paint at the top panel and select “Paint with Brush/Polygon”.
5. Use mouse left click to paint the image.
6. You can also adjust the brush size. The bigger the brush size, the more surface area it covers.
7. You need to save by clicking on Paint at the top panel or click on save or use Enable Autosave.
8. To erase, click on the eraser icon to erase the paint.
Details about output format for 2D semantic segmentation paint labels
Labels for 2D semantic segmentation can be downloaded as the combination of a binary (.npy) file for each frame inside the dataset and two supporting metadata (.json), and colors (.json) files per dataset.
The JSON data present in the metadata file is in the below format :
Here, paint_categories is a list of paint categories that are painted in that particular frame, which are configured at the dataset level.
The JSON data present in the colors file is in the below format :
Here, paint_category_1 is a list of r, g, and b values of the particular category listed in format order, for example here it's blue, green and red.
The npy file is a binary file which contains pixel-level information for the label category. For a file with image_width w pixel and image_height h pixel, npy file contains (h*w) bytes that represent the label category. This npy file will be per frame.
For example, consider a dataset containing a file (file_id_1) with an image_width of 1216 and image_height 2560 pixels, respectively. Assuming that the pixel point at image_width 1200 and image_height 2500 in the file of the dataset is annotated as below :
For the above scenario, the npy file will contain 3112960 bytes (hw). The annotation information for the point will be present at 3041200th (y * image Width + x) (25001216 + 1200) index. The byte value at this index would be 2 which is the based index of "paint_category_2" in the paint categories provided in the metadata for a particular file(file_id_1). The value 0 for the label category is reserved for unpainted points.
To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header
< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": [“Drivable region”, “Uneven terrain”]}
Remember the following information:
You can obtain the paint_categories and compression format from the data above.
The compression format is "pako_compressed,” and you can use pako decompression to retrieve the paint labels for each point.
For visualization purposes, you can use colors.json.
Regarding the paint categories, your understanding is correct. We assign 0 for unlabeled points and use the index from the paint_categories field in the paint labels.
