2D Semantic Painting/Segmentation Output Format

Labels for 2D semantic segmentation can be downloaded as the combination of a binary (.npy) file for each frame inside the dataset and two supporting metadata (.json), and colors (.json) files per dataset.

The JSON data present in the metadata file is in the below format :

                                      {
                                          "sensor_id": {
                                              "file_id_1": [
                                                    "paint_category_1",
                                                    "paint_category_2",
                                                    "paint_category_3",
                                              ],
                                              "file_id_2": [
                                                  "paint_category_2",
                                                    "paint_category_3",
                                              ]
                                          } 
                                     }

Here, paint_categories is a list of paint categories that are painted in that particular frame, which are configured at the dataset level.

The JSON data present in the colors file is in the below format :

                                   {
                      "format": [
                        "b",
                        "g",
                        "r"
                      ],
                      "paint_category_1": [
                        66,
                        45,
                        115
                      ],
                      "paint_category_2": [
                        70,
                        199,
                        184
                      ]
                }

Here, paint_category_1 is a list of r, g, and b values of the particular category listed in format order, for example here it's blue, green and red.

The npy file is a binary file which contains pixel-level information for the label category. For a file with image_width w pixel and image_height h pixel, npy file contains (h*w) bytes that represent the label category. This npy file will be per frame.

For example, consider a dataset containing a file (file_id_1) with an image_width of 1216 and image_height 2560 pixels, respectively. Assuming that the pixel point at image_width 1200 and image_height 2500 in the file of the dataset is annotated as below :

                                      label_category : "paint_category_2"

For the above scenario, the npy file will contain 3112960 bytes (hw). The annotation information for the point will be present at 3041200th (y * image Width + x) (25001216 + 1200) index. The byte value at this index would be 2 which is the based index of "paint_category_2" in the paint categories provided in the metadata for a particular file(file_id_1). The value 0 for the label category is reserved for unpainted points.

To extract the metadata and compression details, you will need to look at the response headers. Below is an example of response header

< HTTP/2 200 < server: gunicorn/20.0.4 < date: Fri, 11 Dec 2020 15:12:39 GMT < content-type: text/html; charset=utf-8 < paint-metadata: {"format": "pako_compressed", "paint_categories": [“Drivable region”, “Uneven terrain”]}

Remember the following information:

• You can obtain the paint_categories and compression format from the data above.

• The compression format is "pako_compressed,” and you can use pako decompression to retrieve the paint labels for each point.

• For visualization purposes, you can use colors.json.

• Regarding the paint categories, your understanding is correct. We assign 0 for unlabeled points and use the index from the paint_categories field in the paint labels.