Assisted 2d Segmentation

Overview: Users can now perform instance segmentation tasks in polygon mode for single-frame 2D image datasets with AI assistance. In order to use the feature, the user needs to follow the steps below.

Creating the Segmentation Mask

• Go to Polygon mode

• Choose the “2D Segmentation” button

• Draw a tight box enclosing the object to be segmented

• AI server will return an output polygon mask covering the object

Refining the Segmentation Mask

Users have an option to edit/refine the masks using the pre-existing tools in the polygon labelling mode or they can use the AI assistance to refine the polygons automatically. To use AI assistance, the user needs to follow the steps below:

• To activate AI refinement mode, select the label which was created using the “2D Segmentation” button.

• Choose the “Refine Polygons” button (pencil) besides the “2D Segmentation” button.

• In the refinement mode, the user can plot two kinds of points as below.

  • Green indicates the regions which the user wants the model to include in the segmentation mask.

  • Red indicates the regions which the user wants the model to exclude from the segmentation mask.

• The user can plot Green points on the image using “Left Mouse Click”.

• Points To add the Red points, the user can use the “Left Mouse Click” with “Alt” hotkey.

• As soon as any of the above points are added, an API call will be made to refine the points.

• The segmentation will be refined using the input provided and the polygon will be updated accordingly. The refinement mode will be deactivated once the polygon is refined. You need to activate the refinement mode again using the “Refine Polygons” button in the secondary menu bar to make further edits.

• To add multiple points at once before refining the polygon, the user can use the "Shift" hotkey along with the process described above. The last point in this scenario needs to be added with “Shift” hotkey released for the polygon to be refined.

• The user can delete existing points using “Ctrl” + “Shift” hotkeys along with "Left Mouse Click" on the point to be deleted.