Finish your CVAT project in 80% of the time it would actually take you



Summary

We ran the following experiment : We took an experienced CVAT Annotator and we have asked him to use CVAT and TrainingSet.AI free annotation tools. The results will help you to save time, money and have a healthier annotator team.


Introduction

The goal of semantic image segmentation is to label each pixel of an image wtith a corresponding class of what is being represented. We are going to take a look at Trainingset.AI’s Semantic Segmentation Editor (from now on, TSSE) and how it compares to one of the best tools out there, CVAT, from an experienced annotators perspective in two key aspects: speed and quality.

For the speed and quality comparison we haven’t used any of the advanced features of Trainingset.AI like: automatic annotations, magic tool, contour detection, etc. The experiment was to run over a group of street images, making the best effort with both tools to get the fastest and quality results. Reynald, the expert annotator, already has experience with CVAT annotation tools, but not with TrainingSet.AI tools. On fig 1 you can see an example of an image annotated using CVAT and on fig 2 an example of an image annotated using TSSE.


Trulli
Fig 1 - Image annotated using CVAT
Trulli
Fig 2 - image annotated using TSSE



Results


On average the street image segmentation took 30.5 minutes using Trainingset.AI tools and 35.75 minutes using CVAT tool.

Trulli

Further works will show a theoretical comparison between the tools.

The key factors to a good annotation tool are:

  • QA process
  • A good mix of algorithms and AI which help the annotator
  • Intuitive, fast and easy workflow

Speed CVAT TSSE
Automatic annotations X X
Label from selection X X
Image filters X
Magic tool X
Undo/redo X X
Class shortcuts X
Load next image without reloading the window X X
Continuous polygon tool X

Automatic annotations

CVAT and TSSE offer Mask RCNN, Faster RCNN and YOLO v3 among others.


Label from selection

CVAT uses DEXTR technique to segmentate an object from a selected area while TSSE uses a non and proprietary model which aims to find the contours from the selection and then create a segmented object. TSSE’s approach seems to be working better in most cases. Besides the detection quality, objects created by CVAT also have higher point density then those created by TSSE which makes correcting the object shape after detection tedious.


Image filters

Although CVAT doesn’t support it, TSSE offers fine control of image brightness, contrast, hue, saturation, lightness, gamma and rescale which helps in use of it’s magic tool and visually helps the annotator.


Magic tool

This tool is very useful for creating segmented instances of objects with sharp edges (e.g. sky, lane, marking). It can be manually tuned with color threshold and blur radius to adjust the algorithm to perfectly fit to an object. It’s quick and requires no calls to the AI model.


Class shortcuts

Class shortcuts are available inside TSSE. Shortcut is shift+{first class letter} (e.g. shift + c for car).


Load next image without reloading the window

This feature enables cloud software to run with the speed of running it on premises. Annotator will spend most of his time annotating an image and will only reload the window while navigating to the next task. Navigating to the next task quickly is important, editors are very heavy software and reloading the whole window takes a lot of time. This feature is especially useful for datasets where each image is segmented in a short time frame and navigation to the next task takes up a lot of annotation time.


Continuous polygon tool

TSSE offers a continuous polygon tool which enables you to draw a contour of an object like you are using a pen. After you have created a polygon you can reduce the number of polygons without losing the shape using an advanced algorithm. A slider is available which allows you to control the reduction intensity.


Conclusion

In this small experiment a conflict of interest is present, since it was done by ourselves, but we encourage you to take a look and do it yourself and find your own conclusions. Also a similar improvement on bounding, line, polygon, point, etc could be expected using TrainingSet.AI Platform

Contact Us

Any question you have, don't hesitate to contact us, we'll be in touch with you


footer-frame