YMIR: A Rapid Data-centric Development Platform for Vision Applications

This paper introduces YMIR¹¹1YMIR denotes You Mine In Recursion. The project is available at: http://www.viesc.com. Code is available at: https://github.com/industryessentials/ymir., a data-centric ML development platform that aims to support developing vision applications at scale.

YMIR uses data labeling, model training, and active learning based data mining as primitive operations to build datasets and models. It embeds version control into the iterative process to track and manage data annotations and models, much like code version control in software engineering. The platform also introduces the project concept, to isolate different jobs and enable parallel development of datasets for multiple applications.

In terms of system design, YMIR treats dataset and model as the key inputs and outputs of the primitive operations, and stores the specification of each operation into metadata. This allows a decoupled implementation of each operation, which has the following advantages: 1) It allows YMIR to flexibly integrate the best third party tools readily available in the open source community; 2) It reduces the adoption cost for experienced teams, as they can quickly integrate their existing tools into the platform.

YMIR adopts existing version control tools to manage datasets and models, but manages the actual data payload outside the tool to better support frequent export of images to operators as well as frequent updates to the annotations in the dataset development process.

To summarize, the highlights of the YMIR platform are as follows:

1. 1.

   The platform prioritizes data instead of model development in its product design and system architecture.

2. 2.

   The platform adopts a modular open design approach to easily integrate third party machine learning tools, and to leverage the contribution of the open source community.

3. 3.

   The platform embeds version control of data and model into the system to track the development process.

In the following sections, we will analyze one of the use cases to point out the necessity of prioritizing the dataset development in ML development process, introduce the details of the platform in the aspects of product design and system design, show the key product functions and discuss potential future works.

Computer vision technology is becoming ready to be used in many industries. For example, object detection alone can help a manufacturing factory to enhance its digital transformation process in many aspects: detect if any worker in the factory is not properly wearing protection gears; check whether all the tools are back into cabinets after a work shift; if the evacuation routes are unintentionally blocked, etc.

Although from many different application verticals, many potential applications share the following traits: the problems can be cast into standard computer vision tasks; the customer has positive image examples, but these are usually rare events buried in their data storage with no annotations; customers do not have a clear cut on the scope of the interested events, so algorithm developing experts need to iterate with end users on task definition through a trial and error process; these applications need to be developed at low cost, as the users are usually hesitate to heavily invest in yet to be verified new technologies.

Due to the challenges described above, a matured tool to semi-automate the active learning based data development process is the key to a viable solution, making project opportunities into real applications. Ultimately, the dataset shall even be curated by the end user directly, as they are the best party to define the task through a labeled dataset. These cases motivate us to develop the YMIR platform, which is useful for developing these applications at scale.

Fig. 2 is an overview of the system architecture. YMIR puts management of system state, i.e. datasets and models as the foundation, to support efficient storage and fast retrieval of models, data and versioned data annotations. The application layer is built on top of this foundation, to support flexible expansions of the platform’s functions, such as the primitive operations, as well as user and project management.

Fig.3 shows the landing page of the YMIR system. Upon login, users have an overview of the datasets, models and ongoing tasks so that users can easily resume working from where it was left off. In the rest of this section, we will briefly list the main functions available in YMIR GUI, as well as the main features inside each function.

Dataset Importing. As is shown in Fig.4, YMIR provides three ways to import a dataset: upload dataset, copy from an existing public dataset, and import from an URI. Before importing, the external dataset should be packaged following the PASCAl VOC [4] dataset format. Also, the administrator of the system can import datasets and set them as public. In this way, a user can easily replicate a dataset by having a shallow copy of it, reducing the time and work needed for packing and importing a dataset. If an user choose to import by URI, the URI can be either a local path, a networked folder address or a http address.

Data Annotations. After data is imported to the system, users can create annotation tasks through YMIR. YMIR then sends the task to Label Studio, monitors its status, and retrieves labeling results from Label Studio after the annotation task is complete. Although not required, YMIR recommends users to provide a task description file, giving examples of what are the right and wrong labels. A well written document can greatly help raters improve their annotation quality, which significantly improves the accuracy of the resulting trained model. Besides a task description file, users can also specify data annotators through YMIR. This additional feature is based on findings that restricting similar labeling tasks to the same group of raters usually makes the annotation quality much more consistent.

Model Training. Fig.6 shows the interface for creating a model training task. A task can be created by specifying a task name, the used datasets, as well as the training targets. Note that only object detection task is supported as of the current release. More types of training tasks and more models for each task can be added by both the YMIR developing team and the third party contributors. The UI displays many training related hyper parameters and their default values, which gives the user the flexibility to adjust these details and compare results from different parameter configurations. Once a model is trained, user can see the accuracy metrics on the task result page.

Data Mining. Compared with some existing AI development platforms, data mining is a differentiating feature of YMIR system, and it is the key step in a dataset and model iteration loop. Similar to the model training task, YMIR currently only provides one method for data mining, but exposes many detailed parameters so that interested users can tune the algorithm and compare its results.

Model Validation. For any models trained in YMIR system, YMIR provides a model validation feature that lets users test a trained model on the fly. Through the interface, users can upload an image and check the detection results on that image in seconds. Fig.8 shows an example detection result of an image on a trained detection model.

Model and Dataset History The model and dataset history page (Fig.9) shows the dependency relations between datasets and models, as models can be used to mine new datasets, and new datsets can then be used to train new models. It provides a story-line view of the entire model and dataset development process, which may provide clues on discovering the good and bad practices in the ML development process.

This paper presents a new platform to support efficient development of computer vision applications at scale. Before made open source, the platform was adopted by pilot users, who reported significant productivity boost compared to using standard manual process.

Although not fully utilized yet, the development histories stored in this platform may further serve as training data to optimize the productivity of users. If widely adopted, sharing and exchange of these records may further help the community to find even better ways to solve data centric AI problems.