ConsInstancy: Learning Instance Representations for Semi-Supervised Panoptic Segmentation of Concrete Aggregate Particles

A very common representation of instances in computer vision applications are bounding boxes FasterRCNN , typically represented as rectangular and axis aligned boxes, enclosing the instance. In MaskRCNN , bounding boxes are enriched by an instance mask, delivering a pixelwise encoding of those pixels which are associated to the instance and, which are not. However, the approaches MaskRCNN ; FasterRCNN allow bounding boxes as well as the segmentation masks of different instances to overlap, rendering them unsuitable for the task of panoptic segmentation, in which it is necessary to assign every pixel to only one unique instance.

In CentroidNet2019 , instances are represented by two-dimensional vectors associated to each pixel and pointing to the nearest centroid of an object. While this representation allows to determine object centroids and consequently, enables locating and counting instances, it does not provide an instance-wise segmentation of the input. To this end, the authors of CentroidNetV22021 enrich the centroid representation by an additional pixel-wise representation of vectors pointing to the nearest object boundary. Similarly, the authors of Xie2021 propose a polar mask as instance representation, which defines each instance by the centre point of the object. In order to obtain the outline of the instance, a number of rays, sampled in uniformly distributed angular intervals, encode the distance to the closest boundary pixel along the respective ray. Likewise, in Schmidt2018 ; Weigert2020 , a star-convex polygon is proposed to provide an instance representation. There, each pixel belonging to an object instance is allocated to the distances to the object’s boundary along a set of predefined equidistant radial directions. In these representations, the instance boundaries are implicitly contained as polygonal shapes defined by the radial rays and the associated length of each ray. Similarly, by proposing a complementary distance transform map in this paper, we also employ an implicit encoding of the instance boundaries, however, providing a much simpler representation compared to the polar mask and the star-convex polygon. In Bai2017 , the authors propose to predict a deep watershed transform, which corresponds to a discretised distance transform map, for the task of instance segmentation. Furthermore, as an intermediate representation, the authors make use of a two-dimensional direction map in order to guide the learning process of the watershed transform. In this map, each pixel is associated with a 2D unit vector pointing in the direction of the closest boundary point to that pixel. However, since pixels belonging to non-object classes do not possess instance boundaries and, therefore, can not be assigned to a meaningful direction vector, this procedure requires a semantic segmentation map in order to be able to discern pixels belonging to an object class (things) from pixels belonging to non-object classes (stuff). In this paper, we built upon the proposed representation of Bai2017 , but overcome the requirement of an a-priorly known semantic segmentation by proposing a three-dimensional, instead of a two-dimensional, orientation map as intermediate instance representation, providing the flexibility to also represent non-object pixels by associating unit vectors pointing into the third dimension.

Research on semi-supervised segmentation focusses on the question of how unlabelled data, which is typically easy to acquire in large amounts, can be used together with small amounts of labelled data to derive additional training signals in order to improve the segmentation performance.

One strategy for making use of unlabelled data is based on entropy minimisation Kalluri2019 ; Wittich2020 , where additional training signals are obtained by maximising the network’s pixel-wise confidence scores of the most probable class using unlabelled data. However, this approach introduces biases for unbalanced class distributions, in which case, the model tends to increase the probability of the most frequent, and not necessarily of the correct classes.

In a semi-supervised segmentation setting using adversarial networks, the segmentation network is extended by a discriminator network that is added on top of the segmentation and which is trained to discriminate between the class labels being generated by the segmentation network and those representing the ground truth labels. By minimising the adversarial loss, the segmentation network is enforced to generate predictions that are closer to the ground truth and, thus, they can be applied as additional training signals in order to improve the segmentation performance. In this context, the discrimination can be performed in an image-wise Luc2016 or pixel-wise Souly2017 ; Hung2018 manner. Since the adversarial loss can be computed without the need for reference labels once the discriminator is trained, the principles of adversarial segmentation learning are adapted for the semi-supervised setting to leverage the availability of unlabelled data Souly2017 ; Hung2018 . Similar to the pixel-wise adversarial learning procedure of Souly2017 ; Hung2018 , the autors of Mendel2020 propose a correction network which is also added on top of the segmentation network and which learns on labelled data to distinguish between correct and incorrect class predictions. In the semi-supervised setting, the correction network is then used to produce additional supervision from unlabelled data based on the predictive certainty of the network. However, learning the discriminator and the correction network, respectively, adds additional demands for labelled data and, therefore, may not reduce the need for such data in a way other strategies do.

Closest to our approach is the line of research on semi-supervised segmentation based on the consensus principle. In this context, the authors of Ouali2020 train multiple auxiliary decoders on unlabelled data by enforcing consistency between the class predictions of the main and the auxiliary decoders. Similarly, in Peng2020 two segmentation networks are trained via supervision on two disjunct datasets and additionally, by applying a co-learning scheme in which consistent predictions of both networks on unlabelled data are enforced. In Coenen2021a , consistency training is additionally enriched by an auto-encoder branch, following the approach of auto-encoder regularisation Myronenko2019 ; Sedai2017 for semi-supervised learning. Another approach based on consensus training is presented in Xiaomeng2018 ; Zhang2020 , where unlabelled data is used in order to train a segmentation network by encouraging consistent predictions for the same input under different geometric transformations. While these approaches tackle the task of semantic segmentation, we extend the idea of consensus regularisation to a panoptic segmentation task by proposing the ConsInstancy loss, which enforces consistency between semantic instance representations and semantic segmentation maps. In Hao2021 , a contour prior is introduced for instance-wise segmentation by assuming that the instance-segmentation boundaries should align with strong image gradients. However, expecting large image gradients at instance boundaries is a rather strong hypothesis which does not necessarily hold true for all scenes, particularly in the case of our fresh concrete data set (cf. Sec. 4.1). Moreover, important to note is that in contrast to Hao2021 and other approaches for weakly supervised instance segmentation, as e.g. in Qizhu2018 ; Hsu2019 where weak annotations in the form of bounding boxes are needed for a weakly-supervised training, no additional annotations are required in our work.

On an abstract level, encoder-decoder networks for semantic segmentation learn a function $f:X\rightarrow Y$ which maps the input images $X$ to pixelwise class predictions $Y$, such that $Y=\mathcal{D}^{\text{seg}}(\mathcal{E}(X))$. In this setting, an encoder $\mathcal{E}(X)$ computes a latent feature embedding $\mathbf{z}$ from the input data and a segmentation decoder $\mathcal{D}^{\text{seg}}(\mathbf{z})$ is used to produce the label maps $Y$ from $\mathbf{z}$. Typically, $Y$ contains $i=1...N_{\_}C$ channels, one for each semantic class $C_{\_}i\in\mathbf{C}$, whose entries contain the pixelwise class-scores for the respective class $C_{\_}i$. In a panoptic segmentation setting, the semantic label set $\mathbf{C}$ distinguishes between subsets $\mathbf{C}^{\text{St}}$ and $\mathbf{C}^{\text{Th}}$, corresponding to stuff and thing classes, respectively, such that $\mathbf{C}=\mathbf{C}^{\text{St}}\cup\mathbf{C}^{\text{Th}}$. In this definition, things comprise classes of countable objects (here: the concrete aggregates) whereas stuff comprises classes of amorphous appearance and of similar texture or material (here: the cement suspension). The task of panoptic segmentation extends the objective of semantic segmentation by mapping each pixel $p$ belonging to the subset $\mathbf{C}^{\text{Th}}$ not only to its semantic class $C_{\_}i^{\text{Th}}$ but in addition, also to an unique instance id $o\in\mathcal{O}$, enabling the differentiation of individual instances, with $\mathcal{O}$ denoting the set of all object instances.

Given a data set $X=\left\{X_{\_}l,X_{\_}u\right\}$, this paper presents a novel strategy to leverage unlabelled data $X_{\_}u$ along with labelled data $X_{\_}l$ for the training of a segmentation network in order to improve its performance compared to only using the labelled data. Specifically, we propose an instance decoder $\mathcal{D}^{\text{inst}}$ which is trained to predict individual object class instances and which is added to the segmentation architecture while sharing the encoder $\mathcal{E}$ with the segmentation decoder $\mathcal{D}^{\text{seg}}$ (cf. Fig. 1). In this paper, we show that the proposed instance decoder serves multiple purposes. On the one hand, by formulating the simultaneous prediction of semantic segmentation maps and instance representations as a multi-task framework, hence exploiting the complementary information of both disentangled but correlated tasks, the discriminative capability of the intermediate feature representations is improved and therefore leads to enhanced segmentation results. On the other hand, we demonstrate how to benefit from largely available unlabelled data, incorporating it into the training procedure by enforcing consistency between the predicted instance representations and the segmentation maps in order to produce additional self-supervised training signals and, thus, to significantly improve the performance of the network. Lastly, we make use of the inferred instance representations to separate clustered objects, enabling a panoptic segmentation by generating instance-level semantic segmentation results.

Based on the latent feature embedding $\mathbf{z}$ produced by the encoder network $\mathcal{E}(X)$, the first output branch of our framework consists of a segmentation decoder $\mathcal{D}^{\text{seg}}$ (cf. Fig. 1) which predicts a pixelwise semantic segmentation $Y$.

The encoder and segmentation decoder architecture used in this work correspond to the Residual depthwise Separable convolutional Network (R-S-Net) proposed in Coenen2021a . This network consists of four encoder and decoder blocks, respectively. Each encoder-block consists of a residual convolution module, in which two intermediate representations are computed. The first representation is produced by a convolutional layer using a stride of 2, and the second one is computed by a sequence of a convolutional layer followed by a depthwise separable convolution layer and max-pooling. As output of the encoder-block, the elementwise sum of both intermediate representations is returned. Similar to that, the decoder-block processes the input in a two-stream path and returns the element-wise sum of the output of both streams. In the first stream, the input is upsampled by a factor of 2, followed by a convolutional layer. The second stream consists of a sequence of one convolutional layer followed by a depthwise separable convolution and an upsampling layer. For more details, we refer the reader to Coenen2021a .

For the supervised training of the segmentation decoder, labelled data $X_{\_}l$ and associated reference labels are used to compute a pixelwise categorical cross-entropy loss $\mathcal{L}_{\_}\text{CE}$.

In this section, we describe the instance representations that are proposed in this paper to represent instances of the thing classes. Examples of our representations are shown in Fig. 2. In particular, we train the instance decoder $\mathcal{D}^{\text{inst}}$ to predict the two instance-aware distance transform maps $\mathcal{I}^{\delta}_{\_}{+}$ and $\mathcal{I}^{\delta}_{\_}{-}$ for each class in $\mathbf{C}^{\text{Th}}$ as final output (cf. Sec. 3.3.1). Furthermore, we propose what we denote as instance orientation map $\mathcal{I}^{\theta}$ as class-agnostic intermediate representation (cf. Sec. 3.3.2). In our framework, this intermediate representation acts as additional guidance for predicting the final distance transform maps (cf. Sec. 3.3.3).

While a semantic segmentation of the input is delivered directly by the network as output of the segmentation decoder (cf. Sec. 3.2), deriving the panoptic segmentation using the proposed framework requires post-processing. In a first step, we make use of the predicted complementary distance transform map $\mathcal{I}^{\delta}_{\_}-$ in order to extract the outlines for the instances of each semantic class in $\mathbf{C}^{\text{Th}}$ by thresholding $\mathcal{I}^{\delta}_{\_}-$ using a threshold of $0.9$. Note that this is in contrast to Bai2017 , where the low distance areas of a regular distance transform map are used as energy cut, which, however, does not define the instance boundaries as well as compared to the complementary distance transform map proposed in this paper. In a second step, we subtract the extracted instance outline map from the binary semantic segmentation map of the corresponding class in $Y$, and subsequently, associate each remaining connected component with an individual instance ID. Finally, instance boundary pixels are allocated to the id of their neighbouring instance, resulting in a panoptic segmentation of the input image.

In this paper, we propose a strategy to incorporate unlabelled data $X_{\_}u$, in addition to the limited amount of labelled data $X_{\_}l$, to the training procedure of our segmentation network in order to improve its performance. To this end, we define the overall training objective as to minimise the overall training loss $\mathcal{L}$ with

being composed of supervised loss functions which require the availability of reference data and an unsupervised loss function which does not require any reference data to be available. The supervised loss functions produce training signals at the outputs of the network for the semantic segmentation mask, the orientation map, and the distance transform maps, respectively. These signals are based on the discrepancy between the predictions and the provided reference data and are computed according to the loss functions described in Sec. 3.2 and Sec. 3.3.3. We would like to point out that in order to compute the supervised loss functions, no additional annotations other than the instance-level annotations are required. Instead, the reference data required for the individual loss terms, i.e. the different instance representations, can directly be derived from the instance-level annotations. As a consequence, our proposed method does not add further annotation efforts but instead, makes use of different representations of existing annotations at instance level to enrich the training procedure by formulating the supervised part of the total loss in Eq. 4 as a multi-task learning problem.

In order to compute the unsupervised loss from unlabelled data, we propose the ConsInstancy loss which we define as

In this definition, we make use of the relationship described by Eq. 2, namely that the sum of the two predicted distance transform maps $\mathcal{I}^{\delta}_{\_}{+,i}$ and $\mathcal{I}^{\delta}_{\_}{-,i}$ for the thing classes $C_{\_}i\in\mathbf{C}^{\text{Th}}$ must result in the binary label representation $Y_{\_}i$ predicted for that class. This relationship allows us to introduce the MSE between the predicted label maps and the sum of the predicted distance transform maps as additional unsupervised training signals, derived from entirely unlabelled data (cf. Fig.1). By minimising this loss based on the discrepancy between the individual outputs, we enforce consistency between the predictions of the segmentation decoder and the instance decoder, enabling the exploitation of the consensus principle Chao2016 . This principle is founded on the rationale that enforcing an agreement between different outputs of the same network restricts the parameter search space to cross-consistent solutions and, therefore, acts as additional regularisation on the shared encoder, thus, enhancing its feature representation and improving its generalisation ability. A high-level overview on the proposed framework is shown in Fig. 1 and an overview on the training procedure of the proposed semi-supervised segmentation approach is given in Alg. 1.

We experimentally evaluate our proposed method on two different data sets of concrete aggregates. Both data sets used in this work distinguish the classes suspension (stuff) and aggregate (thing). The first data set is the concrete sedimentation data set proposed in Coenen2021a . It consists of 612 labelled and 827 unlabelled image tiles of hardened and lengthwise cut concrete cylinders with a resoultion of 448x448$\,\text{px}^{2}$. Exemplary tiles of the sedimentation data are shown in the top row of Fig. 5.

In contrast, the second data set contains images of fresh concrete acquired and annotated by ourselves during the standard approach for quality control of fresh concrete at construction sites, the so called slump test Gambhir2004 . It contains 1096 images of size 480x480$\,\text{px}^{2}$, manually labelled at instance-level. Furthermore, we made use of an additional set of 2000 unlabelled images for the semi-supervised training in our experiments. Exemplary tiles of our fresh concrete data are shown in the bottom row of Fig. 5 and an example image, together with its corresponding reference maps, is depicted in Fig. 6. From Fig. 5, the diversity of the appearance of both, aggregate and suspension in both data sets can be noted. In comparison to the sedimentation data obtained on hardened concrete, the fresh concrete data is more challenging as the particles are embedded in the viscous binder suspension, rendering parts of the instance boundaries indistinct and ambiguous.

Ablation studies: To assess the effect of the individual constituents of our framework, we perform tests using different network variants, considering different components in the evaluation. In the Seg setting, we train a semantic segmentation network by only using the segmentation decoder, while disregarding the instance decoder during training. Consequently, in this setting, we perform a purely supervised training without the incorporation of unlabelled data. In the Inst variant, we perform training using both, the segmentation and the instance decoder, but again, we only perform a purely supervised training. Thus, with this setting, the effect of multi-task learning, which is achieved by not only training the network for semantic segmentation, but at the same time for predicting the instance representations, can be analysed. In the ConsInst variant, we make use of our complete approach proposed for the semi-supervised panoptic segmentation. Thus, in this setting, the unlabelled data is leveraged by computing the ConsInstancy loss for a semi-supervised training.

Training: The networks used in the different variants of the proposed framework are trained from scratch. The convolutional layers are initialised using the He initialiser He2015 . The networks are trained using the Adam optimizer Adam2015 , using the exponential decay rate for the $1^{\text{st}}$ moment estimates $\beta_{\_}1=0.9$ and for the $2^{\text{nd}}$ moment estimates $\beta_{\_}2=0.999$. We apply weight regularisation on the convolutional layers using L2 penalty with a regularisation factor of $10^{-5}$. A mini-batch size of 8 is applied, meaning that in the semi-supervised setting, each mini-batch consists of four labelled and four unlabelled training images. We use an initial learning rate of $10^{-3}$ and decrease the rate by a factor of $10^{-1}$ after 25 epochs with no improvement in the training loss. In all settings, we make use of the same, very limited amount of labelled data for training. In case of the sedimentation data, we use only 17 labelled images and all unlabelled images for training, as suggested in Coenen2021a . In case of our fresh concrete data set, we considered 150 labelled images for training of the supervised components of the framework, and all unlabelled images for the semi-supervised part.

Metrics: The evaluation is carried out based on all annotated images that are not used for training. We report results for different evaluation metrics. In order to evaluate the performance of the approach related to the pixelwise semantic segmentation, we determine values for the class-wise recall (R), precision (P) and $F_{\_}1$ scores as well as the overall segmentation accuracy (OA). Additionally, since the OA can be biased towards more frequent classes, we report the mean $F_{\_}1$ score of the segmentation ($MF_{\_}1^{\text{seg}}$), computed as average of the class-wise $F_{\_}1$ score of all classes. Furthermore, we analyse the performance of the approach towards instance- and panoptic segmentation and report results for the instance-wise $F_{\_}1^{\text{inst}}$ score to assess the performance on instance-level segmentation and results for panoptic quality (PQ), the metric for panoptic segmentation defined in Kirillov2019 with

In this context, $p$ and $g$ denote predicted and ground truth segments, respectively. Furthermore, $TP$ (true positive) denotes correctly matched instances, $FP$ (false positive) and $FN$ (false negative) represent unmatched predicted and ground truth instances, respectively. For both, $F_{\_}1^{\text{inst}}$ and PQ metrics, we require the segmentation masks of instances to have an intersection-over-union (IoU) of 50% or more with a reference instance mask to be counted as true positive segmentation.

Comparison to state-of-the-art: We compare our results with the results achieved by two current state-of-the-art approaches for semi-supervised segmentation of Coenen2021a and Ouali2020 . To enable a fair comparison, we trained both approaches from scratch using the same labelled and unlabelled data that was used for the training of the proposed framework. The results achieved for the task of semantic segmentation can directly be compared to the results obtained by our approach. In order to compute the evaluation metrics for the panoptic segmentation, we identify connected components in the semantic label space. From these components we derive an instance-wise segmentation, since both state-of-the-art methods only deliver a semantic but no panoptic segmentation of the input. Note, that the same is true for the defined Seg variant of our framework, where only the segmentation branch is applied. We point out that a comparison of the panoptic segmentation metrics consequently is not one-hundred percent fair. However, in case of the sedimentation data, the effect that adjacent particles have on an instance-wise evaluation is almost negligible since an overlap of multiple particles in the profiles of the concrete cylinders is physically impossible and the occurrence of directly adjacent particles is very rare. Nevertheless, in case of the fresh concrete data set, the effect of identifying instances from connected components is an issue and, therefore, impacts the metrics for the panoptic segmentation of the approaches, which is why the comparison in Tab. 4 have to be taken with caution. Still, we decided to include the results in this paper, as they also show the contribution of our framework of extending approaches for semi-supervised segmentation by also predicting instance masks in addition to the semantic segmentation mask.

In this section, we evaluate the results achieved by the proposed approach for semi-supervised panoptic segmentation on the two described data sets. In this context, we analyse the effects of the individual components of the approach, nameley the prediction of the proposed instance representations and the usage of the ConsInstancy training.