Sensei: Self-Supervised Sensor Name Segmentation

As sensor names are created by humans (e.g., a technician with knowledge about building particulars), they often follow a certain naming convention (e.g., start with the building name, then room id, and then type). In addition, within a building, segments of sensor names corresponding to the same kind of information (e.g., location or function) would use similar phrases; e.g., the concept of “room” would be encoded as “RM”, “R”, or similar variants. A natural solution follows here: we would want to model the generative patterns in these names such that given the characters seen by far we can predict the next one. This coincides with the language modeling task in NLP.

Since the sensor name segmentation task works on characters, we adopt a popular character-level neural language model to capture the underlying sensor naming pattern. Specifically, we choose the classical Char-RNN Karpathy et al. (2015) architecture in our design and use LSTM Hochreiter and Schmidhuber (1997) as the RNN model. Note that, our method is compatible with any character-level neural language models.

Given a character sequence of length $N$, $X=\langle x_{1},x_{2},\ldots,x_{N}\rangle$, the Char-RNN learns the probability of observing a character given all the previous characters, namely, $p(x_{i+1}|x_{1},x_{2},\ldots,x_{i})$. During this process, we will obtain an embedding vector $\mathbf{x}_{i}$ for each character $x_{i}$, and a hidden state vector $\mathbf{h}_{i}$ after observing the characters from $x_{1}$ to $x_{i}$. A softmax layer is then applied to $\mathbf{h}_{i}$ to predict a distribution $\mathbf{\hat{p}_{i}}$ over the entire vocabulary:

where $\mathbf{w}_{c}$ is the linear transformation for character $c$. The cross-entropy between $\mathbf{\hat{p}_{i}}$ and the one-hot encoding of $x_{i+1}$ is used as the loss function for this character.

Given a building, we train the Char-RNN on all its sensor names. As each sensor name is independent of each other, we can have the same initial hidden state for each sensor name to ensure sensor names do not interfere with each other. Once the model converges, we apply it to all the sensor names to obtain the character transition probabilities, i.e., $\hat{\mathbf{p_{i}}}(x_{i+1})$. The perplexity of the trained Char-RNN in our experiments is typically small (i.e., $<0.3$ per batch with batch size 32). Therefore, we believe it captures the underlying naming pattern within the input building well.

Inspired by Shang et al. (2018), we use Tie and Break to decide the segmentation results. The transition between two adjacent characters ($x_{i},x_{i+1}$) is labeled as (1) Break when we should segment after character $x_{i}$, or (2) Tie otherwise, denoting that the two successive characters belong to the same segment.

For a given character sequence $x_{1},x_{2},\ldots,x_{N}$, we hypothesize that the transition probability $\mathbf{\hat{p}_{i}}(x_{i+1})$ obtained from Char-RNN is closely related to the Tie/Break relation between $x_{i}$ and $x_{x+1}$. Intuitively, the Char-RNN model should produce a high likelihood for common transitions in sensor names, e.g., substrings for building name, room, and common sensor types. Therefore, when Char-RNN suggests a low transition probability, the transition is very likely to be a Break; otherwise, the possibility of a Tie becomes higher.

We empirically verify our hypothesis via data analysis of an example building as shown in Figure 3. We present the probability density from histogram of $\mathbf{\hat{p}_{i}}(x_{i+1})$. In addition, based on the ground-truth segmentation results, we plot the Tie and Break precision curves w.r.t. different thresholds. The Tie Precision refers to the ratio of Tie transitions among all the transitions above a certain threshold, while the Break Precision refers to the ratio of Break transitions among all the transitions below a certain threshold. One can observe that the “turning points” on the break precision curve are highly correlated to the peaks in the histogram.

If one wants to set up a single threshold on $\mathbf{\hat{p}_{i}}(x_{i+1})$ to classify all transitions into {Tie, Break} in an unsupervised manner, the highest peak in the “confidence” interval $[0.550,0.950]$ on the distribution (e.g., $0.771$ in Figure 3) would be a good choice to achieve a high F₁ score. We generalize this threshold selection criterion to the other buildings, and as we shall demonstrate in our experiments, such a selection strategy gives results close to grid search that uses ground-truth labels.

In addition to Tie and Break, we mark those uncertain transitions as Unknown. We need to decide on two thresholds, $t_{0}$ and $t_{1}$, and categorize the transitions according to three transition probability intervals, $[0,t_{0}]$, $(t_{0},t_{1})$, and $[t_{1},1]$, denoting Break, Unknown, and Tie, respectively, as the pseudo labels. We wish these pseudo labels would be of high accuracy while having a sufficient amount of labels. Based on our observations, the above single threshold criterion satisfies $t_{1}$. Considering that Breaks are considerably fewer than Ties, we should decide on a Break more carefully. The highest peak in a narrowed high precision interval $[0.050,0.150]$ would be appropriate (e.g., $0.101$ in Figure 3).

There could exist errors in these automatically induced pseudo labels, so we leverage the idea of ensemble learning to mitigate the effects of these label errors on the final predictions Breiman (1996). Specifically, we independently sample a subset of pseudo labels to train $K$ binary classifiers and then average their predictions. In the pseudo labels, the number of Tie transitions is usually much higher than that of Break. To balance the training data, we sample $\epsilon\cdot M$ Tie and Break labels, respectively, from all the pseudo labels, where $M$ is the number of Break transitions and $\epsilon$ is a small coefficient between 0 to 1 for sampling a subset (e.g., $\epsilon=0.1$). Such a sampling strategy makes the label errors less likely to affect every binary classifier, so the final prediction becomes more accurate.

All types of binary classifiers could be used to construct the ensemble, and we adopt a multi-layer Perceptron (MLP) as our binary classifier. For the $i$-th transition, we retrieve the hidden state vector $\mathbf{h}_{i}$ yielded by the Char-RNN and feed it as input to the MLP. The final prediction is the average of predictions from the $K$ classifiers. As the training data is sampled in a balanced way, we simply use $0.5$ as the threshold to decide on Tie or Break.

To evaluate Sensei, we collect the sensor names from five office buildings (named A through E) of four different building vendors at three different sites located in different geographic regions. We also collect the character-level ground-truth labels of these names from their building vendors. We adopt the BIO tagging scheme in generating labels, marking the beginning (B), inside (I), and outside (O) of each segment (e.g., for location or function). The details of each building are summarized in Table 1.

We evaluate the performance of all the considered methods by the F₁, precision, and recall scores. A segment is represented as a span with the starting and the ending character indices. A predicted segment is correct if and only if there exists an exactly same segment in the ground truth. Therefore, we define the precision and recall as follows:

where $\mathcal{S}_{GT}$ is the set of ground-truth spans and $\mathcal{S}_{Pred}$ is the predicted set. The F₁ score is the harmonic mean of precision and recall. We report the averaged F₁ score of all sensor names, which is relatively unbiased Opitz and Burst (2019).

As we mentioned before, there will be some extra delimiters between segments. Therefore, during the evaluation, we ignore segments containing only delimiter(s) in both ground truth and predicted segments. When calculating the start and end indices for predicted segments, we also skip their prefix and suffix delimiters. The same process here applies to the evaluation of all methods.

We compare Sensei with the following methods:

• •

  Delimiter. There are punctuation (such as “-” and “_”) and whitespace characters in sensor names, and they could indicate the boundaries between segments. Therefore, this method segments a sensor name by delimiters (i.e., non-alphanumeric characters). This method mainly serves as a sanity check.

• •

  NLTK TweetTokenizer. NLTK Bird et al. (2009) provides a tweet tokenizer to segment a string into tokens according to predefined regular expressions (regexes). We directly apply it to segment our sensor names.

• •

  CoreNLP. We adopt the pre-trained tokenizer in the CoreNLP package¹¹1https://stanfordnlp.github.io/CoreNLP/ Manning et al. (2014), which adopts the Universal Dependencies²²2Universal Dependencies is a framework of annotation guidelines by open community effort. https://universaldependencies.org/ version 2 (UD v2) standard for segmentation .

• •

  Stanza. We also adopt Stanza³³3https://stanfordnlp.github.io/stanza/ and use its built-in neural tokenizer Qi et al. (2020) following UD v2. This method combines convolutional filters and bidirectional LSTM to realize tokenization and sentence segmentation as a tagging task Qi et al. (2018).

• •

  BayesSeg. Topic segmentation divides a document into topic-coherent segments. An unsupervised Bayesian model, BayesSeg⁴⁴4https://github.com/jacobeisenstein/bayes-seg Eisenstein and Barzilay (2008), is used to segment characters of sensor names as a topic segmentation task that decides the boundary between sentences. However, this method requires to manually specify the number of segments, which is a parameter we do not know without human input.

• •

  ToPMine. ToPMine El-Kishky et al. (2014) provides a method that groups frequent words into phrases in an unsupervised manner and incorporates these phrases into topic modeling. We adapt the model to work at the character level. That is, we regard each character of sensor names as a word in document and group characters into segments as group words into phrases.

Note that, we do not use custom regular expressions (regexes) to segment sensor names because they require tremendous manual effort to create in order to exhaustively cover all the possible substring patterns, which deviates from our self-supervised problem setting. Moreover, since different buildings follow different sensor naming conventions, manual effort is required from domain experts to create regexes on a per-building basis, which is a costly process.

We modify the Char-RNN library⁵⁵5https://github.com/sherjilozair/char-rnn-tensorflow and use Keras Chollet et al. (2015) to implement our method. As our method is unsupervised, we do not employ the commonly used early-stopping scheme when training the Char-RNN. Instead, we train our models for $100$ epochs and empirically find this to be sufficient. All the thresholds have three decimal places. We assign Ties as positives and Breaks as negatives. For binary classifier, any supervised learning algorithm (e.g., logistic regression, SVM, etc) would accommodate our need in this work. We choose a vanilla Multilayer Perceptron with $2$ fully-connected layers, each with $64$ cells. We set the number of binary classifiers in our ensemble, $K$, at $100$. The subsampling rate for the ensemble, $\epsilon$, is $10\%$ and for each subsampling, we use pandas with the iteration index as seed. Training a Sensei model on a Colab GPU with 12GB RAM takes less than 40 minutes for each building. For the other compared methods, we tune at our best based on the recommended settings in their papers or repositories and report the best performance.

Experimental results for all the methods are summarized in Table 2. Overall, Sensei outperforms all the compared methods significantly, attributed to its strategy of complementing the language model with a self-supervised ensemble classifier. Besides the variants of Sensei, the baseline Delimiter, though simple, has achieved the second best performance among all others methods. On average, Delimiter achieves $33.81\%$ in F₁ across all the buildings. By contrast, our Sensei achieves over $80\%$ in F₁, which demonstrates a $49$-point improvement over Delimiter.

When looking at the F₁ scores of the other baselines, including ToPMine, BayesSeg, and the off-the-shelf tokenizers in NLTK, Stanza, and CoreNLP, they are not competitive; this highlights the need of a solution to our challenging problem.

The performance of Delimiter also confirms the fact that the semantics of these delimiters are mixed. If one recalls the examples in Table 1, vendors usually use delimiters in sensor names. Sometimes, these delimiters well indicate the segment boundaries. However, as we illustrated in the example sensor name “SOD$|$H1$|$______$|$L_L”, punctuation could be also used within the segment, and therefore simply segmenting at delimiters results in a considerable amount of false positives.

From Sensei-FW to Sensei, there is a significant boost, roughly 27 points in F₁ on average. Since the major difference between Sensei and Sensei-FW is our self-supervised ensemble learning module, we empirically verified its power.

Comparing Sensei-FW and Sensei-BW, one can observe that the forward version performs dramatically better. As shown in Table 2, Sensei-FW performs better than Delimiter, ToPMine, and all the pre-trained tokenizers in all cases. By contrast, Sensei-BW takes the reversed sensor names as input but performs much worse than Sensei-FW. We notice that this is because there are not sufficient variations in the sensor string patterns when being looked at backward, compared to the forward case. For example, there are names like “SODA4R731__ASO” and “SODA1R516__VAV”, and the Sensei-FW model can see various substrings (e.g., “ASO” and “VAV”) following the common pattern “SODA0R000__”. Variations as such provide enough information for the model to learn where to segment. However, when reversed, the above example becomes “OSA__000R0ADOS” and the prefix “OSA” sees no variations following, which makes it nearly impossible for Sensei-BW to figure out the right segmentation. Consequently, Sensei-FW better captures generative patterns while Sensei-BW achieves poor segmentation results.

Comparing Sensei-FW and Sensei-GS, one can observe that, in most cases (4 datasets out of 5), Sensei-FW finds the best single threshold found by Sensei-GS. Note that Sensei-GS utilizes the ground truth to exhaustively search among all the possible thresholds, while Sensei-FW decides the threshold based on the transition distribution without requiring any labels. This small difference in performance indicates that our data-driven threshold finding solution based on the distribution is reasonable and reliable.

Since our Sensei framework is fully automated, its performance is solely decided by the amount and variety of available sensor names. As shown in Table 3, Sensei generally gets better performance with more sensor names available with an exception of Building A. We hypothesize that the performance relates more closely to the variety of sensor name patterns in the dataset rather than the size.

We next showcase some examples that Sensei correctly segments, in order to illustrate its capability.