Proactive Network Maintenance using Fast, Accurate Anomaly Localization and Classification on 1-D Data Series

The DOCSIS protocol enables two-way radio-frequency (RF) communication over coax, a technology providing much of the access network for entertainment programming and internet access for the world, not to mention a large amount of networking for businesses.

It relies on analog RF transmission of data, and uses a mix of methods including single carrier quadrature amplitude modulation, OFDM/A (3.1), SC-QAM (3.0), etc., depending on the version and implementation.

For a long time, it has included several resiliency mechanisms including Forward Error Correction (FEC), adjustable profiles with differing data rates per frequency carrier, echo cancellation, and equalization. These resiliency mechanisms adjust service to account for plant problems that create impairments to the RF transmission signal. When these adjustments are made, it indicates in many cases an imperfect transmission medium. When this imperfection goes beyond the designed and installed quality level, it indicates network (plant) damage or degradation. This mechanism therefore can be monitored to indicate changes or problems in the network (plant). The operator can know of a problem before the customer is impacted. When a problem is indicated, a proactive repair opportunity may be created. PNM is the industry term for addressing plant problems before service is impacted.

PNM was envisioned in the cable industry more than a decade ago, when DOCSIS networks were relatively new, and resiliency mechanisms were creating the opportunity[14]. Several measures were identified based on then-understood failure modes and risks associated with deployed technology. At the time, for example, analog optics were common for feeding the coax plant, so in some systems optical clipping was more common than it is today with digital optical systems. But the coax plant remains and will for some time, bringing with it several failure modes that result in RF transmission issues in certain frequencies, and impact on subsequent layers of the communication connection.

General components and their failure modes in the outside plant that can appear as impairments in the RF signal include but are not limited to the following:

• •

  Hard line or drop cable – shield, sheath, conductor, insulator

• •

  Connector – center conductor, shield, thread

• •

  Tap – cover, connector, electronics

• •

  Splitter, combiner, coupler – cover, connector, electronics

• •

  Filter – cover, connector, electronics

• •

  Splice – wrap, fill, shield connection, conductor connection, strap

• •

  Amplifier – cover, connector, electronics

• •

  Power Supply – cover, connector, electronics

• •

  Anchoring – various types such as straps, guy wires, and more.

• •

  Node – cover, connector, electronics

General failure modes of the cable plant which impact RF signal include the following

• •

  Passive – loose or misaligned connection, corroded, poor installation, damage, degradation, wet, cracked, loose and moving, incorrect part.

• •

  Active – poorly made, poorly installed, electrostatic discharge (ESD), lightning, ground fault, degradation or wear out, damaged, incorrect part.

Measurements identified for proactive network maintenance in the specification include the following, though some of these measurements are more general and not specifically identified for PNM:

• •

  Downstream spectrum capture

• •

  Downstream symbol capture

• •

  Downstream channel estimation coefficient

• •

  Downstream constellation display

• •

  Downstream receive modulation error ratio (RxMER) per sub-carrier

• •

  Downstream forward error correction (FEC) summary

• •

  Downstream required quadrature amplitude modulation (QAM) MER

• •

  Downstream histogram

• •

  Downstream modulation profile

• •

  Upstream pre-equalization

• •

  Upstream spectrum capture

• •

  Upstream RxMER per sub-carrier

• •

  Upstream FEC summary

• •

  Power level

For the version of the anomaly detector described in this paper, we will focus on downstream RxMER per sub-carrier as the measure used. While not a complete solution for identifying and removing RF impairments in coax networks, it is an important measure and can address a very significant part of PNM. RxMER per subcarrier is a measure of the signal to noise of each downstream frequency used to carry data to the cable modems. The cable modem can report these data because it monitors the frequencies over which it receives data, and those frequencies occasionally carry calibrated pilot signals. Those signals are known, so they can be compared against a received signal to determine a signal to noise ratio. Ideal signals are mostly flat across all subcarriers, with small amounts of variation acceptable. More detail about this measure can be found in[8][9][10][11][12]. Many impairments show up in RxMER per sub-carrier data, visually, statistically, and otherwise[11].

Impairments that appear in the spectrum (and RxMER per sub-carrier data) are classified in the cable industry as follows

• •

  Standing waves – when an RF signal transmits though coax and encounters an impedance change, the signal loses strength in the forward direction, and an echo returns in the opposite direction. If there are two impedance mismatches in the line, then an echo tunnel can form, causing repeating echo signals. When the impedance mismatches are stable, the echo tunnel appears in the RF signal as a standing wave, which impedes the signal in at least two ways: energy loss due to the impedance mismatches, and noise from the echo. Both of these factors impact the RF signal. Echo cancelers can take care of some of the impact at some points, but only if there is a signal with which to cancel. Power levels can adjust somewhat for fixed signal level effects, but not in all cases.

• •

  Resonant peaking – sometimes in frequency data we see certain frequencies with higher energy than others. We have not seen this in RxMER per sub-carrier data, and due to the nature of the measurement we don’t expect to encounter a resonant peaking, though the same pattern may appear in the data due to other frequencies experiencing impairment issues.

• •

  LTE or FM Ingress – when there are opens in cable plant, RF from the outside can get into the cable network. FM and LTE cellular signals (as do VHF and UHF bands) overlap the spectrum used in DOCSIS networks, so can appear as noise when the cable plant is open due to shield failures (shield integrity problems).

• •

  Suck-out – An ideal cable plant transmits evenly on all frequencies, but actual cable plant transmits better at lower frequencies than at high frequencies, so amplification is adjusted to make it behave more ideally. Typically, we see this as a slope in the power levels, when not adjusted for in transmission. But grounding issues in the cable plant can lead to some frequencies being highly attenuated when they should not be. These appear as a rapid drop in a band of frequencies, like the energy is being sucked out of the system at that frequency. The frequencies affected by the suck-out become unusable when severe enough, and at least must be compensated for otherwise. Because these problems can come and go over time and conditions, and often get worse over time, these are a perfect opportunity for PNM.

• •

  Roll-off – at the edge of a frequency band, frequencies at the edge can be attenuated more as you get closer to the edge. This is called a roll-off, and results in frequencies not being useful near that edge.

• •

  Filter – band filters in the plant can protect noise from getting into the system, but sometimes filters can attenuate frequencies on band edges unintentionally. Finding and removing these issues improves service.

• •

  Adjacency, second order distortion, and phase distortion are less common. Adjacency may appear as a step in the data over some sub-carriers. Second order distortion may appear as noise in RxMER per sub-carrier, and could potentially be captured by correlating with carrier signals whose energy appears in other frequencies, though detecting this impairment type requires correlating with the sent signal. Similarly, phase distortions require more than magnitude data (which RxMER per sub-carrier is) or potentially correlating with the complex I and Q values of the signal.

As access networks are physically a tree structure, an impairment has different impact on customers connected at different places of the plant. Taps include filters that protect customers from some issues. Recognizing RxMER is a measurement of the signal to noise that is received by the cable modem, and it is a measurement of the pilot signals only (which are of a known energy level), not all network problems will be discovered with RxMER. Further, due to the design of cable plant with diplex filters and taps, some impairments further downstream from a given cable modem may not impact that cable modem. These features help in localizing the problem sometimes.

The distinction between the network fault and service failure is a lot like the distinction between a software fault and software execution failure. In software, a fault may not lead to a failure in all cases, and may not be executed frequently or ever in an application. Likewise, a network fault may be hidden by resiliency mechanisms that come with DOCSIS networks and well-designed cable plant, or the impacted frequencies may not be used or used at lower data rates to compensate such that service is not impacted. Yet these impairments can grow to impact service as a failure, or service conditions can change such that the fault becomes a service failure.

We created the Proactive Operations[12][13] (ProOps) platform to enable automation of detection of PNM issues into operations processes that turn the data into action. This platform enables code modules, referred to as workers, that process data input to form statistics output. The data can be raw data, statistics, processed data, or otherwise. The output can be any statistic including a hard decision point. We direct but do not require a four layer architecture for organizing the workers in ProOps to fulfil one or more of the following layers

• •

  Observe – collect data from the network elements (usually cable modems identified by MAC address) and process into soft decisions or statistics for further analysis.

• •

  Orient – analyze the data from the observe layer and decide what additional data are needed, or more frequent collection of some data, or a broader scope of the collection of those data such as over neighbors or over longer periods of time, for example.

• •

  Decide – analyze the data, statistics, and soft decisions from the first two observe and orient layers to turn found anomalies in the data into network impairments, through classification of anomalies, clustering the data from the network elements, and applying either network health or severity measurements to quantify the indicated problems.

• •

  Act – by organizing the found problems by severity or network health scores, we can allow the operator to select which found issues deserve attention, and what type of human intervention or further work is needed, be that a field technician or someone in a network operations center.

ProOps is depicted in Figure 1 below, with the four layers represented. Also shown are the configuration interface, control and scheduling functions, the work queue for workers to process, and the various types of data stores used. The results can be observed on a dashboard, which can include maps showing impairments colored by severity, or graphs of the network health or impact of impairments that rise to attention, for example.

ProOps has been built with several types of workers, and can be configured to use these in various ways. For the sake of this paper, we will limit the discussion to the anomaly detector which will process RxMER data from cable modems and identify anomaly types at the first layer. The subsequent layer will collect additional information to assist in quantifying the severity of the found anomalies. Then the next layer will simply match impairment patterns as a simple clustering, and calculate a severity. The final layer presents a sorted list of the identified impairments to be addressed from most severe to least. At the point of this writing, only the clustering mechanism has not yet been built; we anticipate using a k-means method or like matching algorithm. The focus of this paper is the anomaly detection worker which is very efficient and effective at this particular problem.

The backbone network consists of 1-D convolutional layers with 1-D filters for feature extraction and shortcut layers for feature aggregation. The first 3 convolutional layers only have 4 filters with kernel size 3 and the maximum number of filters in the following layers is 64. This is because 1-dimensional data have much simpler features such that a smaller network can reach high mAP while featuring great performance and low resource consumption. Zero padding is used on both sides of the sample before convolution when the kernel size is greater than 1. This allows the network to maintain the consistency and relevance of the feature map sizes.

The shortcut layers perform addition on the feature maps from the previous layer and the layer it points to. It has multiple benefits such as smoothing the optimization during training and preventing the model from being attracted by spurious local optima as we learned in the previous study[6].

The Route layers are used for feature concatenation. When a route layer only points to one previous layer, the feature map output from the previous layer is forwarded as the output of the route layer. When a route layer points to multiple previous layers, their feature maps are concatenated and forwarded as the output of the route layer. In our network, features from lower level layers that contain more original information are concatenated with outputs from higher level layers to provide more information to the prediction layer to improve mAP.

The up-sampling layers are used to increase the size of the feature map from the previous prediction layer (the layer that predicts at larger grids) to match the size of the forwarded feature map from lower level layers. In our network we use a nearest neighbor interpolation algorithm for the up-sampling process.

Activation functions are implemented as layers in our network. All 1-dimensional convolutional layers are followed with Leaky Rectified Linear Unit (Leaky ReLU) activation layers except the 3 convolutional layers before the 3 prediction layers. The Leaky ReLU activation is calculated as

where $\alpha$ is 0.1 in our network.

We use batch normalization layers[7] in between 1-dimensional convolutional layers to normalize the input layer by adjusting and scaling the activations. Batch normalization improves the training speed, stability, and mAP of our network.

Instead of predicting the start, end, and the center of anomalies directly, we use anchors which are a set of hand-picked priors[1] that the prediction layers use as references. This simplifies the localization problem and makes it easier for the network to learn[2]. With this, the center prediction becomes

where $p^{\prime}_{center}$ is the final value of the predicted anomaly center, $\sigma(p_{center})$ is the network predicted value $p_{center}$ calculated with sigmoid function

and $g_{center}$ is the center of the grid that is performing prediction where a grid is defined as a prediction unit that takes input data from one feature map channel and handles all anomalies which have their center points inside of it. The width prediction of the anomaly becomes

where $p^{\prime}_{width}$ is the final value of the predicted anomaly width, $e^{p_{width}}$ is the network predicted value $p_{width}$ calculated with exponential function, and $\beta_{width}$ is the width of the current anchor.

We use both K-Means and kernel density estimation (KDE) algorithm to calculate proper anchor sizes. This allows the sizes of the prediction anchors to fit the distribution of anomaly width values in datasets as much as possible, and makes the problem easier for our network to learn. In our experiments with cable modems’ downstream RxMER data, the width value of anomalies ranges from 2 to 416 (with input size 416) as shown in Table I.

With specifically designed convolutional layers as the backbone, at the first prediction layer, the size of the input feature map is transformed to

where $S_{feature}$ is the total number of values in the feature map, $S_{input}$ is the original input size, $n_{classes}$ is the number of classes in the training and testing dataset, $n_{anchors}$ is the number of anchors that are being used by the current prediction layer, and $n_{\gamma}$ is the number of convolutional layers in the backbone network with stride 2.

The number of grids doubles when it becomes the second prediction layer for detecting mid-sized anomalies, and it doubles again when it comes to the third prediction layer for small anomalies. For instance, when using $1\times 416$ as the input sample size and there are 5 anomaly classes to predict and 3 anchors at each prediction layer, the size of the input feature map from a single sample at the first prediction layer is $1\times 24\times 13$. The size becomes $1\times 24\times 26$ at the second prediction layer and $1\times 24\times 52$ at the third prediction layer.

Looking further into the anchor prediction vectors from the above example, each of the three anchors predicts a value $p_{center}$ for the center of the anomaly, a value $p_{width}$ for the width of the anomaly, a confidence score $p_{conf}$ for the objectness of the anomaly, and 5 probabilities for each anomaly class. The predictions are initially labeled using softmax function on the class probabilities (if it is multi-label classification, classes with top $n$ probabilities are selected) and then filtered by a confidence threshold, which is $0.5$ in our network. Non-maximum suppression (NMS) is then used (only in inference process) to reduce the number of localization proposals on each anomaly. In our model, we apply NMS to each class predictions instead of all predictions. The final output is a list of predictions each consists of a class label(s), an anomaly confidence score, the center location of the anomaly (proportional), and the width of the anomaly (proportional).

In our experiments, we use downstream receive modulation order ratio (RxMER) per sub-carrier data captures from cable modems’ OFDM channels in DOCSIS 3.1[8] networks to test our algorithms and implementation. DOCSIS 3.1 cable modems are capable of capturing PHY layer data like downstream RxMER per sub-carrier based on requirements defined in DOCSIS 3.1 CM OSSI specification[9], which makes it a promising use case for real-world demonstration.

The training dataset contains 45000 RxMER samples and the testing dataset contains 26000 RxMER samples. Each sample has 1800 to 2000 sub-carrier MER values that are in range $[0,63.75]$. There are 5 classes of PHY layer impairments that are labeled in the datasets: LTE ingress noise, wave, roll-off, suck-out, and spike. The LTE ingress noise is normally identified as LTE signals that are adding interference to the OFDM signal through leakage points on the plant or defected shielding on the modem as shown in Fig.3; wave is identified by wave shaped impairments (as shown in Fig.5) that affect the whole spectrum or part of the spectrum and is normally caused by echos or impedance mismatch; roll-off is identified as rolling off RxMER values on either side of the OFDM spectrum as shown in Fig.6; suck-out is identified as dips with a sharp corner as shown in Fig.4, and is normally caused by issues on amplifiers; spike is identified as sharp dips as shown in Fig.3 and Fig.6, and are normally less than 3 sub-carriers wide. Each impairment on the sample is labeled with the center of the impairment $x$ (proportional), the width of the impairment $w$ (proportional), and the class $c$ of the impairment. Each RxMER sample can have multiple different types of impairments located at different frequency ranges. Example plots of RxMER per sub-carrier data captured from cable modems are shown in Fig.3 through Fig.6, showing various impairment types.

To reduce labor work and increase the number of training samples, we randomly generate synthetic training data based on human-observed samples and anomalies. For instance, in our datasets, we use randomly generated noisy base samples as the starting point: spikes are generated randomly on different frequencies with different MER values; suck-outs are generated using two randomized non-linear functions; waves are generated from different sine waves; roll-offs are generated with one non-linear function. All anomalies are added to different locations on the sample.

The synthetic data are used as an addition to the human labeled datasets for both training and testing. It significantly increases the amount of data that joins training, which helps the model to generalize better. And it adds more samples that are unseen by the model for validation.

While all input values are normalized to range $[0,1]$, the binning minimum algorithm (algorithm 1) is used for preserving low values in the original data series while reducing the input sample size. The algorithm first performs up-sampling (nearest-neighbor algorithm) on the original sample to make it become the nearest dividable size $D_{nearest}$ by the target sample size, then calculates binning minimum values from the up-sampled input.

During training, multiple data augmentation techniques are used to improve training results, improve the model’s generalization ability, and increase the number of training samples. Each of the data augmentation techniques has a probability of being selected for each input sample.

Stochastic gradient descent (SGD) is used as the optimizer in our experiments with 6000 burn-in mini-batches, 0.9 momentum, $1e^{-3}$ learning rate, $5e^{-4}$ weight decay, and mini-batch size 32. During the burn-in time, the learning rate increases gradually until it reaches the target learning rate

where $\alpha_{target}$ is the target learning rate, $\alpha_{current}$ is the learning rate of the current iteration, $N_{batches}$ is the number of trained mini-batches, and $N_{burnin}$ is the number of burn-in mini-batches.

Our loss function for training the model is designed with reference from YOLOv2[2]. To make it more efficient for our anomaly detection task and better balance loss calculation between small anomalies and large anomalies, we add scale weights in the mean square error (MSE) loss for localization loss calculation. This improves the model’s ability to localize small anomalies more precisely.

The localization loss of the center prediction is calculated as

where $N_{grid}$ is the number of prediction grids at the current prediction layer, $N_{anchor}$ is the number of anchors used at the current prediction layer, $C_{i}$ is the ground truth confidence which value can be 0 or 1, $x_{i}$ is the ground truth center location (proportional) of the anomaly, $\hat{x_{i}}$ is the predicted center location (calculated with the anchor) of the anomaly, $w_{i}$ is the ground truth width (proportional) of the anomaly, $\hat{w_{i}}$ is the predicted width (calculated with the anchor) of the anomaly, and $\gamma_{i}$ is the scale weight of the anomaly

where $w_{i}$ is in range $(0,1]$ and anomalies with smaller width have larger weights. The confidence loss is designed for the network to converge to a point that background data corresponds to confidence score 0, and anomaly data corresponds to confidence score 1. The confidence score is first calculated from the prediction layer’s linear output with sigmoid function

The confidence loss calculation is based on binary cross-entropy (BCE) loss, but use different weights for anomalies and background in order to balance the network’s recall and precision

where $C_{i}$ is the ground truth confidence score, $\hat{C_{i}}$ is the predicted confidence score, $\lambda^{conf}_{i}$ is the weight of the confidence loss which can be represented as

The classification loss is calculated based on BCE loss as

The total loss is calculated as

The convergence of the model is shown in Fig.7.

For each anomaly during training, only 1 prediction layer out of all 3 prediction layers is responsible for the prediction, and only 1 anchor out of all 3 anchors is responsible. The responsible prediction layer and anchor are selected by calculating which anchor has the best intersection over union (IoU) against the ground truth anomaly ignoring its center location

where $I_{anchor}$ is the responsible anchor’s index, $w$ is the current anomaly’s width, and $A_{i}$ is the $i_{th}$ anchor’s width. Once the index of the responsible anchor is calculated, the responsible prediction layer is determined by where the responsible anchor is used. Because the loss of all the other anchors used by 3 prediction layers’ is calculated as that these anchors see background with no anomaly, the ground truth confidence scores for these anchors are 0. The predictions from anchors which have IoU with the ground truth that are higher than the ignoring threshold $T_{ignore}$, which is 0.7 in our experiments, do not join the loss calculation for either anomaly or the background.

We use a small mini-batch size (32) in training, and use weight decay to reduce the possibility of overfitting. The extensive data augmentation during training helps the model generalize and train as completely as possible. We also validate the network’s performance every iteration using the testing dataset and stop the training early.