Co-occurrence Background Model with Superpixels for Robust Background Initialization

The working diagram of CPB background modeling is illustrated in Fig. 3 including: the training process and the detecting process. In this work, the target pixel $p$ is compared with the $Q^{B}$ as block, and we define $\{Q_{k}^{B}\}_{k=1,2,...,K}=\{Q_{1}^{B},Q_{2}^{B},...,Q_{K}^{B}\}$ to denote a supporting block set for the target pixel $p$. Each frame is divided into blocks $Q_{K}^{B}$ of size $m\times n$ pixels:

Background changes in scene can affect the current intensity of target pixel $p$ in foreground detection. Hence, it is quite natural that block $Q^{B}$, being strongly correlated with target pixel $p$, can be used to determine the state of the latter. Block $Q^{B}$ can be introduced as a reference to estimate the current intensity of target pixel $p$, that is, there exists a correlation between pixel $p$ and block $Q^{B}$: $I_{p}=\bar{I}_{Q}+\Delta_{k}$ ($\bar{I}_{Q}$ is the average intensity of block $Q^{B}$ in the current detecting frame). In order to reduce the risk of individual error and perform robust background model, to select the sufficient number of block $Q^{B}$ with high correlation as supporting blocks is necessary, defined as follows:

where

where $\gamma$ is the Pearson’s product-moment correlation coefficient. Then, the Gaussian model is used to construct the co-occurrence model for each pixel-block pair:

where $I_{p}$ is the intensity of the pixel $p$ at $t$ frame and $\bar{I}_{Q_{k}}$ is the average intensity of blocks $Q_{k}^{B}$ at $t$ frame. The background model is built as a list consisting of $[I^{P},u_{k},v_{k},b_{k},\sigma_{k}]$, where $I^{P}$ is the average intensity of target pixel $p$ in $T$ sequence frames computed by training ($T$ is the number of training frames) and $(u_{k},v_{k})$ are the coordinates of supporting blocks.

At the detecting process, we use the correlation dependent decision for identifying the state of target pixel $p$ as shown in Fig.3 and more details are described in[13].

Superpixel segmentation has attracted the interest of many computer vision applications as it provides an effective strategy to estimate image features and reduce the complexity of subsequent image processing tasks[18]. Superpixels have been applied in various fields including object recognition[19, 20], image segmentation[21] and object tracking[22].

As most optical flow techniques assumed [23] that the motion field near motion boundary between foreground and background tend to be over-smoothed and blurred. Motion boundaries are the most important regions and incorrect motions near the area often lead a incorrect result in motion estimation. For effective motion estimation in a scene, we introduce the superpixel segmentation algorithm in the proposed algorithm to further acquire and differentiate the spatial texture information of foreground and background[24, 11]. Here, SLIC algorithm[14] is utilized on account of its low complexity and high memory efficiency in computation.

The steps of motion detection are as follows:

1. 1.

   To record the pixels $\left\{p(x_{i},y_{j})\right\}$ of the foreground detected by CPB;

2. 2.

   To estimate the value $V$ of superpixel regions $S$ in these pixels $\left\{p(x_{i},y_{j})\right\}$;

3. 3.

   Then, to detect the motion and acquire the motion mask $M$, when $\forall\{p(x,y)\}\text{in current frame}$ is denoted as:

   $$m(x,y)=\left\{\begin{array}[]{ll}{1}&{\text{ if }p(x,y)\in V}\\ {0}&{\text{ otherwise }}\end{array}\right..$$

   (5)

The motion mask $M=\left\{m(x,y)\right\}$. With the help of superpixel segmentation, the proposed approach can further acquire the spatial information of each pixel and distinguish the different motion information between foreground and background. Based on this, the proposed approach can reinforce the original CPB for extracting motion and avoid errors in information extraction from pixels.

Then, we replace the region of motion mask with the initial CPB background model for background generation as shown in Fig. 3.

In order to fairly evaluate the proposed approach without losing generality, we consider the several challenges in the background initialization algorithm[17]. The following challenges are selected from SBMnet for evaluation:

• •

  Basic: PETS2006 represents a mixture of mild challenges typical of the shadows and intermittent movement.

• •

  Illumination changes: Dataset3Camera2 with the illumination changes during day.

• •

  Background motion: advertisementBoard contains an ever-changing advertising board in the scene.

• •

  Camera jitter: boulevard contains the videos captured by outdoor unstable cameras.

• •

  Intermittent movement: sofa sequence with abandoned objects moving, then stopping for a short while, and then moving again.

Six metrics which are the common measurements for the background initialization algorithm [17, 11] are introduced for performance evaluation in this paper. They are explained as follows:

• •

  AGE (Average Gray-level Error): average of the absolute difference between GT and BI.

• •

  pEPs (Percentage of Error Pixels): number of pixels in BI whose value differs from the value of the corresponding pixel in GT by more than a threshold $\tau$, which is set as 20 in[17].

• •

  pCEPs (Percentage of Clustered Error Pixels): percentage of CEPs (number of pixels whose 4-connected neighbors are also error pixels) with respect to the total number of pixels in the image.

• •

  PSNR (Peak Signal to Noise Ratio): widely used to measure the quality of BI compared with GT, defined as $PSNR=10\cdot{\log}_{10}\left(\dfrac{255^{2}}{MSE}\right)$.

• •

  MS-SSIM (Multi-scale Structural Similarity Index): estimate of the perceived visual distortion defined in [25].

• •

  CQM (Color image Quality Measure): defined in [26]. It assumes values in db and the higher CQM value, the better is the background estimate.

Where, GT means the ground truth of the background image and BI means the generated background image computed by the background initialization approaches.

In this section, the proposed approach is compared with five different state-of-the-art techniques selected from SBMnet benchmark, which are LaBGen-OF[27], MSCL[28], FSBE[29], LaBGen-P-Semantic(MP+U) [30] and SPMD[11]. Four of them are the leading techniques for background initialization in SBMnet benchmark, especially MSCL[28] which is the top ranked techniques at present. All the results of the five different techniques come from SBMnet benchmark.

In experiments, we set each block as $8\times 8$ pixels with input frame size of $320{\times}240$ for CPB. All used parameters are listed in Table II, and a detailed discussion of parameters can be found in[12]. Experimental results of the background initialization are presented in Fig. 4, and Table I lists the overall evaluation of these approaches in different challenges. It can be seen from the above results as shown in Fig. 4 and Table II, that our approach outperforms other techniques in challenges of Basic and Intermittent Movement, and for Background Motion, our approach has a close performance to LaBGen-P-Semantic(MP+U), which is the best in this challenge. For other two different challenges, our approach also leads the intermediate level compared with other techniques and the performance is acceptable. The comparison shows that our approach is robust and effective for background initialization in different challenges.

The processing time for background initialization is close to 0.15 seconds with frame size of $320{\times}240$ in MATLAB platform (Intel i7 2.40 GHZ and 16G).