Towards Real-world X-ray Security Inspection: A High-Quality Benchmark And Lateral Inhibition Module For Prohibited Items Detection

We construct the HiXray dataset in accordance with the following five principles:

Realistic Source. Considering realistic source can make the data more meaningful for research, we gather the images of the HiXray dataset from daily security inspections in an international airport to ensure the authenticity of data.

Data Privacy. We strictly follow the standard de-privacy procedure by deleting private information (name, place, etc.), ensuring that nobody can connect the luggage with owners through the images to guarantee the privacy.

Extensive Diversity. HiXray contains the 8 categories of prohibited items such as lithium battery, liquid, lighter, etc., all of which are frequently seen in daily life.

Professional Annotation. Objects in X-ray images are difficult to be recognized for people without professional training. In HiXray, each instance is manually localized with a box-level annotation by professional security inspectors of the airport, who are very skillful in daily work.

Quality Control. We followed the similar quality control procedure of annotation as the famous Pascal VOC [6]. All inspectors followed the same annotation guidelines including what to annotate, how to annotate bounding, how to treat occlusion, etc. Besides, the accuracy of each annotation was checked by another inspector, including checking for omitted objects to ensure exhaustive labelling.

Instances per category. HiXray contains 45,364 X-ray images, 8 categories of 102,928 common prohibited items. The statistics are shown in Table 2.

Instances per image. On average there are 2.27 instances per image in HiXray dataset. In comparison SIXray has 1.37 (in positive samples), both OPIXray and GDXray have 1 instance per image on average. Obviously, the larger average number of instances per image brings more contextual information, which is more valuable. The statistics is shown in Table 3.

Division of training and testing. The dataset is partitioned into a training set and a testing set, where the ratio is about 4 : 1. The statistics of category distribution of training set and testing set are also shown in Table 2.

Color Information. Different X-ray machine models may have some differences in color imaging, and we adopt one of the most classic color imaging strategy. The colors of objects under X-ray are mainly determined by their chemical composition, which is introduced in detail in Table 4.

Data Quality. All images are stored in JPG format with a 1200*900 resolution, averagely. The maximum resolution of samples can reach 2000*1040.

Our HiXray dataset can further serve the evaluation of various detection tasks including small object detection, occluded object detection, etc.

Small Object Detection. Security inspectors often struggle to find small prohibited items in baggage or suitcase. In our HiXray dataset, there are many small prohibited items. According to the definition of small by SPIE, the size of small object is usually no more than 0.12% of entire image size. We thus define the small object as the object whose ground-truth bounding box accounts for less than 0.1% of entire image, while the large object is defined as the object whose ground-truth bounding box takes up more than 0.2% proportion in the entire image, and the rest is the medium. The images of “Portable Charger 2” and “Mobile Phone” can be divided into three subsets respectively. The categories distribution is illustrated in Table 5.

Occluded Object Detection. The items in baggage or suitcase are often overlapped with each other, causing the occlusion problem in X-ray prohibited items detection. [40] proposed the occluded prohibited items detection task in X-ray security inspection. The occlusion problem also exists in HiXray dataset with large-scale images with more categories and numbers. In order to study the impact brought by object occlusion levels, researchers can divide the HiXray dataset into three (or more) subsets according to different occlusion levels (illustrated in Figure 2).

Figure 3 illustrates the architecture of our LIM. It takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels. Similar to FPN [17] and some other varieties like PANet [39], this process is independent of the backbone architectures.

Specifically, suppose there are $N$ training images $\textbf{X}=\left\{\textbf{x}_{1},\cdots,\textbf{x}_{N}\right\}$ and $L$ convolutional layers in the backbone network. One sample $\textbf{x}\in\textbf{X}$ is fed into the backbone network and computed feed-forwardly, which computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2.

Suppose $\mathcal{F}(\cdot)$ as a composite function of three consecutive operations: Batch Normalization (BN) [13], followed by a rectified linear unit (ReLU) [8] and a 3 $\times$ 3 Convolution (Conv). $\mathcal{F}^{l}(\textbf{x})$ is the feature map generated by the $l$-th layer of the network. Firstly, in the left part of BP, the noisy information is adaptively filtered because their propagation is reduced from high-level to low-level feature maps in the top-down pathway. Secondly, the output feature maps are fed into BA. BA refines the feature maps to activate the boundary information by enhance it from four directions and outputs the refined feature maps. Thirdly, similar to the left, the right part of BP reduces the propagation of noisy information from low-level to high-level feature maps through the bottom-up pathway. Finally, the feature maps outputted by each layer of the right of BP combine those of the corresponding layer from the backbone network. And the combined feature maps is conveyed to the following prediction layers. Algorithm 1 summarizes the whole process (We add explanations about acceleration operation in code implementation in Algorithm 1) and the details of modules are described in the following sections.

To disable the spreading of noisy information of neighboring regions, we mimic this mechanism by designing the bidirectional propagation architecture. Moreover, we add a dense mechanism to enhance the ability of BP to choose proper information to propagate.

As shown in Figure 3, for the dense top-down pathway on the left of BP, up-sampling spatially coarser but semantically stronger feature maps from higher pyramid levels hallucinates higher resolution features. These feature maps are enhanced by the corresponding feature maps from the convolutional layers via lateral connections. Each lateral connection merges feature maps of the same spatial size from the convolutional layer and the top-down pathway. The feature map of low convolutional layer is of lower-level semantics, but its activation is more accurately localized as it was sub-sampled fewer times. Further, we construct the dense connections to ensure maximum the filter.

Specifically, to preserve the feed-forward nature, $\mathcal{F}^{l}(\textbf{x})$ obtains additional inputs from the feature maps $\mathcal{F}^{l+1}(\textbf{x})$, $\cdots$, $\mathcal{F}^{L}(\textbf{x})$ of all preceding layers and passes on its own feature-maps to the feature maps $\mathcal{F}^{l-1}(\textbf{x})$, $\cdots$, $\mathcal{F}^{1}(\textbf{x})$ of all subsequent layers. Figure 3 illustrates this layout schematically. We define $\mathcal{U}^{m}(\cdot)$ as the up-sampling operation ($2^{m}$ times) and $\mathcal{V}(\cdot)$ as a $1\times 1$ convolutional layer to reduce channel dimensions. The process is formulated as follows:

where $\textbf{A}^{l}$ refers to the feature map outputted by the $m$-th layer of the left part of BP.

Regarding the right part of BP, as the Figure 3 illustrated, suppose the input feature map $\textbf{B}^{l}$ refers to the feature map whose boundary has been activated in Eq. $\left(\ref{BA_formular}\right)$ (Boundary Activation will be introduced in the following section). Similar to the previous definition, $\mathcal{D}^{m}(\cdot)$ is the down-sampling operation ($2^{m}$ times). This process can be formulated as follows:

where $\textbf{C}_{\mathrm{t}}^{l}$ refers to the output of $l$-th layer of the bottom-up pathway and $\textbf{C}^{l}$ refers to the feature map generated of the $l$-th layer of BP. Finally, we convey $\textbf{C}^{l}$, outputted by LIM, to the following prediction layers.

To mimic the mechanism that lateral inhibition creates contrast in stimulation that allows increased sensory perception, we activate the boundary information by intensifying it from four directions inside the feature maps outputted by each layer and aggregating them into a whole shape. The schematic diagram is shown in Figure 4.

As is shown in Figure 4, the key to capturing the boundary of object is to determine whether a position is a boundary-point. Motivated by the schematic diagram, we design the module Boundary Activation to perceive the sudden changes of boundary and its surroundings. Suppose we want to capture the left boundary of the object in the feature map $\textbf{A}^{l}\in\mathbb{R}^{{H}\times{W}\times{C}}$ (the output of left part of Bidirectional Propagation). $\textbf{A}^{l}_{c}$ donates the $c$-th channel of $\textbf{A}^{l}$. Further, $\textbf{A}^{l}_{ijc}$ refers to the location ($i,j$) of the feature map $\textbf{A}^{l}_{c}$. To determine whether there is a sudden change between a position and the left of the point, the right-most point $\textbf{A}^{l}_{iWc}$ traverses to the left. The process of perceiving the left boundary can be formulated as Eq. $\left(\ref{BA_formular}\right)$.

where the $\textbf{B}^{l}_{ijc}$ refers to the location ($i,j$) of $c$-th channel of the feature map $\textbf{B}^{l}$ after Boundary Activation.

LIM: LIM is implemented by PyTorch for its high flexibility and powerful automatic differentiation mechanism. The LIM-integrated model refers to the model that we implement this mechanism inside (Section 5.2). Both FPN and PANet contain feature pyramid mechanism similar to LIM, but they are not plug-in model. Therefore, we refer to their published code and re-implemented the mechanism deployed in SSD (Section 5.3). Unless specified, we use the following implementation details.

Backbone Networks: The backbone networks of SSD, FCOS and YOLOv5 are VGG16 [34], ResNet50 [10] and CSPNet [37] respectively. For each backbone network, we modify the corresponding network architecture to implement LIM mechanism.

Parameters: All experiments of LIM and baselines are optimized by the SGD optimizer and the initial learning rate is set to 0.0001. The momentum and weight decay are set to 0.9 and 0.0005 respectively. The batch size is set to 32 with shuffle strategy while training. We evaluate the mean Average Precision (mAP) of the object detection to measure the performance of all models fairly. Besides, the IOU threshold measuring the accuracy of the predicted bounding box against the ground-truth is set to 0.5.

We verify the effectiveness of LIM by implementing this mechanism to several detection approaches, including traditional SSD, the latest FCOS and YOLOv5. We integrate LIM to the three detection approaches and compare the LIM-integrated methods to the original baselines. In addition, we integrate the latest detection method for security inspection DOAM (in the work of OPIXray dataset) into the three detection approaches above and compare the results with our LIM. The experimental results on HiXray dataset and OPIXray dataset are shown in Table 6.

Table 6 demonstrates that in HiXray dataset, the LIM-integrated network improves the average performance by 1.7%, 1.6% and 1.5% over the original base models SSD, FCOS and YOLOv5 respectively. Besides, LIM outperforms DOAM by 1%, 1.1% and 1% with the base model SSD, FCOS and YOLOv5 respectively. In OPIXray dataset, the LIM-integrated network improves the mean performance by 3.7%, 1.1% and 2.8% over the original models SSD, FCOS and YOLOv5 respectively. Besides, LIM outperforms DOAM by 0.6%, 0.7% and 2.6% with the base models SSD, FCOS and YOLOv5 respectively.

Note that the performances are particularly low in two classes (CO and NL) for all models in Table 6, it is mainly because that, compared to other categories, NL and CO are far more difficult to recognize. For NL, it is very small in size and composed of a small piece of iron and a plastic body. The plastic shown under X-ray appears orange, which almost blends in with the background. For CO, the main reason is that there are big differences in the shapes of cosmetics, such as round and square, which are easy to be confused with other kinds of items.

LIM can be regarded as another feature pyramid method with a novel dense connection mechanism with specific feature enhancement. Therefore, We compare our LIM with the classical feature pyramid mechanism FPN and the variety PANet in different base models. Note that there is the same feature pyramid mechanism of FPN in FCOS and a variety of PANet mechanism in YOLOv5, so we replace the feature pyramid mechanism with our LIM in FCOS and YOLOv5 to verify that our mechanism works better in the base model FCOS and YOLOv5 (the same as Section 5.2). The experimental results are shown in Table 7.

LIM improves by both 1.1% to FPN and PANet in the base SSD model, 1.1% to FPN in the base FCOS model and 1.5% to the variety of PANet in the base YOLOv5 model. Further, we observe from the Table 7 that LIM improves significantly than FPN on the categories like “Portable Charger 1” (0.7%), “Portable Charger 2” (1.9%) and “Water” (1.1%). The visual information like boundary of the three categories is more abundant in their X-ray images, demonstrating the effectiveness of Boundary Activation in our LIM and verifies the novel dense connection mechanism with specific feature enhancement.

In this section, we conduct several ablation studies to in-depth investigate our method. We first analyze the effectiveness of the Dense mechanism by implementing the Single-directional Propagation (the left part of the Boundary Propagation) inside the base model. Then, we evaluate the performance of Boundary Activation alone that no boundary information aggregation inside the feature map. Moreover, we add the Boundary Activation module. The experimental results are shown in Table 8.

In Table 8, we can observe that the performance of the network with only Single-directional Propagation improves by 0.7% than the base model, verifying the effectiveness of our dense mechanism. After applying the propagation toward another direction, the performance improves by 1.2% than the base model and 0.5% than the Single-directional Propagation, which demonstrates the effectiveness of our bidirectional mechanism. Further, Table 8 shows that after the integration of our Boundary Activation module, the performance improves 1.7% than the base model and 0.5% than Boundary Propagation alone, indicating the effectiveness of boundary information aggregation inside the feature map. In conclusion, ablation studies have verified the validity of each part of our LIM model.

In this section, we visualize the accuracy of recognition and localization in Figure 5 and the effectiveness of LIM and traditional boundary-enhanced methods in Figure 6.

Figure 5 shows that the LIM-integrated model has a significant improvement over the baseline. In columns 1, 2, 5, 6 and 8, the detection boundaries of prohibited items by the base SSD model are not precise enough and the LIM-integrated model performs better obviously. In column 3, the cosmetic escapes from the detection of the base SSD model but is caught by LIM-integrated model with the confidence of 91%. In column 7, the base SSD model only detects one prohibited item while there are two, but both of them are detected by LIM. Figure 6 illustrated the effectiveness of our LIM and traditional boundary-enhanced methods, including DOAM [40], EEMEFN [44], etc.