Smart Inference for Multidigit Convolutional Neural Network based Barcode Decoding

Linear 1D barcodes appeared in the 1970s and are now become ubiquitous on almost all consumer products and for logistics due to its ease of identification. Some newer tagging technologies emerged over the last decades allowing more information (e.g. RFID, NFC) stored. However, none of them has fully replaced its role in the industry because of its legacy and its economy. The low cost of printing barcode and the durability of the tag under minor damages make it remain an industry standard (standardized by GS1) for the coming decades.

One essential property of the tagging technology is that it must be read quickly, robustly and accurately using readers. For the barcode case, the readers (or the scanners) are categorized into 3 types: laser-based, LED-based, and camera-based. In the first 2 types, the laser/LED ray needs to be close to the barcode, requires no stripe obscured on the ray line, and suffers the problem of emitter overheating. Camera-based readers have some advantages over laser/LED-based solutions. The first advantage is built on the fact that numerous smartphones with high-quality cameras integrated are already in use. With Internet connection, useful mobile applications were born by online retrieval of product information and giving out ingredients information, alerting allergies, calorie intake, comparing prices between sellers; or for retailers, they learn eye-catching products, have consumer feedback and so on (e.g. in [1]). Another advantage of the camera-based solution is the possibility of multiple and long-range recognition by the support of computer vision algorithms.

However, most current techniques (static image processing and pattern matching) being used in camera-based readers have flaws that limit their usability. The main problem with them is the need for well-framed flatbed-scanned-style input than normal captured. Wilder but common-captured conditions such as underexposed, occluded, blurry or curved, non-horizontal position (as in Figure 1) become unrecognizable. This requires the user correction which is unhandy and slows down the scanning process. There are 2 separate tasks in scanning barcode: detecting (i.e. locating) where the barcode region in the image and decoding detected region to barcode sequence. Recent works showed that the first task is nearly solved even in challenging conditions.

On the other hand, decoding those challenging barcodes still needs to improve since existing works still have many limitations. Traditional methods presented in [2, 3, 4] apply traditional techniques like Hough transformations, scanline-based approach with thresholds for binarization based on certain assumptions of barcode characteristics while they are not always true. Many evaluated their tools on unpublished sets, some published their sets but small and not enough challenging conditions. With those limitations and the successes of convolutional neural network (CNN) in many applications, [5] was the first proposed work using CNN to decode these difficult codes. However, their work has some weak points making the performance much lower than CNN potential. Not only their CNN feature extractors are simple but also their input assumption is oversimplified. They only assumed the horizontal barcode as input; their test set is made from printed rectangle-shaped generated barcodes on plain papers while real-life barcode is printed in customized shapes (e.g. coca icon shape) on various materials with many kind of distortion and sometimes covered by film. They did not also consider the possibility of running the task on edge devices because of unoptimized models.

Therefore, in this study, we proposed a CNN-based method to solve decoding task with the following contributive points: (i) we proposed Smart Inference - 3 algorithms leveraging the feature of having checksum and test-time augmentation built on top of trained deep CNN models which considerably boost model accuracies and reduce false prediction; (ii) we made a challenging 2500-sample-cropped EAN13(UPC-A, ISBN13 are its subsets) barcode dataset from real captured images on various (included harsh) conditions on numerous products - this dataset is published for other researchers to evaluate their models and encourage more contribution on this task; (iii) lastly, we applied knowledge distillation technique with a target to have a lightweight model from the best model which is suitable on handheld devices, the experimental result consequently confirmed the possibility by a good inference speed on a real edge board.

Regarding the barcode locating task, there are some methods presented with improving performance over the past decade. In 2011, Lin et al [6] presented the first multiple and rotation tolerated barcode recognition methods. This work focused more on detecting problem by using several image processing schemes such as Gaussian smoothing filtering to segment out barcode regions, enhanced the stripes, rotated the regions to horizontal angle and put into a decoder with voting. Although the method did well on lottery barcodes (printed on plain papers), it was still slow and didn’t get high accuracy on a dataset of merchandise products which was unclear about the challenging level. Katona et al [7] in 2012 proposed a method using morphological operations to also segment out 1D and 2D barcode under blurry, noise, shear and various rotated conditions with good performance. Soros et al [8] continued dealing with blur using structure matrix and saturation from HSV color system to detect blurry barcodes better but with the expense of lowing speed in 2013. Recently, Creusot et al. [9] proposed a faster method for blurry barcodes based on Line Segment Detector after their previous work [10] using Maximal Stable Extremal Region shown sensitive to blur. In another way, Hansen [11] first tried to apply an object detection deep learning model (YOLO) on both 1D and 2D codes with the best bounding box detection rate.

While the task of barcode detection nearly reached its saturation, several works on decoding had been proposed sparsely since 1990s. Early works [2, 3] achieved their goal by techniques as Hough transformation, wavelet-based peak location on their simple (scanned-style) inputs. Wachenfeld et al [4] proposed a scanline-based approach accompanied by an EAN13 dataset (so-called MuensterDB). However, since their method was based on scanline approach at that time, it just worked well on slightly rotated (±15 degrees) barcode, stronger rotated or distorted would be problematic. Similar to [7], Zamberletti et al [12] also tackled the problem of out-of-focus (blurry) barcode by using multilayer perceptron model to find parameters of adaptive thresholding (instead of standard binarization) to restore blurry image to clearer image and put it into Zxing [13] to decode. Nonetheless, this approach is simple with low recall and time-consuming (2 steps). Recently, Yang et al [14] tried to address 2 tasks on 5 rigorous datasets. The work outperformed all other methods very well on EAN13 barcodes, but since the method was heavily based on scanline-base and hand-crafty featuring and analysis for each challenging condition, it’s a less scalable solution to extend the all other 1D barcode types (even though all 1D codes use stripes, they might differ in guard bar layouts, in how black-and-white stripe mixing) and the case of double-obscured condition (Figure 1) would be inapplicable. Lastly, Fridborn [5] in 2017 first leveraged the power of CNN to directly extract features and predict to 13 outputs (correspond to 13 digits) simultaneously (similar to [15] in Street View House Numbers problem). Compared with traditional and hand-crafty featuring methods, CNN-based approach is relatively more straightforward and data-driven rather than case-by-case analysis. One obvious example is the double-obscured condition which is problematic for scanline-based approaches but could be easily learnt and overcome by CNN classifier. Thus, our work is also CNN-based, however, differs from [5] by following points: (i) we use more advanced CNN feature extracting models; (ii) our input assumption is more practical as well as our training set and evaluation set covered more cases; (iii) we proposed Smart Inference exploiting the checksum attribute of barcode sequences to enhance model accuracies; (iv) we considered minimizing and verify the possibility of CNN-based approach on a real edge device.

Referring to test-time augmentation we used in this work for enhancing the inference accuracy a model, the technique is commonly used in deep learning as this survey [16] and can be found in AlexNet paper [17], ResNet paper [18]. While train-time data augmentation gives more variants of the dataset to let the model also learn all possible variants; test-time augmentation also applies some proper modifications to original samples, let the model give multiple predictions on those modified versions and pick the most suitable one among these predictions by voting mechanism. How to augment data for better performance is also one trendy topic in deep learning now with such papers like AutoAugment [19], Smart Augmentation [20]. In our work, we integrated test-time augmentation into Smart Inference quite effectively.

On the topic of model compression and deep learning applicability on mobile, Cheng et al [21] categorized methods into 4 types: Parameter pruning and sharing, low-rank factorization, compact convolutional filters and knowledge distillation. The first one reduces redundant parameters which are not sensitive to the performance while the second one uses matrix decomposition to estimate the informative parameters. The third one builds special filters to save parameters for only convolutional layers whereas the last technique trains a compact neural network with knowledge distilled from a large model which is so-called teacher model. For simplicity, in this paper, we used the original knowledge distillation (KD) method proposed by Hinton et al [22].

The base approach we use in this work is to train a probabilistic model of decoding barcode sequences given barcode images as [5]. Let $\mathbf{D}$ represent the barcode sequence and $X$ represent the input barcode image. The goal is to learn a model of $P(\mathbf{D}|X)$ by maximizing $\log P(\mathbf{D}|X)$ on the training set. $\mathbf{D}$ is modelled as a collection of $n$ random variables $D_{1},...,D_{n}$ representing $n$ digits of the decoded sequence. To simplify, assume that the value of the each digits are independent from each other, so that the probability of a sequence $d=d_{1},...,d_{n}$ is given by $P(\mathbf{D}=\mathbf{d}|X)=\prod^{n}_{i=1}P(D_{i}=d_{i}|X)$. Each of the digits is discrete and has 10 possible values (0 to 9). This means each digit could be represented with a softmax classifier that receives as input features extracted from $X$ by a CNN. This type of model is originally proposed by [15], so we call it Multidigit CNN. During the training phase, the loss is calculated by the sum of all cross-entropy losses of digits as usual. However, in the inference phase, instead of normal inference, we propose a modification named Smart Inference (SI), which is one of the main contributions in this work. The detail of SI is described next paragraphs. The overall model is shown in Figure 2.

In this work, we have proposed Smart Inference for Multidigit CNN based models to improve the performance of 1D barcode decoding. We have collected multiple real barcodes with label data to train and test the proposed model. We have also added better synthesized data to strengthen the training and testing process. The algorithms proposed during testing time boosted the performance over the base models. It not only outperforms the base model in terms of accuracy but also has small inference time which makes it efficient. The Multidigit CNN based approach with Smart Inference is also a scalable solution as it could extend to decode more than one barcode type. We have also shown that distillation technique transfers effectively knowledge from the best model to the shallower model to run on low computational edge devices and performs clearly better than training with normal loss function.

Even though the performance is better in terms of accuracy, the proposed model has a limitation in predicting false records (Dynamsoft also predicts 3 errors). Another limitation of this approach is that it is not applicable for non-fixed length barcode types such as Code39. In the future, the problem of false predicting can be mitigated by applying product recognition techniques.

This work was supported and funded by the Ministry of Science and ICT (MSIT) under the Korea-EU Joint Research Support Project of National Research Foundation of Korea (NRF-2016K1A3A7A0395205414), Main Research Program (E0162502) of the Korea Food Research Institute(KFRI), and Grand Information Technology Research Center support program (IITP-2020-0-01489) supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation).