Millimeter-Scale Ultra-Low-Power Imaging System for Intelligent Edge Monitoring

The system uses un-packaged and thinned (150$\upmu$m) ULP ICs from a family of custom chips that are prior-designed to be vertically stacked and interconnected by attaching wire-bonds between IC bond pads as shown in Fig. 1a. This method increases the number of ICs that can be integrated in the same footprint in comparison to traditional planar 2D chip-to-chip connection on a PCB, allowing the system to achieve a 6.7$\times$7$\times$5mm size and 460mg weight (battery included). The integrated system (Fig. 1 and Fig. 3) consists of:

1) A base-layer: It integrates the following into a single IC die to reduce stacking height and wire bonding complexity.

• •

  A master controller, containing a Cortex-M0 microprocessor and 16kB SRAM, that is designed for ULP operation, provides the clock and acts as a bus mediator for the open-source ULP bus protocol mBus (Pannuto et al., 2015), used for inter-layer communication.

• •

  A power management unit that generates 0.6V, 1.2V and 3.6V domains from a single battery voltage in the range of 0.9-4V, and maintains high conversion efficiency under loads ranging from nW to hundreds of $\upmu$W (Jung et al., 2016).

• •

  A radio IC that uses energy-efficient sparse pulse position modulation consuming ¡70$\upmu$W of average active power (Chen et al., 2016). It is connected to a metal trace loop antenna integrated within the PCB, and uses a carrier frequency of 1.2GHz and sampling frequency of 240kHz generated by an on-chip oscillator without a crystal reference.

2) A ULP image sensor layer: It supports motion-triggered 12-bit VGA image capture and also near-pixel motion detection on a sub-sampled 32$\times$20 pixel frame at a maximum rate of 170 fps (Choo et al., 2019).

3) A ULP image signal processing (ISP) layer: It is a revised version of (An et al., 2020), that performs on-the-fly JPEG (de)compression, optical-black pixel calibration, de-Bayering, RGB-to-YUV conversion, and scene change detection when an image is streamed in. It has a H.264 compression engine and 180kB of SRAM to store compressed frames on-chip. A neural engine (NE) with a peak efficiency of 1.5 TOPS/W enables DNN-based frame analysis, and has 426kB SRAM and a custom RISC-like processor. An internal ARM Cortex-M0 microprocessor orchestrates these components. The ISP’s memory banks have separately-controllable power-gating switches to reduce leakage energy for non-retentive data.

4) Two stacked, custom ultra-low-leakage flash layers: They provide a total of 16kB SRAM storage and 256kB flash storage with 11pJ/bit read energy and pW sleep power (Dong et al., 2017).

5) An energy harvester layer: It is designed to be supplied by a low-voltage source that has a configurable conversion ratio to increase harvesting efficiency for a range of input voltages (Jung et al., 2014).

6) A solar cell layer: It is for energy harvesting and a global optical communication (Moon et al., 2019).

7) A 3V 3.4mAh 5.8$\times$1.8mm rechargeable lithium battery: It weighs 130mg with 150$\upmu$A maximum continuous discharge current and 10$\upmu$A standard discharge current.

8)A polytetrafluoroethylene (PTFE) tube: It is used as a lens barrel, housing a medical endoscopic lens.

Layers are connected in a daisy-chain configuration and forward mBus transactions between a transmitting and a receiving layers as shown in Fig. 2. The ISP’s image interface connects directly to the image sensor IC pads for raw data and synchronization signals. The ISP’s flash interface allows it to stream (de)compressed image data directly to flash. The power management unit generates 3 power domains that are distributed throughout the system. The battery is recharged by the harvester connected to the solar cell layer.

The system uses two 4 layer 10$\times$10$\times$0.8mm PCBs. The front of the first PCB is used for wire-bonding the stacked ICs whereas passive components and the solar cell layer are placed on the back (Fig. 1c). The second PCB provides spacing between passive components and the battery so these can be placed above each other.

Since code development and debug on miniature systems is extremely difficult due to limited physical access, we include castellated vias on the outer rims of the system to expose internal signals that connect to compression contacts in our test setup (Fig.1b). Fig. 3 shows a white outline where the system can be diced a second time into an even smaller form-factor of 6.7$\times$7$\times$5mm (Fig. 1d). With this smaller size, the system loses all exposed probe contacts for debugging and monitoring, thus must be wirelessly programmed through an optical communication interface using the solar cell.

Fig. 1 shows a completed assembly, where the stacked ICs have been encapsulated in black epoxy to block out light due to ULP IC’s light-sensitivity. The solar cell is encapsulated with clear epoxy to be exposed to light. The image sensor is also left exposed by attaching the PTFE lens barrel over it prior to encapsulation.

Most ICs in the proposed system are duty-cycled to optimize the system’s power consumption for the chosen application. Fig.5 shows the power breakdown of each mode. For instance, flash ICs are set to sleep during motion monitoring, image capture, and DNN-based scene analysis, consuming only 0.003$\upmu$W. ISP allows for fine-grained software control of its SRAM banks, which can be in low-power retention mode without loss of data. The NE SRAM is partitioned into 7 variably-sized blocks that can be selectively power-gated to reduce leakage power consumption according to DNN storage requirements. The H.264 engine and image interface memory also have this capability. For example, when the system is in continuous motion-detection mode, the 430kB ISP memory can be power-gated, reducing the leakage power by 1.8$\times$. The power management unit is adjusted at each mode to maximize the efficiency for the dynamic load by modifying current draw, frequency control and up/down conversion ratio, achieving an average efficiency of 64%.

A hierarchical event detection (HED) algorithm is employed to prune out irrelevant events that would otherwise wastefully consume energy, particularly when offloading data without determining its value to the user. This approach uses efficient motion detection in the imager front-end and lower complexity DNN-analysis in the ISP back-end to reduce the frequency at which costly VGA images and higher-complexity DNN algorithms are performed, significantly decreasing the system’s average power/energy.

While the image sensor IC is in motion-detection mode, which is performed on the sensor without requiring external processing or storage, other layers are set to sleep so that the system’s power consumption of 48.8$\upmu$W is dominated by the image sensor. If motion is detected, the system wakes up the ISP, which performs person detection on a sub-sampled frame, consuming 39.6$\upmu$J. If a person is detected, the image sensor takes a VGA frame, consuming 349$\upmu$J and 14.9$\upmu$J to stream the frame into the ISP so that it can perform face detection (643$\upmu$J). If a face is identified, face recognition is performed consuming 622$\upmu$J. Only when an unregistered face is detected, the frame is stored in flash and offloaded wirelessly.

Fig. 6a shows the average system power for the above scenario, when the image sensor’s motion detection event rate is fixed to once per minute while the person detection (PD) and the un-registered face detection (URFD) event probabilities are varied. The URFD rate combines both face detection and face recognition, and it is assumed that data written to flash and sent wirelessly is uncompressed. This analysis motivates the use of HED for the chosen application, whereas the red top corner demarcates the PD and URFD probabilities where performing HED yields no power savings. In all other scenarios, HED enables significant (up to $\approx 100\times$) power reduction especially when both URFD and PD rates are low. Fig. 6b shows the system power breakdown for 3 event cases. When the total event rate is low (‘Case A’), the system power is dominated by the image sensor, while the wireless power dominates as the event rate increases. Notably, the ISP computation power for DNNs is always insignificant compared to the image sensor or radio, showing the benefit of HED in reducing overall power.

High compression of up to 1.5bit/weight is achieved for DNNs by combining weight pruning, non-uniform quantization, Huffman encoding of quantized weights for convolutional layers, and index-based encoding for sparse fully-connected layers. The compressed sizes for person detection, face detection, and face recognition networks are 85kB, 112kB, and 107kB, respectively. Up to 1.75M compressed weights can be stored in the ISP NE on-chip SRAM, which are on-the-fly decompressed when moved to NE local SRAM buffers. Once weights are loaded into the SRAM buffers, they are heavily reused for convolution on large input activations so that the energy overhead of decompressing weights is negligible.

The proposed HED requires multiple DNNs but parameters for less frequently executed DNNs can be stored off-chip in flash to eliminate on-chip SRAM leakage power. This off-chip parameter loading approach, however, increases execution time and incurs energy overhead from inter-chip data movement. We first analyze the event frequency at which this on- vs. off-chip parameter storage trade-off is advantageous for each DNN, and then program each DNN based on the assumption of expected event frequency to minimize the average power consumption.

We use a combination of several compression methods to minimize the data footprint and reduce wireless transmission cost.

The fisheye-like lens distortion is modeled by capturing checkerboard images using our system. Geometric distortion correction can be expressed by a mapping between pixels in the distorted image $I_{d}$ and the pixels in the correct image $I_{t}$: $(x_{d},y_{d})\rightarrow(x_{t},y_{t})\;\;\;\forall(x_{d},y_{d})\in I_{d}$. Various geometric distortion correction algorithms have been extensively studied and implemented (Ahlen et al., 2003)(T.Toutin, 2004). However, for real-time processing, a majority of them require task-specific hardware accelerators which are not available on our system.

Instead, we construct a $640\times 480\times 2$ flow field matrix (FFM) that represents vertical and horizontal pixel-displacement vectors based on the offline constructed fisheye distortion model. This ICL can be performed by either the ISP Cortex-M0 or NE by representing the FFM as a sparse fully connected layer. This is advantageous compared to both conventional and learning-based distortion correction methods since it only requires a relatively small correction matrix, which has low memory and computational complexity.

Color correction is performed using a color correction matrix (CCM) that projects the RGB pixel values in the distorted image to the target RGB values at the same location in the corrected image, expressed by a linear equation $[R^{t}_{ij},G^{t}_{ij},B^{t}_{ij}]=[R^{d}_{ij},G^{d}_{ij},B^{d}_{ij}]\mathbf{A}$ for pixel $(i,j)$ where $\mathbf{A}$ denotes the CCM. Our CCM is obtained by capturing images of an eSFR (edge spatial frequency response) chart using the proposed system. The color restoration ICL is implicitly accomplished with a linear layer in the custom face detection network. Training data images are modified using the color distortion matrix to emulate the system and implicitly learn the CCM.

The proposed ICLs were tested using images in the COCO-faces dataset, a dataset with annotated human faces sourced from COCO 2017 (Lin et al., 2014), and on images captured by our system. We first pre-train the customized YOLO-Lite network without distortion to speed up convergence, then apply the distortion models to the training/validation dataset, and include the pre-determined geometric distortion correction ICL. Finally, we fine-tune the YOLO model for color correction.

The performance of our proposed YOLO-Lite face detection network with ICLs is quantified by evaluating the average intersection over union (IoU) and average recall. In Table 1, we evaluate six schemes: 1) original images, 2) images with modeled imager geometric and color distortions, 3) geometric distortion correction, 4) additionally fine-tuning the face detection network using the color distorted training dataset, 5) additionally applying 50% network pruning, 8-bit fixed point quantization, and parameter compression to the final network (necessary for on-chip parameter storage), and 6) testing scheme 5 on a 112$\times$112 pixel gray-scale input image.

The results show a substantial performance drop after applying the modeled distortion to the original image data. Both performance measures are improved to a level close to the original when lens distortion and color restoration ICLs are applied. We experimented on the impact of using a smaller input gray-scale image, together with down-graded intermediate feature maps (case 6 in Table 1). The results demonstrate that our proposed image correction and face detection pipeline give sensible performance in this case, showing that it can be extended to other highly-constrained systems.