Analog Gated Recurrent Neural Network for Detecting Chewing Events

As demonstrated in Fig. 2, chewing is characterized by quasi-periodic bursts of large amplitude, low frequency signals that can be measured by a contact microphone or accelerometer that is mounted on the head [11, 5]. We can use the root mean square (RMS) and the zero-crossing rate (ZCR) to capture the signal’s amplitude and frequency, respectively. A second ZCR operation applied to the RMS and the initial ZCR will produce information about the signal’s periodicity. We implement the ZCR and RMS blocks based on the well-known rectifying current mirror. The details of the ZCR and RMS design may be found in [19, 20].

Fundamentally, an LSTM is a neuron that selectively retains, updates or erases its memory of input data [21]. The gated recurrent unit (GRU) is a simplified version of the classical LSTM, and it is described with the following set of equations [22]:

where $\mathbf{x}$ is the input, $h_{j}$ is the hidden state, $\tilde{h}_{j}$ is the candidate state, $r_{j}$ is the reset gate and $z_{j}$ is the update gate. Also, $\mathbf{W}_{*}$ and $\mathbf{U}_{*}$ are learnable weight matrices.

To implement the GRU in an efficient analog integrated circuit that contains no DACs, ADcs, operational amplifiers or multipliers, we can transform Eqn. (1)-(4) as follows. The $\sigma$ function of Eqn. (2) gives $z_{j}$ a range of $(0,1)$, and the extrema of this range reveals the basic mechanism of the update equation, Eqn. (4). For $z_{j}=0$, the update equation is $h^{\langle t\rangle}_{j}=\tilde{h}^{\langle t\rangle}_{j}$. For $z_{j}=1$, the update equation becomes $h^{\langle t\rangle}_{j}=h^{\langle t-1\rangle}_{j}$. Without loss of generality, we can replace $(1-z_{j})$ with $z_{j}$ (this merely inverts the logic of the update gate, and inverts the sign of the $\mathbf{W}_{z}$ and $\mathbf{U}_{z}$ weight matrices). So, replacing $(1-z_{j})$ and rearranging the update equation gives us

which is simply a first-order low pass filter with a continuous-time form of

where $\tau=\Delta T$, the time step of the discrete-time system. The gating mechanics of the continuous- versus discrete-time update equations are equivalent, modulo the inverted logic: For $z_{j}(t)=0$, Eqn. (6) is a low-pass filter with an infinitely large time constant, and $h_{j}(t)$ does not change (this is equivalent to $h^{\langle t\rangle}_{j}=h^{\langle t-1\rangle}_{j}$ in discrete time). For $z_{j}(t)=1$, Eqn. (6) is a low-pass filter with a time constant of $\tau=\Delta T$. Since the $\Delta T$ time step is small relative to the GRU’s dynamics, a time constant of $\tau=\Delta T$ produces $h_{j}(t)\approx\tilde{h}_{j}(t)$ (equivalent to $h^{\langle t\rangle}_{j}=\tilde{h}^{\langle t\rangle}_{j}$ in discrete time).

Various studies have found the reset gate unnecessary with slow-changing signals, and also for event detection [12]. Both these scenarios describe our eating detection application, so we can discard the reset gate.

Finally, if we translate the origins [23] of both $h_{j}(t)$ and $\tilde{h}_{j}(t)$ to $1$, then we can replace the $tanh$ with a saturating function that has a range of $(0,2)$. Such a saturating function can easily be implemented in analog circuitry, by taking advantage of the unidirectional nature of a transistor’s drain-source current. We replace both the $tanh$ and the $\sigma$ with the following saturating function,

translate the origin and discard the reset gate to arrive at the Adaptive Filter Unit for Analog LSTM (AFUA):

where $[\cdot]_{j}$ is the j’th element of the vector. Also, $\mathbf{x}$ is the input, $h_{j}$ is the hidden state and $\tilde{h}_{j}$ is the candidate state. The variable $\tau$ is the nominal time constant, while $z_{j}$ controls the state update rate in Eqn. (10). $\mathbf{W_{z}}$, $\mathbf{U_{z}}$, $\mathbf{W}$, $\mathbf{U}$ are learnable weight matrices, while $\mathbf{b_{z}}$, $\mathbf{b}$ are learnable bias vectors. The AFUA resembles the eGRU [12], which we previously showed can be used for cough detection and keyword spotting. But while the eGRU is a conventional digital, discrete-time neural network, the AFUA is a continuous-time system, implementable as an analog integrated circuit.

To realize the AFUA Eqns. (8), (9), (10) and (7) as an analog circuit, we first “dimensionalize” each variable and implement it as the ratio of a time-varying current and a fixed unit current, $I_{\rm unit}$ [24, 25]. For instance, we represent the update gate variable, $z_{j}$, as $I_{z}/I_{\rm unit}$.

The Eqn. (7) function is implemented as the current-starved current mirror shown in Fig. 4. Kirchhoff’s Current Law applied to the source of transistor M₃ gives

The transistors are all sized equally, meaning that, from Kirchhoff’s Voltage Law, the gate source voltage of transistor M₃ is

where we have assumed that the body effect in M₂ and M_b is negligible. If we operate the transistors in the subthreshold region, then Eqn. (12) implies

Combining Eqns. (11) and (13) gives us

Now, the current flowing through a diode-connected nMOS is unidirectional, meaning $I_{1}={\rm max}(I_{\rm in},0)$, and we can write

which is a dimensionalized analog of Eqn. (7). The measurement results in Fig. 5 illustrate the nonlinear, saturating behavior of this activation function.

The AFUA state update, Eqn. (10), is implemented as the adaptive filter shown in Fig. 6. The currents $I_{h}$, $I_{\tilde{h}}$ and $I_{z}$ represent the hidden state $h_{j}$, the candidate state $\tilde{h}_{j}$ and the update gate, $z_{j}$, respectively. From the translinear loop principle, the Fig. 6 circuit’s dynamics can be written as [26, 24]

where $\kappa$ is the body-effect coefficient and $U_{\rm T}$ is the thermal voltage [27]. Just as $z_{j}$ does for $h_{j}$ in Eqn. (10), $I_{z}$ controls the update speed of $I_{h}$ (see Fig. 7).

Figure 8 depicts the components of our vector-matrix multiplication (VMM) block. These are the soma and synapse circuits that are common in the analog neuromorphic literature [28]. Crucially, the soma-synapse architecture is current-in, current-out. This means that, unlike other approaches for implementing GRU and LSTM networks [14, 15, 16], the VMM does not need power-consumptive operational amplifiers to convert signals between the current and voltage domains.

Since the activation function, Eqn. (7), has a range of $(0,1)$, the $z_{j}$ and $\tilde{h}_{j}$ variables are likewise limited to $(0,1)$. Also, from Eqn. (10), $h_{j}$ spans $(0,2)$. This means that all update gate and candidate state currents have a maximum value of $I_{\rm unit}$, while the hidden state currents have a maximum value of $2$$I_{\rm unit}$. With this information, we can calculate upper-bounds on the current consumption of each circuit component.

The power efficiency of neural networks is conventionally measured in operations per Watt. But this metric does not apply directly to a system like the AFUA, since it executes all of its operations continuously and simultaneously. However, we can estimate the AFUA’s power efficiency by considering the performance of an equivalent discrete time system.

To arrive at the discrete-form AFUA unit, we first replace the state variables of Eqns. (8), (9) and (10) with their discrete-time counterparts. This includes the discretization $dh_{j}/dt=(h^{\langle t\rangle}_{j}-h^{\langle t-1\rangle}_{j})/\Delta T$, where $\Delta T$ is the sampling period. Then, we set $\tau=\Delta T$ to produce the following expression.

Recall that $\mathbf{W},\mathbf{W_{z}}$ are $2\times 2$ matrices, $\mathbf{U},\mathbf{U_{z}}$ are $1\times 2$ vectors and $z_{j}$ are scalars, meaning that each discretized AFUA unit executes $14$ multiply operations per time step. Also, there are $2$ divisions due to the two activation functions (see Eqn. (7)). Not counting additions and subtractions, each discretized AFUA unit executes $16$ operations per time step, to make for a total of $32$ operations/step performed by the network. Assuming the sampling period of $\Delta T=2$ ms used in our previous eating detection systems [6, 11], this implies the AFUA performs the equivalent of $16,000$ operations per second.

Now, setting $\tau=\Delta T=2$ ms requires a unit current of

where $C_{z}=57$ fF is the integrating capacitor of the translinear loop filter, $U_{\rm T}=26$ mV at room temperature and $\kappa\approx 0.42$. This gives $I_{\rm unit}=1.8$ pA. With a total current consumption of $62I_{\rm unit}$, a voltage supply of $1.8$ V and $16$K operations per second, the AFUA’s equivalent operations per Watt is $76$ TOps/W.

Due to random variations in doping and geometry, transistors that are nominally identical will exhibit mismatch when fabricated in a physical ASIC. To understand the effect of mismatch and other non-idealities on the AFUA neural network’s performance, we performed Monte Carlo analyses with foundry-provided manufacturing and test data. The Monte Carlo analyses included mismatch and process variation, as well as power supply voltage and temperature corners of $\{1.6V,2V\}$ and $\{0^{\circ}C,35^{\circ}C\}$, respectively.

Figure 10 shows the variation in classification accuracy for $250$ Monte Carlo runs of one implementation of the AFUA neural network. The median accuracy across all runs is $0.90$. Most of the variation in accuracy is due to mismatch, and the AFUA neural network is largely robust to temperature, voltage and process variation. The neural network is also unaffected by circuit noise (this is a direct result of the network’s ability to generalize). To mitigate the effect of mismatch, we can use larger transistors [29], calibrate the network’s learning algorithm for each individual chip [28], or incorporate mismatch data into a fault-tolerant learning algorithm [30].

Training and testing data was collected from study volunteers in a laboratory setting. All aspects of the study protocol were reviewed and approved by the Dartmouth College Institutional Review Board (Committee for the Protection of Human Subjects-Dartmouth; Protocol Number: 00030005).

The data used for this study was previously collected in a controlled laboratory setting from 20 participants (8 females, 12 males; aged 21-30) that were instructed to perform both eating and non-eating-related activities. During these activities, a contact microphone (see Fig. 11) was secured behind the ear with a headband, to measure any acoustic signals present at the tip of the mastoid bone [31]. The output of the contact microphone was digitized and stored using a 20 kSa/s, 24-bit data acquisition device (DAQ).

Participants were asked to eat a variety of foods—including carrots, protein bars, crackers, canned fruit, instant food, and yogurt—for at least 2 minutes per food type. This resulted in a 4 hour total eating dataset. Non-eating activities included talking and silence for 5 minutes each and then coughing, laughing, drinking water, sniffling, and deep breathing for 24 seconds each. This resulted in 4 hours total of non-eating data. Each activity occurred separately and was classified based on activity type as eating or non-eating.

We down-sampled the DAQ data to $500$ Hz and applied a high pass filter with a $20$ Hz cutoff frequency to attenuate noise. We segmented the positive class data (chewing), and negative class data (not chewing) into $24$-second windows with no overlap. The positive and negative class data were labelled with the one-hot encoding $(2,0)$ and $(0,2)$, respectively. Finally, we extracted the ZCR-RMS and ZCR-ZCR features of the windows to produce $2$-dimensional input vectors to be processed by the AFUA network.

For training, the AFUA neural network was implemented in Python, using a custom layer defined by the discretized system of Eqn. (18). Chip-specific parameters were extracted for each neuron and incorporated into the custom layers. The AFUA network was trained and validated on the laboratory data (train/valid/test split: $68/12/20$) using the TensorFlow Keras v2.0 package. Training was performed with the adam optimizer [32] and a weighted binary cross-entropy loss function to learn full-precision weights.

Python training was followed by a quantization step that converted the full-precision weights to signed $3$-bit values ($0,\pm 1,\pm 2,\pm 3$). An alternative approach would have been to directly incorporate the quantization process into the network’s computational graph [12]. However, we found that such an approach only slows down training with no improvement in our network’s classification performance.

The AFUA was implemented, fabricated and tested as an integrated circuit in a standard $0.18~{}\mu$m mixed-signal CMOS process with a $1.8$ V power supply. To simplify the measurement process and associated instrumentation, the ASIC I/O infrastructure includes current buffers that scale input currents by $1/100$ and that multiply output currents by $100$.

The AFUA neural network was programmed by storing the $3$-bit version of each learned weight onto its corresponding on-chip register in the VMM array.

The network was then evaluated on the test dataset. Specifically, each $24$-second long window of $2$-dimensional feature vectors from the test dataset was dimensionalized and scaled to $100\times I_{\rm unit}$ and input to the ASIC with an arbitrary waveform generator. We set $I_{\rm unit}\approx 10$ nA with an off-chip resistor. According to Eqn. (19), this $I_{\rm unit}$ creates a time constant of $\tau=0.36~{}\mu$s, allowing for faster-than-real-time chip measurements—an important consideration, given the large amount of test data to be processed.

Output currents $I_{h0}$, $I_{h1}$ were each measured from the voltage drop across an off-chip sense resistor. The ASIC’s steady-state response was then taken as the classification decision. An output value of $(I_{h1},I_{h0})=(2I_{\rm unit},0)$ means that the circuit classified the input as eating, while $(I_{h1},I_{h0})=(0,2I_{\rm unit})$ corresponds to non-eating. From these measurements, we calculated the algorithm’s test accuracy, loss, precision, recall, and F1-score.

Figure 14 shows the AFUA chip’s typical response to input data. The input currents $I_{x1},I_{x0}$ represent the ZCR-RMS and ZCR-ZCR features extracted from the contact microphone signal. Inputting a stream of $I_{x1},I_{x0}$ patterns produces output currents $I_{h1},I_{h0}$, which represent the hidden states of the AFUA network.

According to our encoding scheme, $(I_{h1},I_{h0})=(2I_{\rm unit},0)$ means that the circuit classified the input as chewing, while $(I_{h1},I_{h0})=(0,2I_{\rm unit})$ corresponds to a prediction of not chewing. But the presence of noise and circuit non-ideality produces some ambiguity in the encoding: some AFUA output patterns can be interpreted as either chewing or not chewing, depending on the choice of threshold used to distinguish between $0~{}$A and $2I_{\rm unit}$. Figure 15 is the receiver operating curve (ROC) produced by varying this threshold current. The highlighted point on the ROC is a representative operating point, where the classifier produced a sensitivity of $0.91$ and a specificity of $0.96$. This corresponds to a false alarm rate of $(1-$specificity$)=0.039$.

In this section, we consider the impact of using the AFUA neural network in a complete eating event detection system. To process a $500$ Hz signal, the ZCR and RMS feature extraction blocks consume a total of $0.68~{}\mu$W [19]. Also, the AFUA network consumes $1.1~{}\mu$W, assuming $I_{\rm unit}=10~{}$nA. Finally, a microcontroller from the MSP430x series (Texas Instruments Inc., Dallas, TX) running at $1~{}$MHz consumes $180~{}\mu$W when active and $0.72~{}\mu$W when in standby mode [34].

The feature extraction and AFUA circuitry are always on, while the microcontroller remains in standby mode until a potential chewing event is detected. The fraction of time the microcontroller is in the active mode depends on how often the user eats, as well as the sensitivity and specificity of the AFUA network. Assuming the user spends $6\%$ of the day eating [35], then, using the classifier operating point highlighted in Fig. 15, the fraction of time that the microcontroller is active is

So, the microcontroller consumes an average of $180~{}\mu W\times 0.09+0.72~{}\mu W\times(1-0.09)=16.9~{}\mu$W. As Fig. 16 shows, the average power consumption of the complete AFUA-based eating detection system is $18.8~{}\mu$W. If we attempted to implement the system with a front-end ADC (12-bit, 500 Sa/s [6, 36]) followed by a digital LSTM [37, 38], then the ADC alone would consume over $240~{}\mu$W of power [39].

Table II compares our work to other recent eating detection solutions. The different approaches all yield generally the same level of classification accuracy, but our work differs in one critical aspect: while others depend on offline processing, or on tens of milliWatts of power to operate, our approach only requires an estimated $18.8~{}\mu$W.