Sequential Routing Framework:
Fully Capsule Network-based Speech Recognition

A CTC network [Graves et al., 2006] is defined as a continuous map $\mathcal{N}_{\theta}:(\mathbb{R}^{F})^{T}\mapsto(\mathbb{R}^{V})^{T}$ from an $F$ dimensional input sequence $x$ of length $T$ to the same length sequence $\hat{y}$ of $V$ dimensional probability vectors with parameters $\theta$. $V$ is the cardinality of an expanded label set $\mathbb{L}^{\prime}$ consisting of the union between the label symbol set $\mathbb{L}$ and a blank symbol, $\mathbb{L}\cup\{<$blank$>\}$. This network contains a softmax layer at the top of it in order to convert logits to valid probability distributions. The softmax layer for a given vector $Z\in\mathbb{R}^{D}$ is computed as follows:

, where $z_{i}$ is the $i$-th element of $Z$. CTC computes a conditional probability of a label sequence $y\in\mathbb{L}^{\leq T}$ for a given input sequence $x$ by summing up every conditional probability of possible paths $\pi$, i.e., $p(y|x)=\sum_{\pi\in\mathcal{B}^{-1}(y)}p(\pi|x)$. The possible paths are computed using an inverse of a map $\mathcal{B}:\mathbb{L}^{\prime T}\mapsto\mathbb{L}^{\leq T}$ from an expanded label sequence $y^{\prime}$ to $y$. $\mathcal{B}$ performs many-to-one mappings by simply removing repeating and blank symbols from the given paths. For example, $\mathcal{B}$(“cc-aaa-tt”) = $\mathcal{B}$(“-cc-aattt”) = “cat”, where “-” indicates a blank symbol. It allows CTC networks to learn alignments solely from the input and output sequence pairs. A conditional probability of each possible path is computed as $p(\pi|x)=\prod_{t=1}^{T}p(y^{\prime t}_{\pi}|x^{t})$, where $y^{\prime t}_{\pi}$ is an expanded label symbol in a path $\pi$ at time $t$ and $x^{t}$ is the $t$-th feature vector of $x$. The CTC loss is defined as a negative of the summation of the log probabilities of all CTC paths. The gradients are computed by differentiating the probability function $p(y|x)$ using the CTC forward-backward algorithm, which is a kind of dynamic programming.

A CapsNet [Hinton et al., 2011, Sabour et al., 2017, Hinton et al., 2018] for a two-dimensional data classification problem is defined with the parameter $\psi$ as a map, $\mathcal{C}_{\psi}:(\mathbb{R}^{X_{W}})^{X_{H}}\mapsto\mathbb{R}^{V}$, which maps an input data of size $X_{W}\times X_{H}$ to a $V$-dimensional probability vector, e.g., $X_{W}$ and $X_{H}$ represent the width and height of the two-dimensional input images respectively as shown in Fig 1. In the figure, the dimension above each capsule group block indicates the dimension of a group of instantiation parameter vectors. To initiate the routing method, the input is converted to a capsule group, $\mathsf{C}=\{\mathsf{A},\mathsf{U}\}$. In the capsulation process, $X_{W}$ and $X_{H}$ are converted to the width $P_{W}$ and the height $P_{H}$ of the lowest capsule group respectively. Thus, $\mathsf{C}$ consists of a pair of an activation group, $\mathsf{A}\in\mathbb{R}^{P_{W}\times P_{H}}$, and a group of instantiation parameter vectors, $\mathsf{U}\in\mathbb{R}^{P_{W}\times P_{H}\times P_{D}}$, where $P_{D}$ is the dimension of parameter vectors of capsules in the lowest level. $\mathsf{A}$ can be computed either from $\mathsf{U}$ [Sabour et al., 2017] or from additional neural layers [Hinton et al., 2018]. Routing iterations are performed between the higher-level capsules and the prediction vectors $\hat{u_{j|i}}$, each of which represents a path from the $i$-th lower-level capsule to the $j$-th higher-level capsule. Thus, in Fig 1, the range of $i$ and $j$ are $[1,P_{W}\times P_{H}]$ and $[1,V]$ respectively. $\hat{u_{j|i}}$ is computed by multiplying an instantiation parameter of the $i$-th lower-level capsule with a transformation matrix $W_{ij}$. During the training process, for a certain entity type, each of the elements of $\mathsf{A}$ and $\mathsf{U}$ can be trained to represent an existence probability and characteristics respectively [Hinton et al., 2011]. Accordingly, CapsNets can potentially [Lenssen et al., 2018] represent invariance of the existence probabilities and also equivariance of the properties of the entity type.

The groups in the lowest, highest, and in-between levels are called as primary, class, and convolutional capsule groups respectively. In this paper, for the sake of brevity, we describe the structure of $\mathsf{U}$ as the structure of the capsule group because the structure of $\mathsf{A}$ can be derived by removing the $D$ dimension from the structure of $\mathsf{U}$. The first and second dimensions of the class capsule groups are $V$ and $V_{D}$ respectively, and their activation groups are the outputs of the CapsNets, i.e., the CapsNets output $V$-dimensional vectors. Gradient based learning methods used for conventional neural networks are applicable. A key procedural difference of CapsNets is that they filter information flow using routing-by-agreement.

Capsulation is a neural layer block that converts from $x$ to $\mathsf{C}_{0}$, as shown in Fig. 3. $\mathsf{A}$ has the structure of width $T$ and height $P_{H}$ and the structure of $\mathsf{U}$ has the $P_{D}$ dimension in addition to the two dimensions of $\mathsf{A}$. The layers to compute $\mathsf{A}$ are optional structures depending on the routing mechanism. $x$ is first encoded into three-dimensional representation with width $T$, height $F$ and depth $F_{D}$ through $L_{C}$ two-dimensional convolutional layers, where $T\leq T^{\prime}$ and $F\leq F^{\prime}$. We employed maxout [Goodfellow et al., 2013] as activation functions of the convolutional layers because of their competitive accuracy in ASR [Zhang et al., 2016]. Thus, the $l_{c}$-th convolutional layer in the first sub-block is defined as follows:

, where $*$ is a convolutional operator, $N_{C}$ is the number of convolutional operations in a layer and $\alpha_{d}$ is the dropout rate. $W_{l_{c},n_{c},f_{d}}$ indicates a convolution weight matrix for the $f_{d}$-th channel of the $n_{c}$-th convolutional operation in the $l_{c}$-th layer. The $0$-th $x$ is an input sequence, i.e., $x_{0}\in\mathbb{R}^{F^{\prime}\times T^{\prime}}$. For all the cases, we set $N_{C}$ and $\alpha_{d}$ to 2 and 0.2 respectively. Subsequently, the output sequence is flattened to $x^{\prime}_{L_{C}}$ by reshaping it to width $T$ and height $F\times F_{d}$, then it is fed into two different layer blocks. They are projected to a normalized vector sequence, i.e., an activation group $\mathsf{A}$, with width $T$ and height $P_{H}$, through a nonlinear layer as follows:

, where $W_{\mathsf{A}}$ is a learnable weight matrix and $g$ is a nonlinear function for normalizing the values. They are also projected to another representation $\mathsf{U}^{\prime}$ having the same shape with $\mathsf{A}$ as follows:

, where $W_{\mathsf{U}}$ is a learnable weight matrix. Afterwards, $\mathsf{U}^{\prime}$ is converted to $\mathsf{U}$ by expanding their channel dimensions into $P_{D}$ through a two-dimensional convolutional layer activated with the maxout function as follows:

, where $W_{n_{c},p_{d}}$ indicates a convolutional weight matrix for the $p_{d}$-th output channel of the $n_{c}$-th convolutional operation.

In SRF, in order to adapt the existing routing methods with minimal changes and to control the required size of parameters regardless of the length of input sequences, each subgroup of a capsule group, $\mathsf{C}=\{\mathsf{A},\mathsf{U}\}$ is sliced by $T$ without overlapping adjacent slices, i.e., $\mathsf{C}^{t}=\{\mathsf{A}^{t},\mathsf{U}^{t}\}$. In order to expand a receptive field on $\mathsf{C}_{0}$, sequential routing is performed between windows of the time slices in the lower level, which consist of the consecutive $\omega$ slices, and single slices in the higher level. The bottom box in Fig. 2 describes an example of sequential routing between the window ($\omega=3$) centered on $t$-th slice in the lower level and the $t$-th slice in the higher level. In this study, we set the stride of the sliding window to one in all the cases and the both side of window contexts beyond the sequence boundaries are padded as zero. Thus when $\Lambda$ is set to 1, the routing iteration is performed $T$ times for a capsule group having width $T$. The receptive field on $\mathsf{C}_{0}$ is computed as $\omega+(L-1)\times(\omega-1)$ for each slice of $\mathsf{C}_{L}$. Accordingly, online decoding is allowed with a time delay corresponding to $L\times\omega_{R}$, where $\omega_{R}$ is the right context size of the window. In each capsule layer, the $t$-th prediction vector $\hat{u}_{j|i}^{t}$ is calculated from the $t$-th instantiation vector $u^{t}$ through a linear transformation using $W_{ij}$ as follows:

, where $I_{H}$ and $O_{H}$ are heights of lower and higher capsule groups respectively. $I_{H}$ of the first layer and $O_{H}$ of the last layer are the $P_{H}$ and $V$ respectively. The two heights are $M_{H}$ in-between layers. Accordingly, each $W_{ij}$ has the shape of the product of depths of lower and higher capsule groups. $W_{ij}$ are shared across all the time slices. Therefore, the number of parameters for the routing mechanism in a layer is controlled only by the shape of capsule groups and $\omega$.

We have an assumption that consecutive slices in sequence data have similar properties. Based on that, we designed an iterative routing mechanism which initializes routing coefficients for the $t$-th slice based on the routing output of the $(t-1)$-th slice. Fig. 4 is a schematic diagram of the two different iterative routing mechanisms of $L=2$ and $\omega=1$ for three consecutive slices. In this figure, the three dimensional block represents the routing procedures, $\lambda$ is an iteration index in the range between 1 and $\Lambda$, and the dashed and solid arrows describe the optional and required flows respectively. In the original version of routing, $r$ is initialized uniformly, then updated within each $t$ iteratively as in Fig. 4a in the order of a maximization and an expectation stage. The expectation stage means to update $r$ to improve the agreements between adjacent capsule groups and the updated coefficients are applied in the maximization stage. Thus, if $\Lambda$ is set to 1, output capsule groups are computed using the uniform $r$ in the original routing method. The proposed sequential routing method has two procedural differences as described in Fig. 4b. First, $r$ is initialized based on the agreement between the routing output of the previous time slice $\mathsf{C}_{2}^{t-1}$ and the current routing input $\mathsf{C}_{1}^{t}$ thus $r$ is uniformly initialized only at $t=1$. The second procedural difference is the sub-procedure order of the routing mechanism which expects the initial $r$ before the first maximization stage. These two modifications make a similar effect of updating $r$ $(t-1)$ times to compute $\mathsf{C}^{t}$ when $\Lambda=1$. In other words, they alleviate the need for iterative updates of $r$ within each slice as an option. Consequently, decoding can be performed in a non-iterative way while minimizing accuracy degradation.

Sequential version of DR (SDR) works as Algorithm 2 for each $t$ between the $l$-th and $(l+1)$-th level. To explain the algorithm, we use the notations $u$, $o$, and their indices $i$ and $j$ respectively as explained in Section 2.2.1. $o^{t-1}$ is zero-initialized at $t=1$. The expectation-maximization clustering is performed from line 5 to 11. At line 7, $r_{ij}$ is updated by accumulating the agreements between $\hat{u}_{j|i}^{t}$ and $o_{j}^{t}$. To maximize the agreements, $s_{j}$ is computed as the summation of $\hat{u}_{j|i}^{t}$ over all $i$ by weighting with the updated $c_{ij}$ at line 9. The number of operations of an iteration remains the same as Algorithm 1 since only the order of the sub-procedures is changed.

TIMIT [Garofolo et al., 1993b] consists of mono-channel read speech sampled at 16Khz. The training and test set consist of 4,620 utterances recorded from 462 speakers and 1,680 utterances recorded from 168 speakers respectively. We used a training set consisting of 3,696 utterances where all dialect utterances, i.e., the utterances tagged as “SA”, were removed and used 192 sentences from the core test set recorded from 24 speakers. A validation set was selected from another portion of the test set and was made up of 400 utterances recorded from 50 different speakers. A total of 63 labels consisting of 61 phonemes plus a padding and blank symbol were used during both training and decoding. At the top layer, the values in $c$ which route to a class capsule corresponding to the padding symbol are masked as zero. To evaluate phoneme error rates (PERs), the phoneme labels were mapped to 39 labels [Lee and Hon, 1989]. The features were extracted with a 10 ms hop size and 25 ms window size, and were encoded with 40-dimensional Fourier-transform-based filterbanks plus energy. Their temporal first and second-order differences were added with the delta-window size 2, thus 123-dimensional vectors were used as inputs. The input features were normalized to zero mean and unit variance per-speaker. The data splitting and feature extraction were performed using Kaldi²²2https://github.com/kaldi-asr/kaldi[Povey et al., 2011].

The learning schedule, $n_{w}$ is set to 1,200. We applied an additional decay policy where the learning rate was started by setting $\kappa$ to 0.5, and then $\kappa$ was reduced to 0.1 after 27 epochs. The models were trained for 200 epochs to ensure sufficient weight updates. In order to avoid the accuracy being dependent on the early stop time, we evaluated PERs with a model which is the averaged checkpoint of the last 10 epochs. Approximately 5K frames are contained in a batch for each training step according to their sequence length thus the learnable weight are updated 42K times per experiment. We first investigated the performance gain from the proposed routing algorithm using small CapsNet models with $L=1$ and $P_{H}=20$.

We compared SDR with DR as in Table 1. $\omega$ and the depth of capsule groups are set to 1 and 8 respectively. Accordingly, the total number of parameters of each model is 207,682. In each of the two cases, statistically significant differences in PERs depending on the number of iterations are not observed. However, in all the cases, SDR shows about 1% lower PERs than that of DR. These experiments will be further analyzed in Section 5 by investigating the heat maps of $c$.

We evaluated SRF models according to the configurations of $\omega$ as in Table 2. Their $\Lambda$ and capsule group depths are set to 1 and 8 respectively. As $\omega$ is expanded from 1 to 6, the numbers of required parameters (Params.) are increased from 0.21 to 0.66 million (M) as explained in Section 3.2. With the consideration of algorithmic delay caused by $\omega_{R}$, all models are set to have $\omega_{L}$ either longer than or equal to $\omega_{R}$. The time delays were estimated by “hop size (10ms) $\times$ look-ahead frames + 12.5ms (the half of window size 25ms)”. The number of look-ahead frames includes 4 more frames in addition to the frames for the setting of $\omega_{R}$ due to computing the delta plus double-delta features. The relative PER reductions (RPERRs) are at most 17.2% and 15.2% for Valid and Test respectively depending on $\omega$. End-of-sentence (EOS) detection rates seem to be one of the reasons why the settings that have unbalanced contexts show worse PERs than settings with balanced contexts since the detection rates show a large range from about 26% to 100% for both sets according to the setting of $\omega$.

The relationship between the configuration of the window (X-axis) and PERs (Y-axis) are more explicitly described in Fig. 5. “2-1” indicates that $\omega_{L}$ and $\omega_{R}$ are 2 and 1 respectively. Blue triangles and orange circles indicate Valid and Test respectively in Fig. 5a. In the figure, the three evenly sliced cases which are “0-0”, “1-1” and “2-2”, show almost linear error reductions according to $\omega$ for the both sets. We can also see multiple cases where the balance of $\omega_{L}$ and $\omega_{R}$ has more of an effect on lowering the PERs than the number of parameters when $\omega>1$. These phenomena are described by the dashed arrows in the figure and the percentages beside the arrows are RPERRs between the balanced and unbalanced window configurations. The models “1-1” and “2-0” have the same receptive field, but the model “1-1” shows relatively 6.4% and 3.3% lower PERs for Valid and Test respectively. In addition, although the model “2-1” has wider receptive field compared to the model “1-1”, it only shows slightly lower PER (relatively 0.4%) on Test but relatively 3.1% higher PER on Valid. Even more, the “5-0” model shows relatively about 9% worse PERs than the “2-2” model on the both evaluation sets. As $\omega_{R}$ of models (X-axis) expands, the EOS detection rates (Y-axis) get higher as shown in Fig. 5b. The detection rates are the averages from the two evaluation sets. The models which have one longer $\omega_{L}$ which are “1-0”, “2-1” and “3-2”, show lower EOS detection rates than that of “0-0”, “1-1”, and “2-2” respectively. As the differences between both sides of the window context size get bigger, the PERs and EOS detection rates of the settings get worse, i.e., phone recognition rates (PRRs) and EOS detection rates of the settings show the relationships such as “5-0” $<$ “4-1” $<$ “3-2” with the same number of parameters.

We also evaluated PERs by doubling the capsule group depth continually from 2 to 16, as in Table 3. In these experiments, $\omega_{L}$ and $\omega_{R}$ are fixed to 1, and $\Lambda$ is set to 1. The required parameters are increased nearly proportional to the depth of capsules from 0.14M to 1.15M as explained in Section 3.2. The RPERRs between the depth 2 and 16 cases for Valid and Test are 22.6% and 21.5% respectively. The PER reduction for both sets is 2.9% when increasing the depth from 2 to 4. Meanwhile, when increasing the depth from 4 to 8, it decreases to less than 2.0%, even though the increase in the number of parameters is 4 times larger from 0.05M to 0.2M.

We compared the SRF models with other architectures as in Table 4. The referenced PERs were evaluated using LSTM[Graves et al., 2013] and CNN[Zhang et al., 2016]-based CTC networks. We also compared with Speech-Transformer [Dong et al., 2018]-based CTC networks which we implemented. The models have the following structures below.

- BLSTM-5L-250H* [Graves et al., 2013]: BLSTM(250, concat)$\times$5-FC(63)

- ULSTM-3L-421H* [Graves et al., 2013]: ULSTM(421)$\times$3-FC(63)

- CNN-10L-Maxout* [Zhang et al., 2016]: Conv(3$\times$5, 64)-MP(3$\times$1)-Conv(3$\times$5,

64)$\times$3-Conv(3$\times$5, 128)$\times$5-Conv(3$\times$5, 24)-FCMO(512)$\times$2-FCMO(63)

- TF-5/10/20L: CNNFE-FCPE(128)-TF(128, 4, 1024)$\times$L-FC(63)

- SRF-1/2/5/7L: CNNFE-FC(60)-Conv(3$\times$3, 8)-SDR(30, 8)$\times$(L-1)-SDR(63, 8)

* Estimated by considering the number of model parameters.

Each layer is defined as the list below.

- BLSTM(cell states, merge mode={concatenation (concat) or average (ave)}): a

BLSTM layer

- ULSTM(cell states): a unidirectional LSTM (ULSTM) layer

- Conv(filter size=frequency$\times$frame, output channels, stride=[frequency=1,

frame=1]): a two dimensional convolutional layer activated with maxout,

the number of feature maps is twice of the channels.

- CNN frontend (CNNFE): Conv(3$\times$3, 64, [2, 2]) $\times$ 2

- MP(pooling size=frequency$\times$frame): a max pooling layer

- FC(output dimension): a fully connected layer

- FCPE(output dimension): a fully connected layer + positional encoding

[Dong et al., 2018]

- FCMO(output dimension): a fully connected layer activated with maxout,

the number of units is twice of the dimensionality of output space.

- TF(embedding dimension, attention heads, inner layer size): a Transformer

encoder layer [Dong et al., 2018]

- SDR($O_{H}$, depth of the output capsule group, [$\omega_{L}$=1, $\omega_{R}$=1], $\Lambda$=1): a SDR layer

$\kappa$ is set to 0.13 and reduced to 0.04 under the same condition as in the cases of the CapsNets and $n_{w}$ is set to 1,000. There are approximately 15K frames in a batch according to the length of utterances. We set dropout rates for the inputs, attention heads, inner layers and residual connections to 0.3, 0.3, 0.4, and 0.4 respectively. Bigger penalties are added to non-diagonal elements in attention maps before applying the softmax function Eq. 1, depending on the distance $\delta$ from the diagonal of the maps in the form of $-\log(1+\delta\times\beta)$ as used in [Dong et al., 2018], where the scaling factor $\beta$ were set to 1.0. TF-5L contains a similar number of parameters as the biggest SRF model SRF-7L.

As the layers of SRF models are stacked up, the receptive fields are increased from 31 to 79 frames. The reason why SRF-1L requires 0.02M more parameters than SRF-2L is that SRF-1L directly connects primary capsule groups to class capsule groups thus SRF-1L needs 11,340 (60 $\times$ 63 $\times$ 3) transformation matrices while SRF-2L needs 11,070 ((60 $\times$ 30 + 30 $\times$ 63) $\times$ 3). When comparing the two models, the number of layers seems to be more effective for reducing PERs than the number of parameters. Among the CTC networks except for SRF models, CNN-10L-Maxout [Zhang et al., 2016] shows the lowest PERs. Compared to CNN-10L-Maxout, although SRF-5L and SRF-7L require more delays, SRF-5L shows similar PERs with 63.3% fewer parameters. Moreover, SRF-7L achieves PERs of 15.9% and 17.5% for Valid and Test respectively, which are about 0.7% lower in PERs for both sets than that of CNN-10L-Maxout, with less than half of the parameters.

We measured the training and decoding time of TF-5/10/20L and SRF-1/2/5/7L on a NVIDIA^TM RTX-3090 as in Table 5. The training time is in seconds to finish one epoch of training. The size of the train-batch is set to 10K frames so that gradients were updated 107 times. We evaluated the decoding time in seconds and real-time factors (xRT) with Test. SRF-7L requires 18.6 times longer training time compared to TF-5L which has the same number of parameters. The training time of SRF models is increased as the size of models get bigger. On the other hand, decoding time is increased as more layers are stacked up. The correlation coefficients ($\mathsf{r}$) between SRF-7L and other models are larger than 0.90 in all the cases.

The Wall Street Journal (WSJ) corpus [Garofolo et al., 1993a, Consortium and Group, 1994] contains mono-channel read speech sampled at 16Khz. The si284 data set, which consist of about 81-hours of transcribed audio data (37,416 utterances), was set for training. The dev-93 (503 utterances, 1.1 hours) and eval-92 (333 utterances, 0.7 hours) data set were used respectively to validate and evaluate the models. For both training and evaluations, a total of 32 labels which consist of 26 uppercase letters, noise marker, apostrophe, space, EOS symbol plus a padding and blank symbol was used. The feature extraction and training configuration are the same as used in Section 4.1 unless it is specified. When it comes to the learning schedule, $n_{w}$ was set to 25,000. $\kappa$ of each model was individually set according to their valid loss curves and was halved at the 71st epoch out of 80 epochs. There were approximately 20K frames in a batch thus the weights were updated 110K times. Word error rates (WERs) were evaluated by averaging the last 5% of checkpoints (4 checkpoints). As in Table 6, the SRF models are compared with other CTC-based ASR systems which we implemented. The number of parameters of the compared models are set to 21.1M. The models in Table 6 were constructed as the list below:

- BLSTM-5L: CNNFE-BLSTM(441, ave)$\times$5-FC(32)

- ULSTM-5L: CNNFE-ULSTM(662)$\times$5-FC(32)

- CNN-10L: CNNFE-Conv(3$\times$5,140)$\times$4-Conv(3$\times$5, 300)$\times$5-Conv(3$\times$5, 66)-

FCMO(1024)$\times$2-FCMO(32)

- CNN-15L: CNNFE-Conv(3$\times$5, 100)$\times$4-Conv(3$\times$5, 215)$\times$10-Conv(3$\times$5, 66)-

FCMO(1024)$\times$2-FCMO(32)

- TF-20L: CNNFE-FCPE(256)-TF(256, 4, 1488)$\times$20-FC(32)

- SRF-7/10-Small: CNNFE-FC(52)-Conv(3$\times$3, 16)-SDR(26, 16, [2, 2])$\times$(L-1)-

SDR(32, 16, [2, 2])

- SRF-7/10-Big: CNNFE-FC(60)-Conv(3$\times$3, 20)-SDR(30, 20, [2, 2])$\times$(L-1)-

SDR(32, 20, [2, 2])

The dropout rates of both LSTM networks are set to 0.3 and 0.4 for input data and in-between layers respectively. The dropout rates are set to 0.2 between layers for both CNN models. The layer normalization [Ba et al., 2016] layers are added after every layer in both the LSTM and CNN-based CTC networks. The learning rates for BLSTM-5L and ULSTM-5L were set to 1e-4 and 5e-5 respectively without learning rate scheduling Eq. 9. The initial $\kappa$ of other models was set to 0.6. For TF-20L and SRF-7L-Big, it decreased to 0.06 and 0.1, respectively. Besides the two models, $\kappa$ was initially decreased to 0.5 and then to 0.1. SRF-10L-Big attains the lowest WER at 16.9% for eval-92 which is 1.1% better than that of BLSTM-5L. SRF-7L-Small requires almost half of the delay compared to CNN-15L and shows 7.3% and 7.1% lower WERs in dev-93 and eval-92 respectively compared to ULSTM-5L. SRF-10L-Small shows lower WERs with 68.0% of the parameters compared to SRF-7L-Big but it requires a 240 ms longer delay. Compared to TF-20L, relative WER reductions (RWERRs) of SRF-10L-Small are 7.9% and 13.9% for dev-93 and eval-92 respectively with half of the parameters.

As in Table 7, we evaluated the training and decoding time of the models explained in Table 6 on a NVIDIA^TM RTX-3090. The size of train-batch was set to 7K frames thus gradient updates were performed 4,155 times for an epoch of training. The decoding time was measured using eval-92. To finish an epoch of training, SRF-10L-Big takes 59 times longer than TF-20L, which has the shortest training time, and 37 times longer than BLSTM-5L, while showing similar word recognition accuracy. As the parameters and layers of the SRF models increase, their training and decoding time increase respectively. $\mathsf{r}$ between SRF-10L-Big and other models is at least 0.91.