Rate Compatible LDPC Neural Decoding Network: A Multi-Task Learning Approach
Abstract
Deep learning based decoding networks have shown significant improvement in decoding LDPC codes, but the neural decoders are limited by rate-matching operations such as puncturing or extending, thus needing to train multiple decoders with different code rates for a variety of channel conditions. In this correspondence, we propose a Multi-Task Learning based rate-compatible LDPC decoding network, which utilizes the structure of raptor-like LDPC codes and can deal with multiple code rates. In the proposed network, different portions of parameters are activated to deal with distinct code rates, which leads to parameter sharing among tasks. Numerical experiments demonstrate the effectiveness of the proposed method. Training the specially designed network under multiple code rates makes the decoder compatible with multiple code rates without sacrificing frame error rate performance.
Index Terms:
Rate-compatible LDPC codes, 5G, parameter-sharing, neural LDPC decoder.I Introduction
LOW-DENSITY parity-check (LDPC) codes were rediscovered in 1996[1] and are still a favored research direction in the field of error correction codes[2, 3, 4].The fixed code rate of conventional capacity-approaching LDPC codes restricts their applications, especially when the system demands flexible code rates e.g. hybrid automatic repeat request (HARQ) mechanisms. In the 5G New Radio (NR), rate-compatible LDPC codes with a nested structure[5, 6] are adopted as the coding scheme for data channels in the 5G enhanced mobile broadband (eMBB)scenario[7].
The convergence rate of the iterative message-passing algorithms such as the belief-propagation (BP) algorithm and min-sum (MS) algorithm gets slower due to the short cycles in the parity-check matrix of LDPC codes. Modified algorithms (e.g. Normalized-BP, Offset-BP) that lower the confidence of the node information can migrate the effect of short cycles[8]. However, these modified algorithms have some parameters for normalization or offset, which are usually determined empirically. It has been shown that the empirically set parameters fail to achieve optimum performance under different codewords and channel conditions.
In recent years, in view of the fact that deep learning (DL) has shown remarkable improvement in various tasks, researchers combine the powerful learning capabilities of DL with traditional LDPC decoding algorithms to reduce computing complexity. In [9], Nachmani et al. introduce learnable weights into the BP decoding algorithm, effectively mitigating the negative impact of short cycles. However, a large number of nonlinear and multiplicative operations in the BP algorithm lead to the high computational complexity of the unrolled neural decoder. MS-based decoding networks (e.g NNMS and NOMS) are then proposed to further reduce the computational complexity and improve the hardware adaptability[10, 11]. To deal with long LDPC code and the gradient vanishing issue, a parameter-sharing mechanism is introduced in [12] to reduce the number of parameters of the neural decoder for PB-LDPC.
DL-based decoding methods assume some fixed code rate. In the 5G NR, flexible LDPC code rates are supported by exploiting rate-matching operations such as puncturing and parity-check matrix extension[13]. Although one could train multiple networks suited to a range of code rates that would then be selected during the decoding process, this would require considerable storage, which restricts its applicability in resource-constrained applications such as the internet of things (IoT). Therefore, it is desired to develop new neural decoding networks capable of adapting to varying LDPC code rates.
In this correspondence, we design a new rate-compatible (RC) decoding network based on Multi-Task Learning (MTL). Utilizing the extension structure, the proposed RC decoder can adaptively adjust the network structure according to the code rate. The parameter-sharing mechanism allows one set of parameters to cope with multiple code rates thus effectively reducing storage cost and training complexity.
II Preliminaries
II-A Conventional BP/MS for LDPC Decoding
According to the parity-check matrix , the Tanner graph defines two kinds of nodes: variable nodes (VNs) and check nodes (CNs). Each VN corresponds to a bit in codeword (a column of ) and, each CN corresponds to a parity-check equation (a row of ). An edge connects a VN and a CN corresponds to a “1” in .
The conventional BP algorithm and MS algorithm use log-likelihood ratio (LLR) as messages passing through the edges. Let and denote a received noisy codeword and its corresponding pure codeword, respectively.
The LLR value of the -th bit is defined as
(1) |
where the denotes the transition probability from to . Let represent the edge connecting the -th VN and the -th CN. For the -th iteration, the CN to VN (C2V) message passing on the edge and the VN to CN (V2C) message passing on the edge are defined as and , respectively. The V2C messages is calculated using
(2) |
where denotes the set of edges which are adjacent to the -th VN, and denotes the edge set excluding the edge adjacent to the -th CN. The C2V message is calculated using
(3) |
where denotes the set of edges which are adjacent to the -th CN, and denotes the edge set excluding the edge adjacent to the -th VN. At the first iteration, is initialized to 0 in Eq. (2). For the -th iterations, the output is the LLR value of , which is defined as
(4) |
then the -th decoded bit can then be obtained using
(5) |
It is worth noting that in the MS algorithm, Eq. (3) will be rewritten as
(6) |
In the decoding process, the V2C message is calculated first, then the C2V information is calculated. When the break conditions are met, the algorithm stops iterative calculation and outputs the final decoding result.
II-B Rate-Compatible (RC) Raptor-Like LDPC Code
In the 5G NR, raptor-like LDPC (RL-LDPC) codes are employed. The parity-check matrix of a RL-LDPC code can be decomposed to a precode submatrix and multiple incremental redundancy code (IRC) submatrices [14]. The precode submatrix defines a low-threshold high-rate precode, and a lower-rate parity check matrix is obtained by extending the precode submatrix using IRC submatrices, which means adding the same number of columns (VNs) and rows (CNs) to the precode matrix.
The connections between CNs in the IRC submatrix and VNs, including a degree-1 VN and several precoded VNs, allow efficient encoding of the incremental redundancy. The adaptive rate of RL-LDPC codes is achieved by performing a varying number exclusive-or operations on the precoded symbols to generate different numbers of parity-check bits, and high-rate codewords are nested in the low-rate codewords.
II-C Multi-Task Learning
MTL trains a model that fits multiple related tasks. The model exhibits better generalization and performance on each task, and reduces the risk of overfitting under a training set including data of different tasks. To reduce storage costs and improve prediction accuracy, sharing of partial network structures as well as weights is applied in MTL applications[15].
III Proposed Rate Compatible Decoding Network
In this section, we present details of the proposed RC decoding network. The existing LDPC neural decoders introduced in the previous section are limited to a predefined code rate, while the BP and MS decoding methods are not restricted to the code rate. Here, we propose a rate-compatible LDPC neural decoder using a special parameter-sharing mechanism. We utilize the structure of the raptor-like LDPC codes and construct an adjustable dynamic neural decoding network.
III-A Rate Compatible Model-Driven Decoding Network
III-A1 Sturcture of the neural decoder
The structure of our decoding network is shown in Fig 1, which is constructed by unrolling the iterative decoding algorithm into a non-fully connected neural network. Each iteration in the conventional decoding algorithm is unrolled into a VN-sublayer and a CN-sublayer. Since the messages are propagated along the edges, each edge becomes a neuron in the sublayers. The input layer is of size , corresponding to the code length. Each hidden layer is of size (including all VN-sublayers and CN-sublayers), where denotes the number of edges in the Tanner graph. The size of the output layer is also . To support different code rates, all layers adjust the number of neurons activated according to the code rate of the received code word, and thus the sharing of neurons for different tasks, i.e., code rates, leads to advantages brought by the MTL.
III-A2 Message flow
The message flow of the decoding network is shown as follows. For a received noisy codeword , the input layer computes its LLR value and propagates this value to each VN-sublayer and output layer. For the -th iteration, the V2C messages are calculated using Eq. (2), where are all set to be 0 at the initialization step.
We add weights and biases to the C2V message passing on the edge in the -th CN-layer, which leads to:
(7) |
If setting weights and biases to 1 and 0, respectively, the decoding network is equivalent to the conventional BP decoder.
The output of the network is the channel information corrected by the refined C2V messages, the equation is
(8) |
where denotes the sigmoid activation function that maps the output into the range of [0,1]. With a hard decision on the output, we can get the predicted codeword .
It should be noted that if one unfolds the neural network based on the MS algorithm rather than the BP algorithm in the same way, then Eq. (7) should be rewritten as
(9) |
where denotes the popular ReLU activation function. This design was proposed in [12]. There are other neural LDPC decoding designs [16, 17, 10] that employ different check node and variable node update equations. Note that the proposed rate-compatible mechanism is not coupled with the neural LDPC update designs, and can be applied to these different neural LDPC decoders.
III-A3 Rate compatible parameter sharing mechanism
Since the network needs to support multiple different code rates, the hidden layer needs to adjust the width according to different code rates. Thus we propose a parameter-sharing mechanism, as shown in Fig. 2, where the code rate controls the activation status of neurons in each sublayer. Owing to the raptor-like structure, the parameters of a high code rate are reused in the decoding process under a low code rate.
The parameter matrix of the decoding network is a hierarchical cascade of the weights and biases of each layer, where denotes the set of real numbers. The -th (even) rows and the -th (odd) rows of the parameter matrix, where , serves as the weights and biases vector of the -th layer, respectively. As high-rate codewords are nested in the low-rate codewords, the whole network including all parameters is used for decoding codewords with the lowest rate, while only part of the network is activated for decoding codewords of the higher rate. The edge set and used in Eq. (2),(7)-(9) are rewritten as and , denoting the set of activated edges that are adjacent to the -th CN and -th VN under the code rate , respectively. This is the key of the proposed parameter-sharing mechanism.
Assuming the decoding network supports different code rates, the parameter matrix can be decomposed into submatrices, i.e., , where the parameter matrix is used for decoding codewords of the highest code rate, and are used for decoding the codewords of the 2ed highest code rate, and are used for decoding the codewords of the -th highest code rate.
We consider the decoding tasks at different code rates as related tasks and employ MTL to train a decoding model that can adapt to multiple rates. Specifically, by employing the proposed parameter-sharing mechanism, same parameters can cope with multiple code rates, which leads to the reduction of storage cost and training complexity compared to training multiple networks for different code rates.
The proposed scheme can be also seen as the result of another strategy for multiple rates, where the incoming LLRs are loaded into an LDPC coder with the lowest rate, with the punctured bits filled as zero LLRs. In this way, the punctured degree-one VNs only connect to a single CN. During the iterative decoding, the V2C messages of these VNs are generated solely from the received channel information. When filled with zero LLR, the C2V messages from the connected CN to other VNs all can be computed into zeros. In our proposed scheme, the corresponding neural nodes are inactivate, which is equivalent to C2V messages with zero LLRs.
III-B Training Method
We use binary cross-entropy between the predict codeword and the transmitted codeword as the loss function, which is written as
(10) |
where and represent the -th bit of the decoded message and the transmitted codeword, respectively. To learn parameters in a deep network, we adopt the layer-by-layer greedy training method.
In the MTL framework, the parameters involved in multiple decoding tasks are trained on various datasets corresponding to different code rates. Specifically, parameters used for decoding codewords of some code rate would be learned from all data with code rates no higher than that rate, e.g. the optimize step of can be written as where denotes the decoding loss of the -th highest code rate.
IV Performance evaluation
In this section, we construct neural RC decoders for the 5G NR LDPC codes and evaluate the frame error rate (FER) performance under the BPSK/QPSK modulation and the additive white Gaussian noise (AWGN) channel at different signal-to-noise ratios (SNRs). The experiments are conducted on the TensorFlow platform and GTX1050 GPU. Note that the proposed decoders do not have to run in GPUs, although the use of GPU significantly reduces the training time.
We employ the base code BG1 defined in 5G standard[7] with lifting factor =16 for BP-based decoders and MS-based decoders. The training set employs mixed codewords that are randomly generated under multiple SNRs ranging from 0dB to 6dB. Additionally, these codewords are generated with various code rates randomly selected from [0.2451, 0.3701, 0.5137], and these code rates are chosen from the 5G NR modulation and coding scheme (MCS) table for physical downlink shared channel (PDSCH) [18].
The neural RC decoders have 20 hidden layers, which correspond to 20 full iterations of the BP/MS algorithm. For each update, the parameters are trained over 10000 batches, with 300 samples in each batch. We applied adaptive moment estimation (Adam) optimizer with an initial learning rate equal to 0.0001. To stabilize the operation of the learned BP decoder, the absolute value of the messages transferred between the neurons is clipped to 20[9].
IV-A Simulation Results
To evaluate the performance of the proposed RC neural normalized BP (RC-NNBP) decoder and RC neural normalized MS (RC-NNMS) decoder, we conduct comparisons against the conventional iterative decoding algorithms (with 20 iterations) and the neural decoding networks trained under each code rate (with 20 hidden layers). For the BP-based decoders, the standard BP algorithm and neural normalized BP (NNBP) decoders are selected as competitors. For the MS-based decoders, the conventional normalized MS (CNMS) algorithm with weights 0.8 [19], and the neural normalized MS (NNMS) decoders are considered in the comparison. Note that early termination can be implemented in these neural decoders by examining the decoded bits after each iteration. For a fair comparison, we performed 20 iterations for all decoders.
IV-A1 BP-based Decoders
The FER results of (160, 653), (160, 430), and (160, 311) BG2 codes are given in Fig. 3, whose code rates are 0.2451, 0.3701 and 0.5137, respectively. We observe that neural BP-based decoders outperform traditional BP algorithm. The proposed RC-NNBP decoder exhibits similar FER performance to NNBP decoders trained at different code rates, which in practice require multiple networks and result in greater storage costs.
IV-A2 MS-based Decoders
The FER results of MS-based decoders are given in Fig. 4. We observe that the neural MS-based decoders exhibit superior performance compared to traditional algorithms, at the code rate of 0.2451, the neural decoder outperforms the CNMS algorithm by around 0.3dB. The proposed RC-NNOMS decoder also exhibits similar FER performance to NNOMS decoders, which further demonstrates that the proposed RC decoder can effectively replace multiple conventional neural decoders, thereby reducing storage costs. Understandably, a similar phenomenon can be seen with the BG1 codes.
Furthermore, the performance of the three MS-based decoders in terms of FER-iterations is presented in Fig. 5. The experiment was conducted using an SNR of 3dB and the (160, 430) BG2 code. Remarkably, the neural decoder demonstrates superior convergence compared to the conventional algorithm. When aiming for an FER of , the neural decoder requires 6 fewer iterations to achieve it.
IV-A3 Protograph-Based Rate Compatible (PBRC) Decoder
The protograph-based LDPC (PB-LDPC) codes are adopted in 5G NR. By exploiting the lifting structure of PB-LDPC codes, [12] proposed a method to deal with different lifting factors with one neural decoder, which uses the same parameter among a bundle of edges belonging to the same edge in the base graph. This approach in [12] can be integrated into our method to deal with possible lifting factors and further reduce storage cost and training complexity.
Figs. 6 show the FER results of the PBRC Decoders, where the code configurations are the same as that in Fig. 4. The results indicate that our rate-compatible mechanism still proving useful in the PB-LDPC decoders.
The high performance of our proposed neural decoder benefits from two aspects: the well-learned parameters mitigating the effects of short cycles in the parity-check matrix, which leads the decoder to a faster convergence; the other is that the rate-compatible mechanism accommodates the decoder to multiple code rates, thus achieve a better FER performance and a lower storage cost at the same time.
IV-B Complexity
Operation | tanh | + | comp. | sign flip | Storage | |
BP | - | - | - | |||
CNMS | - | 1 | ||||
NNOBP | - | - | ||||
RC-NNOBP | - | - | ||||
NNOMS | - | |||||
RC-NNOMS | - |
The required operation and storage costs in one iteration of different schemes are presented in Table I, where and represent the total activated numbers of the neurons under code rate 0.2451, 0.3701 and 0.5137, respectively. The neural decoders have similar processing procedure in each iteration as BP and CNMS. The proposed RC decoders, i.e., RC-NNOBP/RC-NNOMS, retain the same computing complexity as conventional neural decoders, i.e., NNOBP/NNOMS, but require less memory to store the parameters to deal with multiple code rates. According to the hardware implementation in [20], the weights and biases in the neural decoder can be loaded into the processing unit for each iteration, which enables the hardware to be reused in different iterations.
V CONCLUSIONS
In this correspondence, neural RC LDPC decoding networks that can deal with multiple code rates are proposed. The network exploits the structure of rate-compatible LDPC codes and dynamically activates partial neurons for different code rates during the decoding process. Experimental results show that neural decoding networks exhibit better FER performance than traditional decoding algorithms. The proposed neural RC decoder can replace multiple neural decoders without loss of FER performance.
References
- [1] D. J. MacKay and R. M. Neal, “Near shannon limit performance of low density parity check codes,” Electronics letters, vol. 33, no. 6, pp. 457–458, 1997.
- [2] O. Ferraz, S. Subramaniyan, R. Chinthala, J. Andrade, J. R. Cavallaro, S. K. Nandy, V. Silva, X. Zhang, M. Purnaprajna, and G. Falcao, “A survey on high-throughput non-binary ldpc decoders: Asic, fpga, and gpu architectures,” IEEE Communications Surveys & Tutorials, vol. 24, no. 1, pp. 524–556, 2022.
- [3] Q. Wang, S. Cai, W. Lin, S. Zhao, L. Chen, and X. Ma, “Spatially coupled ldpc codes via partial superposition and their application to harq,” IEEE Transactions on Vehicular Technology, vol. 70, no. 4, pp. 3493–3504, 2021.
- [4] C. Tarver, M. Tonnemacher, H. Chen, J. Zhang, and J. R. Cavallaro, “Gpu-based, ldpc decoding for 5g and beyond,” IEEE Open Journal of Circuits and Systems, vol. 2, pp. 278–290, 2021.
- [5] M. El-Khamy, J. Hou, and N. Bhushan, “Design of rate-compatible structured ldpc codes for hybrid arq applications,” IEEE Journal on Selected Areas in Communications, vol. 27, no. 6, pp. 965–973, 2009.
- [6] T. V. Nguyen and A. Nosratinia, “Rate-compatible short-length protograph ldpc codes,” IEEE Communications Letters, vol. 17, no. 5, pp. 948–951, 2013.
- [7] 5G; NR; Multiplexing and channel coding (3GPP TS 38.212 version 16.3.0 Release 16), 3rd Generation Partnership Project (3GPP), Nov. 2020.
- [8] J. Chen and M. Fossorier, “Density evolution for two improved bp-based decoding algorithms of ldpc codes,” IEEE Communications Letters, vol. 6, no. 5, pp. 208–210, 2002.
- [9] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2016, pp. 341–346.
- [10] L. Lugosch and W. J. Gross, “Neural offset min-sum decoding,” in 2017 IEEE International Symposium on Information Theory (ISIT), 2017, pp. 1361–1365.
- [11] A. Verma and R. Shrestha, “Low computational-complexity soms-algorithm and high-throughput decoder architecture for qc-ldpc codes,” IEEE Transactions on Vehicular Technology, pp. 1–14, 2022.
- [12] J. Dai, K. Tan, Z. Si, K. Niu, M. Chen, H. V. Poor, and S. Cui, “Learning to decode protograph ldpc codes,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 7, pp. 1983–1999, 2021.
- [13] S. Ahmadi, “Chapter 4 - new radio access physical layer aspects (part 2),” in 5G NR, S. Ahmadi, Ed. Academic Press, 2019, pp. 411–654. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B978008102267200021X
- [14] T.-Y. Chen, K. Vakilinia, D. Divsalar, and R. D. Wesel, “Protograph-based raptor-like ldpc codes,” IEEE Transactions on Communications, vol. 63, no. 5, pp. 1522–1532, 2015.
- [15] Y. Zhang and Q. Yang, “A survey on multi-task learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 12, pp. 5586–5609, 2022.
- [16] Q. Wang, S. Wang, H. Fang, L. Chen, L. Chen, and Y. Guo, “A model-driven deep learning method for normalized min-sum ldpc decoding,” in 2020 IEEE International Conference on Communications Workshops (ICC Workshops), 2020, pp. 1–6.
- [17] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, and Y. Be’ery, “Deep learning methods for improved decoding of linear codes,” IEEE Journal of Selected Topics in Signal Processing, vol. 12, no. 1, pp. 119–131, 2018.
- [18] 5G; NR; Physical layer procedures for data (3GPP TS 38.214 version 16.2.0 Release 16), 3rd Generation Partnership Project (3GPP), Jul. 2020.
- [19] J. Chen, A. Dholakia, E. Eleftheriou, M. Fossorier, and X.-Y. Hu, “Reduced-complexity decoding of ldpc codes,” IEEE Transactions on Communications, vol. 53, no. 8, pp. 1288–1299, 2005.
- [20] B. Zhang, Y. Sui, L. Huang, S. Liao, C. Deng, and B. Yuan, “Algorithm and hardware co-design for deep learning-powered channel decoder: A case study,” in 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD), 2021, pp. 1–6.