ReFormer: Generating Radio Fakes for Data Augmentation

Our dataset comprises of datapoints which represent sampled RF signals, of the types used in modern wireless communications. Each datapoint arises from a specifically modulated sequence of random information bits, converted to a baseband RF signal. The datapoint $x$ can be represented as $x=\left[Re_{i}+jIm_{i}\right],i=1\cdots p$ with $j=\sqrt{-1}$. Here, a modulated signal $u$ is obtained as $u=M_{s}(b)$, where $s\in\mathcal{S}$ is the employed modulation scheme, with $S$ denoting the finite set of available digital modulation schemes. In this paper,

For any $s$, $M_{s}=\left\{0,1\right\}^{m}\rightarrow\mathcal{C}^{n}$ describes the modulation function of modulation class $s$. The random sequence of bits $b=\left\{0,1\right\}^{m}$ of length $m$ is encoded into a sequence of complex valued numbers of length $n,$ where the complex sample $c_{i}=Re_{i}+jIm_{i},1\leq i\leq n$ encodes the modulation phase $\phi=\arctan{Re_{i}/Im_{i}},$ and amplitude $a_{i}=\sqrt{Re_{i}^{2}+Im_{i}^{2}}.$ We create datapoints as sub-sequences $x$ of $u\in\mathcal{C}_{n},$ of length $p=1024.$ We prepared the training dataset $X_{train}$ by using an open-source library torchsig featured in [7]. $X_{train}$ contains RF samples of high SNR, for both simplicity and future utility in creating various signal augmentations controlled by a prompt. The torchsig library function ComplexTo2D is used to transform vectors of complex-valued numbers into 2-channel datapoints, traditionally referred to as I and Q components. Each channel is comprised of $p$ real numbers, previously normalized. Channel 1 contains real components $I\in\mathbb{R}^{p}$ and channel 2 contains the imaginary ones $Q\in\mathbb{R}^{p}.$ Depending on the modulation, $x$ contains more or less mappings of the original random sequence of bits $b.$ $\mathcal{S}$ contains diverse modulation orders (how many original bits are represented by a single complex value). By using $p>1024,$ we may be able to better train the proposed generative model. However, this would require more complex neural net architecture and longer training. Future work will explore the effect of $p$ on the quality of the generated RF fakes.

Prior work [8, 9] used a hierarchical VQVAE model to learn several posterior distributions, including $P(Z_{q}|x)$ in its first level of hierarchy, where $Z_{q}$ is the vector-quantized version of the latent $Z_{e}$ mapped from the RF datapoint $x$ by the trained encoder. Now we want to learn the prior $P(x),$ which will allow us to sample $P(x)$ and generate similar RF fakes. However, to decrease the training complexity of the model which parameterizes this distribution, we choose to leverage $P(Z_{q}|x)$ for mapping $X_{train}$ to a discrete space and than learn the prior of $Z_{q}.$ Generating the fakes in the discrete latent domain is easier, and we can directly leverage the power of the transformer architecture [10]. Once we obtain a fake $\hat{Z}_{Q},$ we will use the VQVAE decoder to map it back to the original space, resulting in a fake RF datapoint $x_{g}.$

Using a Vector Quantized Variational Autoencoder model [11] to learn an efficient, discrete, low-dimensional representation $Z_{Q}$ of every RF datapoint $x,$ allows us to create a discrete mapping of $X_{train}.$ The dataset $ZD$ is mapped from the training dataset $X_{train},$ using the mapping $\Phi_{Q}(E_{\Theta}()):X_{train}\rightarrow ZD,$ where $E_{\Theta}(x),x\in X_{train}$ is the trained VQVAE encoder with parameters $\Theta,$ and $\Phi_{Q}(Z)$ is the vector quantizer based on the trained codebook $Q.$ Equations (1) represent the trained VQVAE model utilized for the above mapping.

Subsection II-D will show how we learn the prior $P(Z_{Q})$ of the discrete latent sequences generated from $X_{train}$: we learn $P(Z_{Q})$ probability model by training the transformer to learn its autoregressive form $P(Z^{i}_{Q}|Z^{1\cdots(i-1)}_{Q})$. Our VQVAE model has three components: (1) an encoder block $E:\mathbb{R}^{2\times 1024}\xrightarrow{}\mathbb{R}^{\ell\times d_{s}}$ which down-samples the input by a factor of 2 in each convolutional channel dimension and produces an output $z$ such that the number of channels of the output is equal to the codebook dimension $\ell=64$; we refer to each $z$ column-partition of size $\left(64\times 1\right)$ as a slice $z_{i},$ resulting in $d_{s}=512$ slices (2) a decoder block $D:\mathbb{R}^{\ell\times d_{s}}\xrightarrow{}\mathbb{R}^{2\times 1024}.$ (3) a vector quantizer $\Phi$, which is applied to the output of $E$ amd provides an output to be decoded by $D.$

Vector quantization (VQ) is a process which discretizes the latent space (output of $E$). Each latent slice is discretized by applying the quantization function $\Phi$, which maps it to an element of $Q=\{e_{i}\mid 0\leq i<N\},$ consisting of $N$ codewords, the learnable embedding vectors of the same length ${\ell}$. VQ is effectively mapping each slice to an index of the ordered set of codewords, referred to as the codebook $Q.$ The VQ-VAE model aims to learn the optimal codebook vectors through the training process. The decoder block, denoted $D(z_{q})$, is tasked with reconstructing the original input $x$ from the mapped codebook vectors. While the most intuitive approach for $\Phi$ is to simply perform a nearest neighbor lookup, more advanced techniques such as stochastic codeword selection $\Phi_{SC}$ [12] or heuristics such as exponential moving averages (EMA) $\Phi_{EMA}$ [13] can be employed instead. We here present the results obtained using stochastic mapping $\Phi_{SC}.$ In the equations (1), $z$ is the latent representation of $x$ at the output of $E$ while $z_{q}$ is its quantized version.

We are interested in another form of the quantized latent $z_{q}$, denoted in Fig.1 by $Z_{Q},$ which replaces the codewords quantizing the $z$ slices with their indices in the codebook. We refer to those indices as tokens $Z_{Q}^{j}\in\left\{0,\cdots N-1\right\},N=\left|Q\right|=128,$ where $j$ is the slice indices $j\in\left\{1,\cdots,d_{s}\right\}.$

To learn the prior in the discrete space, we train a decoder-only transformer (DoT) on the dataset $ZD,$ mapped from the training dataset $X_{train},$ using the mapping $\Phi_{Q}(E_{\Theta}()):X_{train}\rightarrow ZD.$

The DoT model [10], also known as the autoregressive transformer, is a popular generative model derived from the original encoder-decoder transformer model [16]. The encoder stack is eliminated from this transformer, leaving the decoder stack to learn the causal attention structure in $Z_{Q}.$ Once trained, the model can perform autoregressive sequence generation. The self-attention mechanism enables the model to predict the next token given all previous tokens. DoT can also include cross-attention, allowing for different creative methods of obtaining the cross-attending context. The training relies on the cross-entropy loss (CE) between the masked indexed output of the VQ-VAE quantizer $Z_{Q}$ and its estimate $\hat{Z}_{Q}$ at the output of the transformer (see Fig. 1). The latent fake generation, is an auto-regressive inference. It starts from a random token, and continues generating subsequent tokens of a fake $\hat{Z}_{Q}$ auto-regressively. We trained the transformer using a simple form of cross-attending context by preceding each datapoint $Z_{Q}\in ZD$ with its class token $C\in\left\{0,\cdots,5\right\}.$ Hence, during the inference, to indicate what class of the latent fake we aim to generate, we start the autoregressive generation with the token $C.$ Once the latent fake $\hat{Z}_{Q}$ is generated, we map it into its non-indexed version $\hat{z_{q}}$ and transform it by the VQ-VAE decoder into a RF fake.

To train the transformer model, we used two different architectures: the nano-GPT architecture [17] and the MONAI transformer [18]. Both models were trained to learn the autoregressive representation of the discrete latent space $Z_{Q}$, which was generated by the VQ-VAE quantization process.

The discrete latent sequences $Z_{Q}$ contain $d_{s}$ tokens, where each token represents an index of the nearest codeword from the trained VQ-VAE codebook. To prepare the training data for the transformer, we constructed $ZD_{\text{X}}$ and $ZD_{\text{Y}}$ as follows:

• •

  $ZD_{\text{X}}$ consists of $d_{s}$ tokens, where the first token is a class label $C\in\{0,1,\dots,5\}$ representing the modulation scheme. The remaining $d_{s}-1$ tokens are indices derived from the quantized $Z_{Q}$ sequence.

• •

  $ZD_{\text{Y}}$ consists of $d_{s}-1$ tokens, representing the indices from the second token onward in the $Z_{Q}$ sequence.

The VQVAE model was evaluated using a variety of reconstruction metrics. Most importantly, we analyzed the codebook usage for each class (Figure 6). This helps identify potential mode collapse, which would result in skewed histograms of codebook usage for one or more classes. Ensuring uniform codebook usage across classes validates proper training of the VQVAE and indicates robustness in the learned representations. Results of these evaluations are discussed in subsequent sections. Additionally, we created 10 VQVAE reconstructions for each class and visualized the constellation diagrams for the I/Q samples. These diagrams (Fig. 3) were compared to the original class constellations (Fig. 2) to assess the fidelity of reconstructions visually.

For evaluating the fidelity and diversity of the generated fakes, we adopted the metrics described in [19], specifically Topological Precision and Recall (TopP&R). These metrics are robust and reliable for evaluating generative models, offering statistical consistency under noise and perturbations.

Similarly like with VQVAE reconstructions, the perceptual method of evaluation involves plotting the I/Q constellations for each class and visually comparing them against the original constellations (Figures IV-D, IV-D). This qualitative analysis provides an intuitive understanding of how well the reconstructions preserve class-specific signal characteristics.

The fidelity, diversity, and Top-F1 metrics, computed for both transformers, are summarized in Table I.

Both transformers achieved a fidelity score of 1.0, indicating that the generated samples closely resemble the real data. However, the nano-GPT transformer outperformed the MONAI transformer in both diversity (0.8455 vs. 0.6909) and Top-F1 (0.9163 vs. 0.8172), demonstrating its ability to produce a more diverse range of samples while maintaining high accuracy.

To further validate the quality of the generated RF fakes, a pretrained classifier, initially trained on the original dataset, was tested on various datasets: the original test data, VQVAE reconstructions of the test data, and fakes generated by both the MONAI and nano-GPT transformers. The test data consisted of 500 samples per modulation class. Table II summarizes these results.

The classification accuracy metrics provide insight into the practical utility of the generated datasets. Both the original and reconstructed test data achieved a perfect classification accuracy of 100%, demonstrating that the VQVAE model effectively captures and retains the essential characteristics of the RF signals, such that the classifier remains highly accurate.

However, there is a notable difference in the performance of the fakes: the nano-GPT transformer produced fakes with a classification accuracy of 81.80%, significantly higher than that of the MONAI transformer, which only achieved 44.07%. This disparity underscores the nano-GPT transformer’s superior capability to generate more realistic and complex RF signal fakes which maintain salient characteristics.

The performance loss in the MONAI-generated fakes suggests potential issues in capturing the finer details of the signal characteristics, or the overfitting to less generalizable features during the training phase (see the conclusions).

To further assess the performance, we visualized:

• •

  Reconstructions: Figures IV-D and  IV-D illustrate a random single-fake-based I/Q constellation for both the MONAI and GPT transformers, respectively. This includes I/Q constellations for all six modulation classes. Both generative models preserve the class-specific characteristics of the signals. The nano-GPT based fakes produced constellation diagrams that closely resemble the original, indicating effective signal learning.

• •

  Codebook Usage: We analyzed the codebook usage across all signals for the both transformers and the VQVAE latents. The codebook usage histograms for all six classes were combined into a single figure, revealing uniform usage across all codewords, thereby validating the proper training of the VQVAE.

Figure 7 shows the codebook usage histograms for all six modulation classes generated by the GPT transformer. Similarly, Figure V presents the codebook usage for the MONAI transformer. Each figure contains six subplots, one for each modulation class. The uniform usage of codewords across all classes indicates that the VQVAE was properly trained and did not suffer from mode collapse.

The results indicate that both transformers are capable of generating high-fidelity RF signal fakes. While the MONAI transformer achieves satisfactory performance, the GPT transformer demonstrates superior diversity and accuracy, making it a more robust choice for generating a wide range of RF signal variations. The qualitative evaluations further support these findings, showing consistent reconstruction quality and uniform codebook usage. The fact the loss curves for the MONAI training start diverging after 100 epochs, indicate that this model is at the edge of overfitting, possibly because its number of parameters is 10 times the number of nano-GPT parameters. This may be the cause of the slightly inferior performance, both in terms of qualitative (Figures  and ) and quantitative indicators (Table I).