Generalizable Audio Deepfake Detection via Latent Space Refinement and Augmentation ^††thanks: ^† Corresponding author

To capture the inherent variations within the spoof class, we introduce multiple learnable prototypes that refine the latent distribution. Assume there are $K$ prototypes for each class, denoted as $\{c_{1},\dots,c_{K}\}$. For the bonafide class, $K=1$, while for the spoof class, $K$ is a hyperparameter chosen based on the complexity of the data. To determine the probability of a sample $x$ belonging to a particular class, we compute the maximum cosine similarity between its embedding $z$ and each of the class prototypes :

where $\langle x,y\rangle=\frac{x\cdot y}{\|x\|\|y\|}$ represents the cosine similarity between two vectors, and $\gamma$ is the scaling factor, set to 10. We smooth the maximum operator using a softmax-like operation to prevent sensitivity between multiple prototypes.

To guide the learning of these prototypes, we design a prototype-based classification loss, inspired by the additive angular margin loss [18]:

Here, $y\in\{0,1\}$ is the label of sample $x$, $m$ is an angular margin penalty, and $s$ is a scaling factor. This loss function encourages the model to push the embeddings of genuine samples closer to the bonafide prototype and spoofed samples closer to their corresponding prototypes.

While prototypes are learned during the training process, there’s a risk that they may collapse to a single center. To mitigate this, we introduce an intra-class regularization for the spoof prototypes $\{c^{s}\}$:

This regularization term calculates the mean similarity between the spoof prototypes, encouraging them to spread out in the latent space, thereby preventing prototype collapse.

To further enhance the distinction between spoof and bonafide prototypes, we introduce an inter-class regularization term. This term calculates the smoothed maximum cosine similarity between the spoof prototypes $\{c^{s}\}$ and the single bonafide prototype $c^{b}$:

here $\delta$ is a regularization coefficient that prevents the loss from becoming negative.

Hence, the overall objective function for LSR is defined as follows:

In addition, the LSR loss can be incorporated alongside a binary classification loss, such as Weighted Cross Entropy (WCE), to refine the latent distribution and reduce intra-class variance.

While multi-prototypical refinement enhances the representation of the spoofed class, further generalization can be achieved by augmenting the diversity of the training data. Instead of solely augmenting raw input data, we apply augmentation directly in the latent space, where lower dimensionality allows for more targeted variations. By focusing these augmentations on spoofed latent features, we generate diverse spoofing variations. Notably, these augmentations are not applied to bonafide latent features, preserving their authenticity.

Given $z$ a batch of embeddings, we denote the spoof embeddings in this batch as $z^{s}$ and the bonafide embeddings as $z^{b}$. To create diverse variations of spoof embeddings, we design five latent augmentation patterns for $z^{s}$:

Additive Noise (AN). A simple yet efficient idea is to add random perturbation to latent features. Here we apply the additive noise drawn from a Gaussian distribution as follows:

where $\mathcal{N}(0,\mathbf{I})$ is the standard normal distribution, $\mathbf{I}$ is the identity matrix, and $\beta$ is a scaling factor sampled from $\mathcal{N}(0,1)$.

Affine Transformation (AT). This common transformation for 1D vectors involves scaling and translating the latent features:

where $a$ is sampled from $\mathcal{U}(0.9,1.1)$ and $b$ is set to 0.

Batch Mixup (BM). Inspired by data mixup strategies [19], we creates new latent features by blending pairs of spoof features in the batch, creating smoother transitions and intermediate variations:

where $i$ indexes the batch, $\pi$ denotes a random permutation of the batch indices and $\alpha$ is a mixup coefficient sampled from $\mathrm{Beta}(0.5,0.5)$.

The following two techniques rely on the prototypes learned in latent space refinement:

Linear Interpolation (LI). To create more challenging examples targeting the decision boundary, we perform linear interpolation on spoof embeddings towards bonafide prototype $c^{b}$. Since the prototypes in LSR the prototypes are normalized to lie on a unit hypersphere due to the use of cosine similarity, the norm of the vectors is incorporated to adjust for the transition to Euclidean space:

where $\lambda_{i}$ is an interpolation coefficient sampled from $\mathcal{U}(0,0.1)$, and the norm term $\|z^{s}\|/\|c^{b}\|$ aligns the scales of the vectors.

Linear Extrapolation (LE). In addition to interpolation, we also perform extrapolation from the nearest spoof prototype to create new features:

where $c^{s}_{n}$ corresponds the nearest spoof prototype of $z^{s}$ and $\lambda_{e}$ is an extrapolation coefficient sampled from $\mathcal{U}(0,0.1)$. Similarly, we use the norm $\|z^{s}\|/\|c^{s}_{n}\|$ to adjust for the Euclidean representation. This method extends the spoof features further away from the nearest prototype, generating more diverse variations.

Finally, the augmented latent features $\hat{z^{s}}$ are concatenated with the original features $z$, forming $z^{\prime}=[z\parallel\hat{z^{s}}]$. These enhanced features are then used for loss calculation during subsequent training, allowing the model to learn from a more varied set of spoofed data.

To evaluate the overall performance of the proposed methods, we tested the system on four datasets and compared the results with those from the literature that used the same training dataset, as shown in Table I. Across all datasets, LSR+LSA consistently outperforms LSR alone and often ranks among the top performers, highlighting the effectiveness of integrating latent space refinement with latent space augmentation. To further enhance the results, we applied additional data augmentation, which led to EERs of 0.12% on 19LA, 1.05% on 21LA, 1.86% on 21DF, and 5.54% on ITW. This places our method on par with, or ahead of, the current state-of-the-art methods. Notably, our method focuses on refining and augmenting the latent space, which contrasts with recent approaches that focus on modifying the model architecture [28, 29]. These two strategies—latent space manipulation and architectural improvements—target different aspects of the problem and could potentially be combined for even better results. This highlights the flexibility and advantage of our method, as it enhances generalization without needing to alter the underlying model architecture. In summary, the proposed LSR+LSA method consistently delivers strong results, matching or outperforming state-of-the-art performance across various datasets, demonstrating its robustness and effectiveness in generalizing across diverse deepfake detection tasks.

Table II presents the performance of various loss configurations during training. The baseline configuration uses weighted cross entropy (WCE) loss for binary classification, with OC Softmax [7] included for comparison. Incorporating Latent Space Refinement (LSR) improves performance over both WCE and OC Softmax. We further examine the effects of LSR’s loss terms. Removing inter-class regularization results in minimal degradation, indicating that the core prototype-based loss sufficiently handles prototype separation. However, removing intra-class regularization significantly reduces performance, as this term is crucial for maintaining prototype diversity within the spoof class and preventing collapse. When both regularizations are removed, performance drops to baseline levels. Additionally, combining LSR with WCE yields the best overall results. While WCE provides a solid foundation for binary classification, LSR refines the latent space to better capture variations in spoofed data. This combination leads to improved generalization across the datasets.

Meanwhile, we evaluated the impact of the number of prototypes on performance, as shown in Fig. 3. Increasing the prototypes from 1 to 8 improves performance, but further increasing to 16 shows diminishing returns. At 20 prototypes, performance declines, suggesting that too many prototypes can hinder generalization.

To assess the impact of different latent space augmentation methods, we conducted experiments for each method, as summarized in Table III, and visualized their effects using t-SNE in Fig. 2. Notably, since LI and LE rely on LSR prototypes, all systems were trained with LSR+WCE loss. Among the first three augmentations that are independent of the prototypes, AN and AT produced more dispersed and varied distributions, leading to better performance. In contrast, BM’s distribution remained closer to the original due to its mixup nature, which limited its effectiveness. For the prototype-dependent augmentations, LI, while beneficial, underperformed compared to the others, likely due to the consistent generation of challenging examples. LE, however, achieved the best results, as it effectively expanded the distribution into new regions of the latent space, offering a more balanced diversity. Ultimately, combining all augmentation methods led to the most diverse latent space, resulting in the highest overall performance.

While we have demonstrated the effectiveness of augmentation in latent space, we were curious whether applying the same augmentations in the input space could yield comparable or even better results. To explore this, we conducted comparison experiments between augmentations applied in the input space versus the latent space, focusing on three methods that do not depend on latent prototypes or embeddings: AN, AT, and BM. All experiments were conducted using WCE loss without LSR. As shown in Table IV, applying augmentation, whether in the input or latent space, improves the baseline to some extent. For AN and AT, augmentations performed in the latent space consistently yield better results than those in the input space. This suggests that latent space augmentations may more effectively capture the underlying data distributions that the model needs to learn. Interestingly, BM yields better results when applied in the input space than in the latent space. This outcome may be attributed to the nature of Mixup augmentation, which has been widely proven effective in various audio-related tasks when performed on the input data. The input space BM likely benefits from preserving more of the original data characteristics while still introducing beneficial variability.