SEAL: Entangled White-box Watermarks on Low-Rank Adaptation

In SEAL, the forward path produces the output $W^{\prime}$ by adding a learnable offset $\Delta W$ on top of the base weights $W$:

Here, $B$ and $A$ are trainable matrices, while $C$ is a fixed passport matrix that carries the watermark. Unlike traditional LoRA layers that use $\Delta W=BA$ alone, SEAL inserts $C$ between $B$ and $A$. This additional matrix:

• •

  Forces the resulting offset $\Delta W$ to pass through an extra linear transformation, potentially mixing or reorienting the learned directions.

• •

  Ties the final weight update $\Delta W$ to the presence of $C$; removing or altering $C$ would disrupt $\Delta W$ and hence the model’s functionality.

If $C$ were diagonal, it would merely scale each dimension independently, which can be easier to isolate or undo. However, when $C$ is a full (non-diagonal) matrix, the learned offset $\Delta W$ may exhibit more complex structures, as the multiplication by $C$ intermixes channels or dimensions. Such a design can lead to a more wider singular value distribution (see Fig. 6), where the watermark is spread across multiple directions, thus making it less prone to straightforward removal.

The backward path computes gradients of the loss function $\phi$ with respect to $A$ and $B$, revealing how $C$ influences the updates. Let

where $\Delta x$ represents applying $\Delta$ to some input $x$. Then, by the chain rule,

These expressions highlight two key points:

1. (1)

   Transformation of Gradients. Each gradient, $\nabla_{A}$ and $\nabla_{B}$, is multiplied (from the left or right) by $C^{T}$. If $C$ were diagonal, this would reduce to element-wise scaling of the gradient, which is relatively simple to reverse or interpret. In contrast, a full $C$ applies a more general linear transformation—potentially a rotation or mixing—to the gradient directions.

2. (2)

   Entanglement of Learnable Parameters. Because $C$ is fixed but non-trivial, both $B$ and $A$ are continually updated in a manner dependent on $C$. Over many gradient steps, $\Delta W=BCA$ becomes entangled across multiple dimensions; single-direction modifications in $B$ or $A$ cannot easily isolate the watermark without affecting other directions.

By design, the publicly distributed weights are simply $B^{\prime}\in\mathbb{R}^{b\times r}$ and $A^{\prime}\in\mathbb{R}^{r\times a}$, analogous to standard LoRA. No additional matrix parameters (or suspicious metadata) are visible. Hence, without insider knowledge, an attacker cannot tell a priori if $(B^{\prime},A^{\prime})$ derives from SEAL or a conventional LoRA finetuning. This alone imposes a significant hurdle:

Only then might they attempt forging hidden passports.

Assume, hypothetically, that an attacker somehow knows a given $(B^{\prime},A^{\prime})$ came from SEAL. They might try a factorization of the form:

so that $\widetilde{B}\,\widetilde{C}\,\widetilde{A}=B^{\prime}A^{\prime}$. Then they could designate $\widetilde{C}$ as a forged version of the original $C$.

• •

  Concurrent Entanglement is Required. In SEAL, $B$ and $A$ are co-trained (entangled) with both $C$ and $C_{p}$ at the same time during finetuning. This ensures that, for any batch, either $C$ or $C_{p}$ is used, such that $B,A$ adapt to both passports. Merely performing a post-hoc factorization on $(B^{\prime},A^{\prime})$ does not replicate this simultaneous learning process.

• •

  One Factorization Yields One Mapping. A single factorization typically captures one equivalence, e.g. $\widetilde{C}$. Generating an additional $\widetilde{C}_{p\text{-adv}}$ that also achieves the same function (or fidelity) using the same $\widetilde{B},\widetilde{A}$ is a significantly more constrained problem. In practice, an attacker would need to re-finetune $(\widetilde{B},\widetilde{A})$ twice, once for each passport, effectively mimicking the original training—but without knowledge of the original dataset $\mathcal{D}$.

• •

  Costly and Uncertain Outcome. Even if the attacker invests major computational resources, re-training two passports from scratch is as expensive as (or more expensive than) training a brand-new LoRA model. Moreover, success is not guaranteed, since the attacker must ensure $\widetilde{C}_{p\text{-adv}}\neq\widetilde{C}$ but still replicates near-identical behavior on the entire dataset, all while not knowing the original dataset $\mathcal{D}$ or training schedule.

1. 1.

   Attackers Typically Lack Data. To even begin constructing $(\widetilde{C},\widetilde{C}_{p\text{-adv}})$, attackers must have access to the original training data (or certain proportion of dataset with similar distribution) and be certain SEAL was used. Both are high barriers. Training dataset is not a part of SEAL, and is mostly proprietary. It does not violate Kerckhoff’s principal.

2. 2.

   Equivalent to Costly Re-Training. Producing two passports that match all fidelity checks essentially replicates the original multi-passport entanglement from scratch. This yields no distinct advantage over simply training a new LoRA.

3. 3.

   Cannot Disprove Legitimate Ownership. Even if they succeed in forging $\widetilde{C},\widetilde{C}_{p\text{-adv}}$, the legitimate owner’s original pair $(C,C_{p})$ still correctly verifies, preserving the rightful ownership claim.

In summary, forging multiple passports from a single factorization of $(B^{\prime},A^{\prime})$ is infeasible because SEAL’s multi-passport structure relies on concurrent entanglement of $B,A$ with both passports $C$ and $C_{p}$ during training. A single post-hoc factorization can at best replicate one equivalent mapping, but not two functionally interchangeable mappings without a re-finetuning process that is as expensive and uncertain as building a new model. Furthermore, since SEAL weights are indistinguishable from standard LoRA, the attacker generally cannot even detect the scheme in the first place. Therefore, this approach does not offer a viable pathway to break or circumvent SEAL’s multi-passport verification procedure.

LoRA-FA (LoRA with frozen down blocks) modifies LoRA by keeping the down block frozen during training, while only the up block is trained. Structurally, however, it does not alter the fundamental matmul operator. Consequently, integrating SEAL follows the same procedure as standard LoRA: one can embed the passport $C$ into the product $B\,C\,A$ without requiring any special adjustments. The difference in training rules (i.e. freezing $A$) does not affect how $C$ is placed or how it is decomposed into $(C_{1},C_{2})$ for final public release.

LoRA+ investigates the training dynamics of LoRA’s up ($B$) and down ($A$) blocks. In particular, it emphasizes the disparity in gradient magnitudes and proposes using different learning rates:

where $\lambda\gg 1$ is a scale factor, $\eta$ is the base learning rate, and $G_{A},G_{B}$ are the respective gradients. LoRA+ does not alter the structural operator (still matrix multiplication). Therefore, SEAL can be employed by introducing $C\in\mathbb{R}^{r\times r}$ between $B$ and $A$, yielding $\Delta W=B\,C\,A$. The difference in gradient scaling does not impact the usage of a non-trainable passport matrix $C$.

VeRA introduces two diagonal matrices, $\Lambda_{b}$ and $\Lambda_{d}$, to scale different parts of the low-rank factors:

where $B,A$ may be random, frozen, shared across layers and the diagonal elements in $\Lambda_{b},\Lambda_{d}$ are trainable. Despite these diagonal scalings, the core operator remains matrix multiplication. Hence, embedding a passport $C$ is still feasible. By leveraging the commutative property of diagonal matrices and $C$ (assuming $C$ commutes with $\Lambda_{d}$ in the sense that one can re-factor $C$ into $C_{1}\Lambda_{d}C_{2}$ or $\Lambda_{d}C$), SEAL can be inserted:

which is functionally identical to $\Lambda_{b}\,B\,\Lambda_{d}\,A$ except for the hidden passport $C=C_{1}C_{2}$. Implementing SEAL in VeRA may require converting the final trained weights back into a standard $(B^{\prime},A^{\prime})$ form plus a diagonal scaling term, but the fundamental principle is straightforward.

AdaLoRA applies a dynamic rank-allocating approach inspired by SVD. It factorizes the weight update into:

where $\Lambda$ is a diagonal matrix, and $P,Q$ are regularized to maintain near-orthogonality. Since diagonal matrices commute under multiplication (up to a re-factorization), one can embed a passport $C$ by decomposing it ($f(C)\rightarrow(C_{1},C_{2})$). In essence,

where $P^{\prime}=PC_{1}$ and $Q^{\prime}=C_{2}Q$. This preserves the rank-$r$ structure and does not disrupt AdaLoRA’s optimization logic. Regularization terms that enforce $P^{\prime T}P^{\prime}\approx I$ and $Q^{\prime}Q^{\prime T}\approx I$ remain valid, though one may incorporate $C_{1},C_{2}$ into the initialization or adapt them carefully so as not to degrade the orthogonality constraints.

DoRA modifies the final LoRA update using a column-wise norm factor:

where $\|\cdot\|_{c}$ computes column-wise norms and the ratio is (by design) often detached from gradients to reduce memory overhead. Replacing $\Delta W$ with $B\,C\,A$ in DoRA does not alter the external gradient manipulation logic, since $C$ is non-trainable. Thus,

remains valid. The presence of $C$ does not interfere with DoRA’s approach to scaling or norm-based constraints.

All of the above variants preserve the core LoRA assumption of a matrix multiplication operator for the rank-$r$ adaptation. However, certain approaches introduce non-multiplicative adaptations (e.g., Hadamard product, Kronecker product, or other specialized transforms). In the following section, for these cases, which discuss how SEAL can be generalized to any bilinear or multilinear operator $\star$.

Let $\star$ be any bilinear or multilinear operator used for low-rank adaptation.¹¹1Here, bilinear means $(X\star Y)$ is linear in both $X$ and $Y$ when one is held fixed, e.g. standard matrix multiplication, Kronecker product, or Hadamard product. We can then write the trainable adaptation layer as

where $B,A$ are the trainable low-rank parameters, and $C$ is the non-trainable passport in SEAL. During training, $B$ and $A$ are optimized in conjunction with $C$ held fixed (just as in the matrix multiplication case).

• •

  Broader Applicability. By permitting $\star$ to be any bilinear or multilinear operator (Kronecker, Hadamard, etc.), SEAL naturally extends beyond the canonical matrix multiplication used in most LoRA implementations. This flexibility can be valuable for advanced parameter-efficient tuning methods (Edalati et al., 2022; Hyeon-Woo et al., 2021; Yeh et al., 2023).

• •

  Same Security Guarantees. The central watermarking principle (embedding a non-trainable passport $C$ into the adaptation) does not change. An adversary attempting to re-factor $B^{\prime}\star A^{\prime}$ to recover $C$ faces the same challenges described in the main text and Appendix C—non-identifiability, cost of reconstruction, and multi-passport verification barriers.

• •

  Potential Operator-Specific Designs. Certain operators (e.g., Kronecker product) may admit additional constraints or factorization strategies that could be exploited for improved stealth or efficiency. Investigating these is an interesting direction for future work.

We conduct evaluations on commonsense reasoning tasks using eight distinct sub-tasks: Boolean Questions (BoolQ) (Clark et al., 2019), Physical Interaction QA (PIQA) (Bisk et al., 2020), Social Interaction QA (SIQA) (Sap et al., 2019), Narrative Completion (HellaSwag) (Zellers et al., 2019), Winograd Schema Challenge (Wino) (Sakaguchi et al., 2021), ARC Easy (ARC-e), ARC Challenge (ARC-c) (Clark et al., 2018), and Open Book QA (OBQA) (Mihaylov et al., 2018).

We benchmark SEAL and LoRA against LLaMA-2-7B/13B (Touvron et al., 2023), LLaMA-3-8B (AI@Meta, 2024), Gemma-2B (Team et al., 2024), and Mistral-7B-v0.1 (Jiang et al., 2023) across these commonsense reasoning tasks.

The hyperparameters used for these evaluations are listed in Table 14.

We conducted textual instruction tuning using Alpaca dataset (Taori et al., 2023) on LLaMA-2-7B (Touvron et al., 2023), trained for 3 epochs. The hyperparameters used for this process are detailed in Table 8.

We compared the fidelity of SEAL, LoRA, and FT on the visual instruction tuning tasks with LLaVA-1.5-7B (Liu et al., 2024a). To ensure a fair comparison, we used the same original model provided by (Liu et al., 2024a) uses the same configuration as the LoRA setup with the same training dataset. We adhere to (Liu et al., 2024a) setting to filter the training data and design the tuning prompt format. The finetuned models are subsequently assessed on seven vision-language benchmarks: VQAv2(Goyal et al., 2017), GQA(Hudson & Manning, 2019), VisWiz(Gurari et al., 2018), SQA(Lu et al., 2022), VQAT(Singh et al., 2019), POPE(Li et al., 2023b), and MMBench(Liu et al., 2023).

The DreamBooth dataset (Ruiz et al., 2023) encompasses 30 distinct subjects from 15 different classes, featuring a diverse array of unique objects and live subjects, including items such as backpacks and vases, as well as pets like cats and dogs. Each of the subjects contains 4-6 images. These subjects are categorized into two primary groups: inanimate objects and live subjects/pets. Of the 30 subjects, 21 are dedicated to objects, while the remaining 9 represent live subjects/pets.

For subject fidelity, following (Ruiz et al., 2023), we use CLIP-I, DINO. CLIP-I, an image-text similarity metric, compares the CLIP (Radford et al., 2021) visual features of the generated images with those of the same subject images. DINO (Caron et al., 2021), trained in a self-supervised manner to distinguish different images, is suitable for comparing the visual attributes of the same object generated by models trained with different methods. For prompt fidelity, the image-text similarity metric CLIP-T compares the CLIP features of the generated images and the corresponding text prompts without placeholders, as mentioned in (Ruiz et al., 2023; Nam et al., 2024). For the evaluation, we generated four images for each of the 30 subjects and 25 prompts, resulting in a total of 3,000 images. The prompts used for this evaluation are identical to those originally used in (Ruiz et al., 2023) to ensure consistency and comparability across models. These prompts are designed to evaluate subject fidelity and prompt fidelity across diverse scenarios, as detailed in Table 11

Fig. 7 visually compares LoRA and SEAL on representative subjects from the DreamBooth dataset. The top row shows example reference images for each subject, the middle row shows images generated by LoRA, and the bottom row shows images from our SEAL. Qualitatively, both methods faithfully capture key attributes of each subject (e.g., shape, color, general pose) and produce images of comparable visual quality. That is, SEAL does not degrade or alter the original subject’s appearance relative to LoRA, suggesting that incorporating the constant matrix $C$ does not introduce noticeable artifacts or reduce fidelity. These results align with the quantitative metrics on subject and prompt fidelity, indicating that SEAL maintains a quality level on par with LoRA while embedding a watermark in the learned parameters.

In order to provide a concrete illustration of our watermark extraction process, we construct a small 32$\times$32 grayscale image as the passport $C$ (or $C_{p}$). Specifically, we sampled 100 frames from a publicly available YouTube clip, applied center-cropping on each frame, converted them to grayscale, and then downsampled to 32$\times$32. From these frames, we selected one representative image (shown in Fig. 3) to embed as the non-trainable matrix $C$ in our SEAL pipeline Sec. 3.3.

This tiny passport image, while derived from a movie clip, is both unrecognizable at 32$\times$32 and used exclusively for educational, non-commercial purposes. Nevertheless, it visually demonstrates how a low-resolution bitmap can be incorporated into the model’s parameter space and later extracted (possibly with minor distortions) to verify ownership.

To evaluate versatility of the proposed SEAL method under varying configurations, we conducted additional experiments focusing on different rank settings (4, 8, 16). The results are summarized in Table 15. We used the Gemma-2B model (Team et al., 2024) on commonsense reasoning tasks, as described previously. For comparison, we included the results of LoRA with $r=32$ and SEAL with $r=32$ as mentioned in Table 2.

To analyze how the magnitude of the passport $C$ influences the final output, we train the model with $\Delta W=B\,C\,A$, but at inference time remove $C$ (i.e., $\mathbb{N}(B,A,\emptyset)$) to observe the resulting images under different standard deviations std of $C$. Specifically, we sample $C\sim\mathcal{N}(0,\texttt{std}^{2})$ with $\texttt{std}\in\{0.01,0.1,1.0,10.0,100.0\}$ and keep $B$ and $A$ trainable. Fig. 9 shows that lower std (e.g., $0.01$) produces markedly different images relative to the vanilla model without $C$, while higher std (e.g., $10.0$ or $100.0$) yields outputs closer to the vanilla Stable Diffusion model⁴⁴4https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5. The original weight had been taken down..