Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for VPaaS via Watermark

Visual Prompt Learning (VPL) bahng2022exploring ; huang2023diversity ; bar2022visual ; chen2023understanding is a novel learning paradigm proposed to address the computational and storage constraints associated with fine-tuning large models for downstream tasks adaptation. Its primary objective is to acquire a task-specific visual prompt, enabling the reuse of pre-trained models across diverse tasks. Domain experts design an appropriate visual prompt, refined through extensive data analysis and computational resources. Generating visual prompts involves two key stages: input and output transformations.

Input transformation. This phase focuses on devising a suitable prompt to convert downstream target task data $D_{t}$ into the source data $D_{s}$ distribution. To achieve this, visual prompts inject a task-relevant perturbation pattern $\delta$ into the pixel space of the input sample $x$, where $x\in R^{N_{t}}$. The general form is expressed as:

$$x^{{}^{\prime}}(\delta)=h(\delta,x)\in R^{N_{s}},x\in R^{N_{t}}$$

(1)

Here, $h(.,.)$ denotes the input conversion function, with perturbation patterns typically encompassing random position patches, fixed position patches, and padding bahng2022exploring . Most studies utilize additive transformation functions based on padding patterns, as illustrated in Figure 1.

Once the prompt model in Equation (1) is designed and parameters initialized, given a frozen target pre-training model $f_{t}$ and a downstream task dataset $D_{t}=\left\{(x_{1},y_{1}),...,(x_{m},y_{m})\right\}$, the training process resembles that of generating high-precision models under supervised learning. The cross-entropy loss function $L(.,.)$ guides backpropagation and the stochastic gradient descent method optimizes the prompts. The specific losses are defined as:

$$\underset{\delta}{argmin}E_{(x,y)\in D_{t}}[L(f_{t}(x^{{}^{\prime}}(\delta)),y)]$$

(2)

Output transformation: In classification tasks, the output category $K_{s}$ of the source pre-trained model often differs from the target downstream task category $K_{t}$, typically $K_{t}<K_{s}$. Thus, establishing a suitable label mapper is necessary to create a one-to-one correspondence between the source label space $Y_{s}$ and the target label space $Y_{t}$, facilitating the direct prediction of the correct target label by the pre-trained model. Current research primarily explores three tag mapping methods: ① random mapping bahng2022exploring ; elsayed2018adversarial , ② pre-defined, one-shot frequency-based mapping tsai2020transfer ; chen2021adversarial , and ③ iterative frequency-based mapping chen2023understanding . These methods are discussed in detail below.

①Random Label Mapping-Based Visual Prompting (RLMVP): RLMVP does not leverage any prior knowledge of the pre-trained model and the source dataset to guide the label mapping process. It randomly selects $K_{t}$ labels from the source label space $Y_{s}$ and maps them directly to the target labels. For instance, when employing the pre-trained RN50 model on the source task dataset (ImageNet) for the downstream target dataset (SVHN), the top 10 model outputs are typically directly used as target indices, i.e., ImageNet label i → SVHN label i, despite the lack of interpretation.

②Frequency label mapping-based visual prompting (FLMVP), FLMVP selects the top $K_{t}$ labels with the highest output label frequency in $f_{t}$. It utilizes $\delta=0$, inputs $X+\delta$ into $f_{t}$, and maps a target label $K_{t}$ to the source label $K_{s}$ according to the equation:

	$\displaystyle y_{s}(y_{t})=\underset{y_{s}}{argmax}\ Pr\left\{Top-1\ prediction\ of\ f_{t}(h(0,x))\ \right.$		(3)
	$\displaystyle\left.is\ y_{s}\mid\forall x_{t}\in\Gamma_{y_{t}}\right\}$

Here, $y_{s}(y_{t})$ explicitly expresses the dependence of the mapped source label on the target label, $\Gamma_{y_{t}}$ represents the target dataset in class $y_{t}$, and $Pr(.,.)$ is the top-1 prediction of $f_{t}$, representing the probability of the source class under the zero-padded target data point in $\Gamma_{y_{t}}$.

③Iterative Label Mapping-Based Visual Prompting (ILMVP): Building upon FLMVP, ILMVP accounts for dynamic changes in the mapping between the source label space and the target label space, incorporating interpretable considerations. It employs a two-layer iterative optimization method (BLO) to optimize the underlying label mapping and prompts. In each epoch, it uses the $\delta$ (non-zero) from the previous round of optimization and employs the frequency calculation method in FLMVP to map the target label $y_{t}$ to the source label $y_{s}$ , as shown in Equation (4). Subsequently, Equation (2) is utilized to calculate the loss for backpropagation optimization of the prompt:

	$\displaystyle y_{s}(y_{t})=\underset{y_{s}}{argmax}\ Pr\left\{Top-1\ prediction\ of\ f_{t}(h(\delta,x))\ \right.$		(4)
	$\displaystyle\left.is\ y_{s}\mid\forall x_{t}\in\Gamma_{y_{t}}\right\}$

Visual Prompting as a Service (VPaaS) has seen rapid development, as illustrated in the Praas pipeline in Figure 2. Like prompt learning in large language models yao2023promptcare , visual prompting involves three primary stakeholders: VPaaS developers, pre-trained model service providers, and end users. VPaaS developers utilize their downstream task datasets and pre-training model services they have purchased to collaboratively optimize the training of visual prompts based on specific requirements. These crafted prompts, which demand significant data and computational resources, are sold or shared with authorized pre-trained model service providers to build a prompt library. Customers who procure services from a pre-trained model provider submit query requests via the public API. Subsequently, the model provider selects suitable prompts from the library, integrates them with the query content, and delivers the computed results back to the customer. In this business model, unauthorized acquisition of visual prompts by a pre-trained model service provider, such as model theft, malicious copying, or distribution, can lead to substantial economic losses for VPaaS developers and serious infringement of their intellectual property (IP). Therefore, protective technologies must be developed to safeguard the IP of VPaaS developers while allowing for external verification of prompt ownership.

This paper focuses on protecting the copyright of visual prompts through watermarking. The scenario involves two entities: the attacker and the defender. We assume the defender to be a VPaaS developer aiming to publish its prompts and detect copyright infringements in suspicious prompts. The defender has full control over prompts but can only access the pre-trained model provider’s API in a black-box manner to obtain classification output labels. During the training phase, the defender can covertly embed watermark information into prompts and subsequently compare the consistency between the extracted and embedded watermark information during verification to detect piracy. On the other hand, the attacker is a malicious pre-trained model service provider attempting to obtain prompts through unauthorized copying or theft. The attacker refrains from training specialized downstream datasets for task prompts but possesses some understanding of task input and output. Additionally, the attacker can fine-tune and prune pirated prompts to evade piracy detection. Both the attacker and defender have access to the pre-trained model.

This paper assumes the defender embeds secret information into $\delta$ to protect prompt intellectual property (IP). As depicted in Figure 3, our method consists of two primary steps: (1) watermark injection and (2) watermark verification. Specifically, we employ a backdoor-based method to inject watermarks into prompts and design hypothesis tests to guide prompt verification. The technical details of each step will be described in the following subsections.

We consider a visual prompt $\delta$ added to a raw image $x$ and then processed by an API within a pre-trained model $f$ to generate the mapped label $y$. The VPaaS developer (the defender) aims to embed secret watermark information into $\delta$, transforming it into a watermarked version $\hat{\delta}$ that ensures effectiveness, harmlessness, and robustness. The defender possesses a comprehensive understanding of the complete training process for $\delta$ and has access to the entire downstream task dataset $D_{c}=\left\{(x_{i},y_{i})\right\}_{i=1}^{N}(x_{i}\in X,y_{i}\in Y)$, where $x_{i}\in X$ and $y_{i}\in Y$ represent the input and output spaces of the downstream data.

To embed the watermark, the defender randomly selects a subset $D_{s}$ from $D_{c}$ to generate a modified version $D_{p}$ using a specialized poisoning generator $G$ and a target label $y_{t}$. Typically, $D_{s}\in D_{c}$ and $D_{p}=\left\{(x^{\prime},y_{t})\mid x^{\prime}=G(x),(x,y)\in D_{s}\right\}$. The final poisoned dataset $D_{p}$ is merged with the original clean dataset $D_{c}$ to create the watermark dataset $D$. Particularly, $r_{p}=\frac{\mid D_{p}\mid}{\mid D_{c}\mid}$ is recorded as the poisoning ratio. In this paper, we follow the common construction method of the poisoning generator $G$ gu2019badnets :

$$x^{\prime}=G(x)=(1-\alpha)\otimes x+\alpha\otimes t$$

(5)

Here, $t$ represents the trigger mode, $\alpha$ denotes the trigger’s position, and this paper adopts a cross iteration of black and white pixel blocks to construct $t$. Simultaneously, the following loss function performs watermark injection on $\delta$.

Effectiveness. To ensure differentiability, we need to ensure that the watermarked $\hat{\delta}$ and the clean $\delta$ yield different predictions when combined with the triggered image $x$ and inputted into the pre-trained model $f_{t}$. As the original real label of $x$ in $D_{p}$ is not $y_{t}$, and the label after poisoning is $y_{t}$, it is essential to ensure that the sample $x^{\prime}$ with a trigger minimizes the loss under both $\hat{\delta}$ and $f$. The losses are as follows:

$$L_{p}=\frac{1}{\mid D_{p}\mid}\sum_{(x^{\prime},y_{t})\in D_{p}}L(f(h(\hat{\delta},x^{\prime})),y_{t})$$

(6)

Harmlessness. We need to ensure that embedding watermarks will not affect the usefulness of prompt $\hat{\delta}$ on clean samples $x$, i.e., the prediction accuracy in visual classification tasks. Therefore, we follow the standard prompt training process and ensure that $x$ minimizes the loss with $y_{t}$ under both $\hat{\delta}$ and $f$. The loss is as follows:

$$L_{c}=\frac{1}{\mid D_{c}\mid}\sum_{(x^{\prime},y_{t})\in D_{c}}L(f(h(\hat{\delta},x)),y)$$

(7)

Based on the above analysis, the watermark injection process is then formulated as an optimization problem:

$$\underset{\hat{\delta}}{argmin}(L_{c}+\beta\cdot L_{p})$$

(8)

Here $L(.,.)$ denotes a cross-entropy loss function, $\beta$ represents a hyperparameter used to balance distinguishability and harmlessness, and $\beta$ is one unless otherwise specified. We apply the stochastic gradient optimization method to solve this optimization problem.

Robustness: In conventional machine learning trained neural network models, backdoor watermarks usually show high Robustness to common model transformations (such as model pruning and fine-tuning). For visual prompts, we believe that the watermarking method in this paper is also robust in prompting fine-tuning and pruning. We verify this conclusion in Section 4.4.

Give a suspicious prompt to $\delta$, prompting the owner (VPaaS developer) to verify whether it originates from $\hat{\delta}$ without IP authorization by detecting a specific watermark backdoor. Let $x^{\prime}$ represent the poisoned sample and $y_{t}$ denote the target label. By querying the pre-trained model $f$, the owner can check for suspicious prompts via the results of $f(h(\delta,x^{\prime}))$. If $f(h(\delta,x^{\prime}))=y_{t}$, the suspicious prompt is regarded as a piracy prompt. However, this result may be affected by the randomness of selecting sample $x^{\prime}$. Therefore, we introduce a hypothesis testing guided method in this paper to enhance the credibility of the verification results.

Proposition 1 (Vision Prompt Ownership Verification), assuming $C(x)$ represents the predicted label obtained by inputting the pre-trained model $f$, such as $C(x)=f(h(\delta,x))$. Let variable $X$ denote clean samples of non-target label $y_{t}$, with the size of $X$ denoted as $m$. $X^{\prime}$ is the version generated by the poisoner $G$, such as $X^{\prime}=G(X)$. Given a full hypothesis $H_{0}:C(X^{\prime})\neq y_{t},(H_{1}:C(X^{\prime})=y_{t})$, where $y_{t}$ is the predefined target label. We claim the suspicious prompt $\delta$ is a pirated copy of $\hat{\delta}$ if and only if $H_{0}$ is rejected.

In practice, without specific instructions, we randomly select $m$ $(m\neq 100)$ distinct benign samples with non-target labels. We conduct a one-tailed paired t-test hogg2013introduction and compute its p-value. Experimental results represent the average of three random selections. If the p-value is below the significance level $\alpha$, we reject the null hypothesis $H_{0}$.

Datasets and pre-trained Models. We conducted experiments on three benchmark visual image classification datasets: CIFAR10, EuroSAT, and SVHN. CIFAR10 comprises 60,000 images across ten object classes, EuroSAT includes 27,000 images spanning ten land use categories, and SVHN comprises 99,289 images representing 10 street view house numbers. The dataset was divided into non-overlapping train, test, and validation sets, with equal-sized test and validation sets. The validation set primarily assessed prompt robustness to various post-processing operations. Detailed partition information is provided in Table 1.

Three representative visual pre-training models were selected: ResNet trained on ImageNet-1K (RN50) he2016deep , Big Transfer (BiT-M) kolesnikov2020big , and ResNeXt trained on 3.5B Instagram images (Instagram) mahajan2018exploring . Table 2 describes the details of the pre-trained models.

Prompt Post-processing. Attackers often employ post-processing techniques to circumvent IP detection mechanisms to modify prompt parameters, such as prompt fine-tuning and pruning. The goal is to ensure that visual prompts still pass WVPrompt validation even after these modifications.

• •

  Prompt fine-tuning: This process involves updating all parameters in visual prompts. In this setting, we freeze the pre-trained model and conduct ten epochs of iterative training using the validation set to complete fine-tuning.

• •

  Prompt Pruning: Pruning is a popular neural network model compression method and has been widely used in previous work to remove embedded identity watermark information. Similarly, this paper proposes a method of prompt pruning, which is primarily implemented by iteratively resetting the $q\%$ parameters with the smallest absolute value of the visual prompt to 0. In this case, we gradually increase $q$ from 0.1 to 0.9 in steps of 0.1. Generally speaking, padding-based prompts consist of four blocks (or layers): top, bottom, left, and right, and the parameter sizes of different blocks overlap. Therefore, this paper proposes two pruning methods: independent pruning of each block (pruning-block) and joint pruning of all blocks (pruning-all).

Evaluation Metrics: To verify the effectiveness of WVPrompt in watermark verification, we conducted a two-sample hypothesis test and used p-values to evaluate our method (Proposition 1). Additionally, we utilized two metrics to evaluate the effect of watermark injection: downstream task accuracy (DAcc) and watermark success rate (WSR). DAcc represents the accuracy of the watermarked prompts combined with the pre-trained model on the clean test set, which is used to verify the harmlessness of WVPrompt. WSR represents the accuracy of the test set with backdoor triggers, which is used to prove the robustness of WVPrompt. Generally, a more minor impact on DAcc and a larger WSR imply greater watermark effectiveness.

Detailed settings: We followed the default experimental settings in visual prompt learning bahng2022exploring . The prompt template was padded with a size $p$ of 30 on all sides. The number of parameters per prompt is $2\times C\times p\times(H+W-2p)$, where $p,C,H$, and $W$ are the prompt size, image channels, height, and width, respectively. All images were resized to 224 × 224 to match the input to the pre-trained model. Therefore, the number of parameters per prompt is 69,840. In the specific downstream task, 100 epochs were used, with a batch size of 128, an initial learning rate of 40, a momentum of 0.9, and a stochastic gradient descent (SGD) optimizer with a cross-entropy loss function. During prompt watermark injection, a 4*4 black and white block was used as the backdoor trigger, added to the lower right corner of the sample image, and $\beta$ was set to 1. The first category in each public dataset served as the target category. We adopt the first category in each public dataset as the target category $y_{t}$, such as ”automobile” for CIFAR10, ”1” for SVHN, and ”forest” for EuroSAT. Unless otherwise specified, the poisoning rate $r_{p}$ is 0.1.

In this section, we present experiments to evaluate the effectiveness of WVPrompt under three representative visual prompt learning methods (i.e., RLMVP, FLMVP, ILMVP). Specifically, we randomly select m as 100 samples and input them into the poisoner G to obtain samples with triggers. These samples are combined with the watermarked victim prompts and suspicious prompts before input into the large-scale pre-training model. We obtain two sets of label sequences, P1 and P2, output by the pre-trained model. Finally, we use a T hypothesis test to calculate its p-value.

To avoid the IP detection mechanism, attackers often use the post-processing technology described in Section 4.1.2 to modify stolen prompt parameters inexpensively. Therefore, this study evaluates p-values under five prompt design scenarios: unauthorized, fine-tuning, pruning-blocks, pruning-all, and independent prompts. Among these scenarios, in the first four, the suspicious prompt is considered pirated and infringing on the IP of the victim VPaaS developer. The experimental results are depicted in Table 3, demonstrating that our method accurately identifies piracy with high confidence (i.e., $p>10^{-7}$), with $96\%$ having $p>10^{-4}$. There is minimal evidence to reject the null hypothesis $H_{0}$. Conversely, there is a smaller $p$ value ($p<10^{-19}$) for independent prompts, indicating that the null hypothesis $H_{0}$ can be accepted with strong confidence. Thus, WVPrompt effectively verifies the ownership of visual prompts.

One of the primary goals of WVPrompt was to maintain its usefulness in the original downstream tasks. To verify the harmlessness of prompt watermark injection, we first compared the accuracy of clean and watermarked prompts on the clean test set of downstream tasks. The results are shown in Figure 4, where $xx_{C}$ and $xx_{W}$ represent the clean and watermarked prompts optimized under the $xx$ prompt learning method. In most datasets and pre-trained model structures, the decrease in DAcc of watermarked prompts is less than $5\%$, indicating that the injected watermark minimally affects prompt performance. Moreover, the DAcc of some watermarked prompts is even higher than that of clean prompts. This could be attributed to the training set for watermarked prompts, which consists of the clean, prompt training set $D_{c}$ and the toxic sample set $D_{p}$, enhancing the generalization capabilities of watermarked prompts. Additionally, in approximately 90% of experiments, FLMVP exhibits superior performance compared to ILMVP and RLMVP. This may be because FLMVP selects the label with the highest output probability calculated on the downstream task dataset. In contrast, ILMVP gives more consideration to the interpretability of label mappings, and RLMVP adopts a simple one-to-one mapping of the top K model labels. Overall, the WVPrompt method demonstrates harmlessness.

To evade IP protection mechanisms, attackers may modify the parameters of stolen visual prompts through prompt fine-tuning and pruning. Therefore, assessing the robustness of watermarked prompts under these post-processing techniques is crucial.

prompt fine-tuning. We fine-tune all parameters in the watermarked visual prompts using a clean validation set of the same size as the test set. Other experimental parameters remained constant. Figure 5 shows the changes in DAcc and WSR when fine-tuned on CIFAR10, EuroSAT, and SVHN datasets. It is observed that as the number of epochs increases, WSR remains unchanged or decreases slightly. The most significant drop occurs after eight fine-tuning rounds on the SVHN data in RMLVP prompt learning mode. However, overall, it remains above $82\%$. Meanwhile, the change amplitude of DAcc remains within $3\%$ in $92\%$ of cases, which we consider to be a normal change in visual prompt learning.

Prompt pruning. We set the first $q\%$ (from $10\%$ to $90\%$) minimum absolute value parameters in the prompt parameters to 0 in scenarios by pruning-block or pruning-all. Finally, the performance of prompt pruning is measured using the downstream task test set. Figures 6 and 7 illustrate that WSR and DAcc gradually decrease with increasing pruning rates. Overall, pruning-all is more stable than pruning-block. Specifically, when deleting parameters below $40\%$ by pruning-block and $50\%$ for pruning-all, WSR, and Dacc are not significantly affected. However, from $70\%$ to $90\%$, both WSR and Dacc decrease significantly. This decrease is likely due to the fact that compared to the pre-trained model, there are too few parameters in the prompt, and each bit plays a larger role.

In this section, we quantitatively analyze the core parameters in WVPrompt, including the impact of the sample number $m$ and the poisoning rate $r_{p}$ in the T-test. For simplicity, we discuss different visual prompt learning methods and different datasets under the pre-training model of BiT_M.

The influence of sample number $m$. When conducting the T-test, triggers are added to $m$ clean test samples, which are input into the pre-training model along with the victim’s watermarked and suspicious prompts. Two sets of output sequences are obtained, and their P-values are calculated. As shown in Table 5, the verification performance improves with an increase in the sample number $m$. These results are expected as our approach achieves promising WSR. Generally, a larger $m$ reduces the adverse impact of randomness in verification, leading to a higher confidence level. However, it’s essential to note that a larger $m$ requires more queries to the model API, which can be expensive and raise suspicion.

For the impact of the poisoning rate $r_{p}$. We evaluated the changes in WSR and DAcc of watermarked prompts across poisoning rates ranging from $1\%$ to $15\%$. As shown in Table 4, WSR increases with the poisoning rate $r_{p}$ in all cases. These results indicate that defenders can enhance verification confidence using a relatively large $r_{p}$. Notably, even with a small poisoning rate (e.g., $1\%$), nearly all evaluated attacks achieve watermark success rates above $90\%$. However, in some scenarios, the downstream task accuracy decreases as $r_{p}$ increases. Specifically, for the RLMVP algorithm on the SVHN dataset, when $r_{p}$ is $15\%$, the DAcc value is reduced by about $7\%$ compared to when $r_{p}$ is $1\%$. In other words, there is a trade-off between WSR and DAcc to some extent. Defenders should allocate $r_{p}$ based on specific needs in practice.