Combining Text-based and Drag-based Editing for Precise and Flexible Image Editing

Recently, notable breakthroughs in diffusion models (Ho et al., 2020; Song et al., 2020a; b), have led to many impressive applications (Meng et al., 2021; Dong et al., 2023; Kumari et al., 2023). Among them, image editing is recognized as a significant area of innovation and has gained enormous attention (Kim et al., 2022; Nichol et al., 2021; Sheynin et al., 2024; Valevski et al., 2023). Generally, the goal of this task is to edit realistic images based on various editing instructions. Mainstream methods for image editing can be coarsely categorized into two groups: 1) Global Editing, edits given images by a text prompt (Li et al., 2023; Kawar et al., 2023) or an extra image (Zhang et al., 2023a; Epstein et al., 2023) containing global information of the desired modification. Most of them involve finetuning a pre-trained diffusion model. 2) Local Editing, mainly consists of drag-based methods  (Pan et al., 2023). This framework requires users to click several handle points (handles) and target points (targets) on an image, then perform the semantic edit to move the content of handles to corresponding targets. Typical methods  (Mou et al., 2023; Shi et al., 2024) usually contain a motion supervision phase that progressively transfers the features of handles to targets by updating the DDIM inversion latent, and a point tracking phase to track the position of handles by performing a nearest search on neighbor candidate points.

Although the aforementioned methods have gained significant prominence, two drawbacks of them cannot be overlooked. 1) Imprecise Description in Global Editing. Global editing such as text-based image methods (Kawar et al., 2023) is unambiguous but is hard to provide detailed edit instructions. For example, in Figure 1(b), the prompt (i.e.“The sculpture is smiling”) tells the model how to edit the image, but it is difficult to provide fine-grained editing information, like the extent of the smile. 2) Ambiguity Issues in Local Editing. Although local editing like drag-based editing methods can perform precise pixel-level spatial control, they suffer from ambiguity because the same handles and targets can correspond to multiple potential edited results. For example, in Figure 1(a), there exist two edited results meeting the drag requirements: one enlarging the face, the other making the woman smile. A natural approach to solving the ambiguity problem is to add more point pairs, but it does not work in real practice. This is because the diffusion network is a Markov chain and related errors accumulate as the update continues. Thus when adding more points, it usually means more update iterations, which will result in the degradation in image fidelity. In conclusion, local editing is precise but ambiguous while global editing is exactly the opposite.

Since these two kinds of edit are complementary, it is natural to ask: can we combine these two control signals to guide the image editing process? In this way, text signals can serve as global information to reduce ambiguity. Meanwhile, drag signals can act as local control signals, providing more detailed control. However, combining these two signals presents two challenges: 1) How to integrate two different kinds of signals efficiently? This is difficult because text-based and drag-based methods have completely different training strategies. Specifically, most text-based methods (Kim et al., 2022) require finetuning a pre-trained diffusion model to gradually inject the prompt information. But drag-based methods (Shi et al., 2024) typically involve freezing the diffusion model and only optimizing the DDIM inversion latent of the given image. Besides, the update in text-based editing often involves the whole denoising timesteps while drag-based approaches only focus on a specific timestep. 2) How to solve the optimization problem and maintain the image quality? Previous drag-based methods are very slow in some situations. Sometimes the handles will stuck at one position for many iterations. In worse situations, they even move in the opposite direction of targets. This phenomenon becomes more serious when adding text signals because combining two different signals usually requires more optimization steps. Thus we need a better approach to optimize the update process while maintaining fidelity as much as possible.

To address these problems, we propose CLIPDrag, the first method to combine text-based and drag-based approaches to achieve precise and flexible image editing (For brevity, we denote this kind of editing as text-drag edit). This approach was built based on the general drag-based diffusion editing framework, which optimizes the DDIM inversion latent of the original image at one specific timestep. Specifically, we propose two modules to solve the aforementioned problems after a typical identity finetuning process. 1) Global-Local Motion Supervision (GLMS). The key of GLMS is to utilize the gradient from text and drag signals together for ambiguity-elimination. Specifically, for text information, we obtain the global gradient by backward of the global CLIP loss for the latent. Similarly, for drag points, the local gradient is calculated using a similar paradigm in DragDiffuion (Shi et al., 2024). Because CLIP guidance methods cannot operate at a single fixed step, simply adding these two gradients is ineffective. Specifically, in GLMS, We disentangle the global gradient into two components: Identity Component perpendicular to the local gradient to maintain the image’s global structure information, and Edit Component parallel to the local gradient, which will be combined with the local gradient from drag signals to edit the image. By comparing their direction, we choose different gradient fusion strategies to either perform the edit or maintain the image identity. 2) Fast Point Tracking (FPT). As we mentioned before, combining drag and text signals will result in slow convergence in the latent optimization. Thus we propose a faster tracking method to accelerate the optimization process. FPT is introduced to update the position of handles after each GLMS operation. Specifically, when performing the nearest neighbor search algorithm, FPT masks all candidates that are far away from current targets. Consequently, handles will become closer to targets after each update, and FPT also ensures that the moving trajectory of handles will not be repetitive. It is observed that FPT significantly accelerates the image editing process and improves the image quality to some extent.

Combining these good practices, CLIPDrag achieves high-quality results for text-drag editing. Extensive experiments demonstrate the effectiveness of CLIPDrag, outperforming the state-of-the-art approaches both quantitatively and qualitatively. To summarize, our contributions are as follows:

• •

  We have pointed out that previous drag-based and text-based image editing methods have the problems of ambiguity and inaccuracy, respectively.

• •

  We propose CLIPDrag, a solution that incorporates text signals into drag-based methods by using text signals as global information.

• •

  Extensive experiments demonstrate the superiority and stability of CLIPDrag in text-drag image editing, marking a significant advancement in the field of flexible and precise image editing.

Text-based Image Editing. Unlike text-to-image generation which involves creating an image from scratch (Ho et al., 2022; Dhariwal & Nichol, 2021; Saharia et al., 2022; Gu et al., 2022), text-based image editing is about altering certain areas of a given image. Typically, training-based methods can achieve reliable performance in various editing tasks through training techniques like weak supervision  (Kim et al., 2022; Zhu et al., 2017), self-supervision (Zhang et al., 2023b; Xie et al., 2023) or full supervision  (Sheynin et al., 2024; Xie et al., 2023). Test-time finetuning editing approaches edit images on a significant step by different strategies, ranging from model fine-tuning  (Valevski et al., 2023) to embedding finetuning  (Yang et al., 2023). Another line is training and fine-tuning free approaches. Instead of optimizing the model of intermediate embeddings, these methods perform the edit by refining the text prompt  (Lin et al., 2023) or modifying the attention map  (Hertz et al., 2022; Cao et al., 2023).

Drag-based Image Editing. This kind of method achieves precise spatial control over specific regions of the image based on user-provided drag instructions, first proposed by DragGAN (Pan et al., 2023). DragDiffusion (Shi et al., 2024) transfers this idea to the realm of diffusion model. Building on this foundation, GoodDrag (Zhang et al., 2024), StableDrag (Cui et al., 2024), DragNoise (Liu et al., 2024), and FreeDrag (Ling et al., 2023) have made significant improvements to this method. Besides, by utilizing feature correspondences, DragonDiffusion (Mou et al., 2023) and DiffEditor (Mou et al., 2024) transform the image editing task into a gradient-based process by formulating an energy function that aligns with the desired edit results.

Problem Formulation. Given an image $I$ and two text prompts $P_{o}$ and $P_{e}$, where $P_{o}$ describes the original image and $P_{e}$ depicts the edited image. After choosing $n$ handles $\{h_{1},h_{2},\ldots,h_{n}\}$ and corresponding targets $\{g_{1},g_{2},\ldots,g_{n}\}$, a qualified text-drag edit method has to satisfy the following two requirements: 1) Move the feature of point $h_{i}$ to $g_{i}$, while preserving the irrelevant content of the original image. 2) The edited image must align with the edit prompt $P_{e}$.

General Framework: As shown in Figure 2, our method CLIPDrag, consists of three steps:

1) Identity-Preserving Finetuning (Sec 3.1): Given the input image $I$ and its description $P_{o}$, We first finetune a pre-trained latent diffusion model $\epsilon_{\theta}$ using the technique of low-rank adpation (LoRA (Hu et al., 2021)). After the finetuning, we encode the image into the latent space and obtain the latent $z_{t}$ at a specific timestep $t$ through DDIM inversion. $z_{t}$ will be optimized for many iterations to achieve the desired edit, and each iteration consists of two following phases.

2) Global-Local Motion Supervision (Sec 3.2): We denote the latent at $t$-th step and handle point $p_{i}$ during the $k$-th iteration as $z^{k}_{t}$ and $p^{k}_{i}$ respectively. At $k$-th iteration, we can calculate global and local gradient vectors for $z^{k}_{t}$. These two gradients will be combined by our Global-Local Gradient Fusion method, and the final result is used to update $z^{k}_{t}$ to $z^{k+1}_{t}$.

3) Fast Point Tracking (Sec 3.3): After GLMS, we can update the position of handles with the nearest neighbor search. To accelerate the editing speed while preserving the image fidelity, we propose FPT to ensure that new handles are closer to their corresponding targets by filtering all candidates that are far away from the targets.

In this work, we tackled the ongoing challenge of achieving precise and flexible image editing within the field of computer vision. We observed that existing text-based methods often lack the precision for specific modifications, while drag-based techniques are prone to ambiguity. To overcome these challenges, we introduced CLIPDrag, a pioneering method that uniquely integrates text and drag signals to enable accurate and unambiguous manipulations. We enhanced existing drag-based methods by treating text features as global guidance and drag points as local cues. Our novel global-local gradient fusion method further optimized the editing process during motion supervision. Additionally, to address the issue of slow convergence in CLIPDrag, we developed a FPT method that efficiently guides handle points toward their target positions. Eextensive experiments clearly demonstrated that CLIPDrag significantly outperforms existing methods that rely solely on drag or text-based inputs.