A Survey on Personalized Content Synthesis with Diffusion Models

Unique Modifer. A crucial aspect of optimization-based personalization is how the SoI is represented in text descriptions so that the users can flexibly generate new prompts. To this end, a unique modifier is designed to symbolize the SoI [9, 10], as shown in Fig. 4. More specifically, this modifier allows for the reuse and combination with other descriptions (e.g., “V* on beach”) during the inference phase. The construction of the unique modifier can be divided into three categories:

• •

  Learnable embedding. This method adds a new token and its corresponding embedding vector to the word dictionary. We call it pseudo token because it does not exist in the original dictionary. The pseudo token acts as the modifier, with adjustable weights during fine-tuning, while the embeddings of other tokens in the pre-defined dictionary will not be influenced.

• •

  Plain text. This approach utilizes an explicit text description of the SoI. For example, words such as cat or yellow cat could directly represent the user’s cat in the references. This provides detailed semantic information to improve the subject fidelity. However, it also alters the original meaning of the text, limiting the general applicability of these words, like generating other kinds of cats.

• •

  Rare token. Employing infrequently used tokens minimizes their impact on commonly used vocabularies and the generalization capabilities of the pre-trained model. However, these tokens cannot provide useful information and still exhibit weak representation in the text encoder, potentially causing ambiguity between the original text and the SoI.

Training Prompt Construction. The construction of training prompts for samples typically starts with adding prefix words, such as “Photo of V*”. However, DreamBooth [10] noted that such a simple description causes a long training time and unsatisfied performance. To address this, they incorporate the unique modifier with a class noun to describe the SoI in the references (e.g., “Photo of V* cat”). Also, the text prompt for each training reference can be more precious for better disentanglement of SoI and irrelevant concepts, such as “Photo of V* cat on the chair” [11]. This follows the tending that high-quality captions in the training set could assist in further improvement of accurate text control [12].

Training Objective. As depicted in Fig. 4, the primary goal of optimization-based methods is to refine a specific cluster of parameters, denoted as $\mathbf{\theta^{\prime}}$, for each personalization request. This process, often called test-time fine-tuning, involves adjusting $\mathbf{\theta^{\prime}}$ to reconstruct the SoI conditioned on the reference prompt. The fine-tuning is quantified by a reconstruction loss defined as

Compared to the large-scale pre-training described in Eq. 5, modifications can be seen in the learnable parameters. The commonly adopted options include optimizing token embeddings [9, 13], the entire diffusion model [10], specific subsets of parameters [14, 15, 16], or introducing new parameters such as adapters [17, 18], and LoRA [19, 20, 21]. The choice of learnable parameters impacts several factors including subject fidelity, tuning speed, and storage requirements. A fundamental observation is that an increase in the number of parameters correlates with enhanced visual fidelity.

Inference. Once the model has been fine-tuned with the optimized parameters $\mathbf{\theta^{\prime}}$, it is ready for the inference stage, where the personalized image generation takes place. By constructing new input descriptions that include the unique modifier associated with the SoI, it is easy to generate any desired images.

Overview. Recently, the learning-based framework for PCS has gained significant attention due to the rapid inference generation capability without the need for test-time fine-tuning. The basic idea is to leverage large-scale datasets to train robust models capable of personalizing diverse subject inputs. The training process involves minimizing a reconstruction loss between the generated images and the ground truth, similar to Eq. 5, to optimize the learnable parameters. However, it is not easy to train such a powerful model. The success of current methods hinges on three critical factors: 1) How to design an effective architecture to facilitate test-time personalization, 2) How to preserve the utmost information of SoI to ensure visual fidelity, 3) What is the appropriate size of the training dataset to use, In the following sub-section, we present a comprehensive analysis of these factors.

Architecture. In personalization tasks, users typically provide two types of information: one or more reference images and a textual description for content synthesis. These inputs are indispensable for structuring the architecture of the learning-based framework. According to the methods used to fuse features from these two modalities, we can categorize the learning-based approaches into two main groups: placeholder-based and reference-conditioned architectures.

Inspired by the unique modifier used in optimization-based methods, the placeholder-based methods introduce a placeholder that precedes the class noun to represent the visual characteristics of the SoI, as shown at the top of Fig. 5. The placeholder, which stores the extracted image features, is concatenated with the text embeddings processed by the text encoder. These combined features are then fused within subsequent learnable modules, such as adapters or cross-attention layers, to enhance contextual relevance.

Alternatively, the reference-conditioned architecture modifies the U-Net backbone to be conditioned on image reference. This method employs additional layers like cross-attention or adapters specifically designed to handle the integration of extra visual input. For instance, IP-Adapter [22] trains a lightweight decoupled cross-attention module, in which the image features and text features are separately processed with query features. The final output is defined as the addition result after softmax operation. In this case, no placeholder is required.

Moreover, some systems, like Subject-Diffusion [23], integrate both placeholder-based and reference-conditioned modules, taking advantage of the strengths of each approach to enhance the overall personalization capability.

SoI Feature Representation. Extracting representative features of the SoI is crucial in the creation of personalized content. A common approach is to employ an encoder, leveraging pre-trained models such as CLIP [24] and BLIP [25]. While these models excel at capturing global features, they often include irrelevant information that can detract from the fidelity, potentially compromising the quality of the personalized output, such as including the same background in the generation. To mitigate this issue, some studies incorporate additional prior knowledge to guide the learning process so as to focus on the targeted SoI. For instance, the SoI-specific mask [26, 27, 28, 29, 30, 31] contributes to the effective exclusion of the influence of the background. Moreover, using facial landmarks [32] in the context of human face customization helps improve identity preservation.

Handling multiple input references presents another challenge but is essential for real-world deployment. This necessitates an ensemble of features from the multiple references to augment the framework’s adaptability. Yet, the majority of current learning-based systems are limited to handling one reference input. Some research works [32, 33] propose to average or stack features extracted from multiple references to form a composite SoI representation.

Training Data. Training a learning-based model for PCS necessitates a large-scale dataset. There are primarily two types of training samples utilized:

• •

  Triplet Data (Reference Image, Target Image, Target Caption). This dataset format is directly aligned with the PCS objectives, establishing a clear relation between the reference and the personalized content. However, collecting large-scale triplet samples poses challenges. Several strategies have been proposed to mitigate this issue: 1) Data Augmentation. Techniques such as foreground segmentation followed by placement in a different background are used to construct triplet data [34]. 2) Synthetic Sample Generation. Methods like SuTI [35] utilize multiple optimization-based models to generate synthetic samples, which are then paired with original references. 3) Utilizing Recognizable SoIs. Collecting images of easily recognizable subjects, such as celebrities, significantly facilitates face personalization [36].

• •

  Dual Data (Reference Image, Reference Caption). This dataset is essentially a simplified version of the triplet format, where the personalized content is the original image itself. Such datasets are more accessible, including collections like LAION [37] and LAION-FACE [38]. However, a notable drawback is that training tends to focus more on reconstructing the reference image rather than incorporating the text prompts. Consequently, models trained on this type of data might struggle with complex prompts that require substantial modifications or interactions with objects.

Personalized object generation is a fundamental task, which refers to the process of creating a customized visual representation of specific object or entity. In Tab. I, we present a comparative analysis of three classical methods used in PCS—Textual Inversion [9], DreamBooth [10], and ELITE [39]. Each method employs a different approach. We will introduce the specifics of each method and explore their subsequent developments

Textual Inversion [9] applies a simple yet effective method that inserts a new token in the tokenizer to represent the subject of interest. By reconstructing the SoI references with a noisy input, the learnable pseudo token is optimized. One of the significant benefits of this method is its minimal storage requirement, with the new tokens consuming just a few kilobytes. However, the method has some drawbacks. It compresses complex visual features into a small set of parameters, which can lead to long convergence times and a potential loss in visual fidelity. To address the issue of prolonged training times, the study by [40] identifies the injected noise causes the failure of traditional convergence metrics in determining the precise end of training. After eliminating all randomness, the reconstruction loss becomes significantly more informative, and a stopping criterion that evaluates the loss variance is designed. Recent efforts in enhancing the capabilities of pseudo token embeddings are evident in several innovative approaches. P+ [13] introduces distinct textual conditions across different layers of the U-Net architecture, thereby offering better attribute control through additional learnable parameters. NeTI [41] advances this concept by proposing a neural mapper that adaptively outputs token embeddings based on the denoising timestep and specific U-Net layers. Further, ProSpect [42] demonstrates that different types of prompts—layout, color, structure, and texture—are activated at different stages of the denoising process. Inspired by this, they also recommend optimizing multiple token embeddings tailored to different denoising timesteps. Similarly, a study by [43] this layered activation insight to learn distinct attributes by selectively activating the tokens within their respective scopes. Later, HiFiTuner [27] integrates multiple techniques to achieve higher text alignment and subject fidelity. These include a mask-guided loss function, parameter regularization, time-dependent embedding, and generation refinement assisted by the nearest reference. In addition to time-dependent embedding, optimizing both negative and positive prompt embeddings as suggested by DreamArtist [44] represents another method to refine the training process. Although these advanced techniques mainly aim to enhance the accuracy of subject representation, they also contribute to expediting the training phase by introducing more parameters. In addition to these approaches, further refinements and innovations are continually being explored by injecting other knowledge prior or targeting specific requirements. For example, InstructBooth [45] introduces a reinforcement learning framework that utilizes a human preference scorer [46] as a reward model to provide feedback on text fidelity. [47] introduces a gradient-free evolutionary algorithm to iteratively update the learnable token. In summary, recent developments following the foundational work of Textual Inversion focus on reducing training times and enhancing the visual quality of generated images.

In the realm of optimization-based methods for PCS, there is a clear shift towards fine-tuning model weights rather than just the token embeddings. This approach often addresses the limitations where token embeddings alone struggle to capture complex semantics uncovered in the pre-training data [19, 48]. DreamBooth [10] proposes to use a unique modifier by a rare token to represent the SoI and fine-tune the whole parameters of the diffusion model. Besides, a regularization dataset containing 20-30 images with the same category as SoI is adopted to overcome the overfitting problem. These two combined approaches achieve impressive performance that largely promotes the progress of the research on image personalization. However, fine-tuning the entire model or significant portions for each new object causes considerable storage costs, potentially rendering widespread application. To address this, Custom Diffusion [14] focuses on identifying and fine-tuning critical parameters, particularly the key-value projections in cross-attention layers, to achieve a balance of visual fidelity and storage efficiency. Further approach, Perfusion [15], also adopts the cross-attention fine-tuning and proposes to regularize the update direction of the K (key) projection towards the super-category token embedding and the V (value) projection towards the learnable token embedding. COMCAT [49] introduces a low-rank approximation of attention matrices, which drastically reduces storage requirements to 6 MB while maintaining high fidelity in the outputs. Additionally, methods like adapters [17] and LoRA [19, 20, 21, 30, 31] and their variants [50] are increasingly utilized in personalized generation for parameter-efficient fine-tuning. It is worth noting that the pseudo token embedding fine-tuning is compatible with diffusion weight fine-tuning. For instance, the fine-tuned prompt embedding can be regarded as an effective initialization for the subsequent diffusion weight fine-tuning [51]. Also, these two parts can be simultaneously optimized with different learning rates [28, 52].

Fast response time is a crucial factor for a user-oriented application. Training a powerful model that is able to process any personalization purpose is the ultimate goal. Re-Imagen [53] introduces a retrieval-augmented generative approach, which leverages features from text-image pairs retrieved via a specific prompt. While it is not specifically tailored for object personalization, it demonstrates the feasibility of training such frameworks. Later, ELITE [39] specifically targets image personalization by combining the global reference features with text embedding while incorporating local features that exclude irrelevant backgrounds. Both fused features and local features serve as conditions for the denoising process. Similarly, InstantBooth [54] retrains CLIP models to extract image features and patch features, which are injected into the diffusion model via the attention mechanism and learnable adapter, respectively. Additionally, UMM-Diffusion [55] designs a multi-modal encoder that produces fused features based on the reference image and text prompt. The text features and multi-modal hidden state are seen as guidance signals to predict a mixed noise. Another work, SuTI [35], adopts the same architecture as Re-Imagen. The difference lies in the training samples which are produced by a massive number of optimization-based models, each tuned on a particular subject set. This strategy promotes a more precise alignment with personalization at an instance level rather than the class level of Re-Imagen. Moreover, [56] combines a contrastive-based regularization technique to push the pseudo embedding produced by the image encoder towards the existing nearest pre-trained token. Besides, They introduce a dual-path attention module separately conditioned on the nearest token and pseudo embedding. Compared to the methods that use separate encoders to process a single modality, some works have explored the usage of pre-trained multi-modal large language models (MLLM) that can process text and image modality within a unified framework. For example, BLIP-Diffusion [34] utilizes the pre-trained BLIP2 [57] that encodes multimodal inputs including the SoI reference and a class noun. The output embedding is then concatenated with context description and serves as a condition to generate images. Further, Customization Assistant [58] and KOSMOS-G [59] replace the text encoder of Stable Diffusion with a pre-trained MLLM to output a fused feature based on the reference and context description. Meanwhile, to meet the standard format of Stable Diffusion, a network is trained to align the dimension of the output embedding.

Recently, some works have started to explore the combination of optimization-based and learning-based methods. Learning-based methods provide a general framework capable of handling a wide range of common objects, while optimization-based techniques enable fine-tuning to specific instances, improving the preservation of fine-grained details. [20]. DreamTuner [52] pre-trains a subject encoder that outputs diffusion conditions for accurate reconstruction. In the second stage, they adopt regularization images that are similar to the reference so as to preserve fine-grained details.

Personalized style generation seeks to tailor the aesthetic elements of reference images. The notion of “style” now encompasses a broad range of artistic elements, including brush strokes, material textures, color schemes, structural forms, lighting techniques, and cultural influences.

In this field, StyleDrop [18] leverages adapter tuning to efficiently capture the style from a single reference image. This method demonstrates the effectiveness through iterative training, utilizing synthesized images refined by feedback mechanisms, like human evaluations and CLIP scores. This approach not only enhances style learning but also ensures that the generated styles align closely with human aesthetic judgments. Later, GAL [60] explores the active learning in generative models. They propose an uncertainty-based evaluation strategy for synthetic data sampling and a weighted schema to balance the contribution of the additional samples and the original reference. Furthermore, StyleAligned [61] focuses on maintaining stylistic consistency across a batch of images. This is achieved by using the first image as a reference, which acts as an additional key and value in the self-attention layers, ensuring that all subsequent images in the batch adhere to the same stylistic guidelines.

On another front, StyleAdapter [62] employs a dual-path cross-attention mechanism within the learning-based framework. This model introduces a specialized embedding module designed to extract and integrate global features from multiple style references.

Personalized face generation aims to generate diverse ID images that adhere to text prompt specifications, utilizing only a few initial face images. Compared to general object personalization, the scope is narrowed to a specific class, humans. It is easy to acquire large-scale human-centric datasets [38, 63, 64] and utilize pre-trained models in well-developed areas, like face landmark detection [65, 66] and face recognition [67].

As for the optimization-based methods, [68] trains a diffusion-based PromptNet that encodes input image and noisy latent to a pseudo word embedding. To alleviate the overfitting problem, the noises predicted by the pseudo embedding and context description are balanced through fusion sampling during classifier-free guidance. HyperDreamBooth [20] proposes a second-time fine-tuning strategy to further enhance the fidelity after training a learning-based model on large-scale dataset. Additionally, [36] provides a novel idea that the personalized ID can be viewed as the composition of celebrity’s face which has been learned by the pre-trained diffusion model. Based on this hypothesis, a simple MLP is optimized to transform face features into the celeb embedding space.

In addition to these optimization-based methods, the number of works in learning-based frameworks is rapidly increasing. Face0 [69] detects and crops the face region to extract refined embedding. During the sampling phase, the output of classifier-free guidance is replaced by a weighted combination of the noise patterns predicted by face-only embedding, text-only embedding, and concatenated face-text embedding. The $\mathcal{W}+$ Adapter [70] constructs a mapping network and residual cross-attention modules to transform the facial features from the StyleGAN [71] $\mathcal{W}+$ space into the text embedding space of Stable Diffusion. FaceStudio [72] adapts the cross-attention layer to support hybrid guidance including stylized images, facial images, and textual prompts. Moreover, PhotoMaker [33] constructs a high-quality dataset through a meticulous data collection and filtering pipeline. They use a two-layer MLP to fuse ID features and class embeddings for an overall representation of human portrait. PortraitBooth [73] also employs a simple MLP, which fuses the text condition and shallow features of a pre-trained face recognition model. To ensure expression manipulation and facial fidelity, they add another expression token and incorporate the identity preservation loss and mask-based cross-attention loss. InstantID [32] additionally introduces a variant of ControlNet that takes facial landmarks as input, providing stronger guiding signals compared to the methods that solely rely on attention fusion.

Sometimes, users have the intention to compose multiple SoI together, which results in the new task, multiple subject composition. However, this task presents a challenge for the optimization-based methods, particularly in how to integrate the parameters within the same module which are separately fine-tuned for individual SoI.

Some works try to integrate multiple parameters into a single unified parameter. For instance, Custom Diffusion [14] proposes a constrained optimization method to merge the cross-attention key-value projection weights with the goal of maximizing reconstruction performance for each subject. Similarly, Mix-of-Show [19] update the LoRA [74] weights using the same objective.

Additionally, some works opt for the one-for-one generation following a fusion mechanism. StyleDrop [18] dynamically summarizes noise predictions from each personalized diffusion model. In OMG [21], the latent predicted by each LoRA-tuned model is spatially composited using the subject mask.

Another straightforward solution is to train a union model on a dataset containing all expected subjects. SVDiff [16] employs a data augmentation method called Cut-Mix to compose several subjects together and applies a location loss to regularize attention maps, ensuring alignment between each subject and its corresponding token. Similar joint training strategies are found in other works [14, 36] which train a single model by reconstructing the appearance of every SoI.

Except for these categories, Cones [75] aims to find a small cluster of neurons that preserve the most information about SoI. For multi-concept generation, the neurons belonging to different SoI will be simultaneously activated to generate the combination.

Towards using learning-based methods for multi-subject generation, as they naturally accommodate the integration of multiple subject features without conflict. These methods can place each feature in its corresponding placeholder, ensuring a seamless and efficient combination [76].

The field of image personalization is expanding to include not just direct visual attributes but also complex semantic relationships and high-level concepts. Different approaches have been developed to enhance the capability of models to understand and manipulate these abstract elements.

ReVersion [77] intends to invert object relations from references. Specifically, they use a contrastive loss to guide the optimization of the token embedding towards specific clusters of Part-of-Speech tags, such as prepositions, nouns, and verbs. Meanwhile, they also increase the likelihood of adding noise at larger timesteps during the training process to emphasize the extraction of high-level semantic features.

On the other hand, Lego [78] focuses on more general concepts, such as adjectives, which are frequently intertwined with the subject appearance. The concept can be learned from a contrastive loss applied to the dataset comprising clean subject images and images that embody the desired adjectives.

Moreover, ADI [79] aims to learn the action-specific identifier from the references. To ensure the inversion only focuses on the desired action, ADI extracts gradient invariance from a constructed triplet sample and applies a threshold to mask out the irrelevant feature channels.

The advancing technologies also present a challenge in terms of potential risky usage. To mitigate this, Anti-DreamBooth [80] aims to add a subtle noise perturbation to the references so that any personalized model trained on these samples only produces terrible results. The basic idea is to maximize the reconstruction loss of the surrogate model. Additionally, [81] suggests to predefine a collection of trigger words and meaningless images. These data are paired and incorporated during the training phase. Once the trigger words are encountered, the synthesized image will be intentionally altered for safeguarding.

Some personalization tasks include additional conditions for content customization. One popular task uses an additional source image as a base for personalization. The target is to replace a subject in the source image with the SoI. It can be seen as a cross-domain task of personalization and image editing. To address this requirement, PhotoSwap [82] first fine-tunes a diffusion model on the references to obtain a personalized model. To better preserve the background of the source image, they initialize the noise with DDIM inversion [83] and replace the intermediate feature maps with those derived from source image generation. Later, MagiCapture [70] broadens the scope to face customization. Another similar application is found in Virtual Try-on technologies, which involve fitting selected clothing onto a target person. The complexities of this task have been thoroughly analyzed in a survey by [84]. Additional conditions in personalization tasks may include adjusting the layout [85], transforming sketches [86], controlling viewpoint [87, 88], or modifying poses [32]. Each of these conditions presents unique challenges and requires specialized approaches to integrate these elements seamlessly into the personalized content

In video personalization, the primary inversion objectives can be categorized into three distinct types: appearance, motion, and the combination of both subject and motion.

In appearance-based personalization, authors typically use an image as the reference point and employ video diffusion models as the foundational technology. The process involves leveraging sophisticated methods from 2D personalization, such as parameter-efficient fine-tuning [89], data augmentation [90, 89], and attention manipulation [91, 92, 89]. Additionally, several studies [93, 91, 92, 94] have explored the learning-based framework. These diffusion models are specifically tailored to synthesize videos based on the image references.

For personalization centered around motion, the reference switches to a video clip containing a consistent action. A common approach is to fine-tune the video diffusion model by reconstructing the action clip [95, 96, 97, 98, 99, 100, 101]. However, distinguishing between appearance and motion within the reference video can be challenging. First, SAVE [96] applies appearance learning to ensure that appearance is excluded from the motion learning phase. Additionally, VMC [97] removes the background information during training prompt construction.

When integrating both subject appearance and motion, innovative methods are employed to address the complexities of learning both aspects simultaneously. MotionDirector [102] utilizes spatial and temporal losses to facilitate learning across these dimensions. Another approach, DreamVideo [103], incorporates residual features from a randomly selected frame to emphasize subject information. This technique enables the fine-tuned module to primarily focus on learning motion dynamics.

In summary, video personalization strategies vary significantly based on the specific aspects, appearance, motion, or both. Moreover, due to the current limitations in robust video feature representation, the process of video synthesis that is directly conditioned on another video remains an area under exploration.

Basically, the pipeline of 3D personalization begins by fine-tuning a 2D diffusion model using optimization-based methods. This tuned model then guides the optimization of a 3D Neural Radiance Field (NeRF) model [104] for each specific prompt [105, 106]. In the second phase, DreamFusion [107] is the key technique, which introduces Score Distillation Sampling (SDS) to train a 3D model capable of rendering images aligned with 2D diffusion model. Building on this foundation, several methods have been developed to improve the workflow. DreamBooth3D [108] structures the process into three phases: initializing and optimizing a NeRF from a DreamBooth model, rendering multi-view images, and fine-tuning a secondary DreamBooth for the final 3D NeRF refinement. Consist3D [109] enhances text embeddings by training two distinct tokens, a semantic and a geometric token, during 3D model optimization. TextureDreamer [110] focuses on extracting texture maps from optimized spatially-varying bidirectional reflectance distribution (BRDF) fields for rendering texture on wide-range 3D subjects.

Additionally, advancements extend to 3D avatar rendering and dynamic scenes. Animate124 [111] and Dream-in-4D [112] integrate video diffusion for 4D dynamic scene support within the 3D optimization process. In avatar rendering, PAS [113] generates 3D body poses configurable by avatar settings, StyleAvatar3D [114] facilitates 3D avatar generation based on images, and AvatarBooth [115] employs dual fine-tuned diffusion models for separate face and body generation.

Some works have introduced different personalization tasks. For example, SVG personalization is introduced by [116], in which a parameter-efficient fine-tuning method is applied to create SVGs. After first-step generation, the SVGs are refined through a process that includes semantic alignment and a dual optimization approach, which utilizes both image-level and vector-level losses to enhance the final output. [117] introduces continual learning into personalization tasks, which requires fine-tuning a model on sequential SoI inputs while preventing catastrophic forgetting. To address this issue, a loss is used to minimize the weight changes on each training step and cross-attention parameters trained on different SoI references are regularized into an union set. Another application, 360-degree panorama customization [118], is also emerging as a potential tool for personalization in the digital imaging realm.

Attention-based operations are a crucial technique in model learning, particularly for processing features effectively. These operations generally involve manipulating the way a model focuses on different parts of data, often through a method known as the Query-Key-Value (QKV) scheme. However, problems also arise in the delicate module, where the unique modifier, may dominate the attention map, leading to a focus solely on the SoI and neglecting other details [119].

To counteract this, a cluster of studies like [56, 62, 19, 52] aim to refine this mechanism to enhance feature processing. For example, Mix-of-Show [19] enhances contextual relevance through region-aware cross-attention by substituting the feature map, which is initially generated by the global prompt and replaced with distinct regional features corresponding to each entity. DreamTuner [52] designs an attention layer that takes the features of the generated image as query, the concatenation of generated features as key, and reference features as value. Additionally, the background of the reference is ignored in the attention modules.

Another research branch focuses on restricting the influence of the SoI token within the attention layers. For instance, Layout-Control [85] introduces a method to adjust attention weights specifically around the layout without additional training. Cones 2 [120] also defines some negative attention areas to penalize the illegal occupation to allow multiple object generation. VICO [121] inserts a new attention layer where a binary mask is deployed to selectively obscure the attention map between the noisy latent and the reference image features. In addition to these explicit attention weights modification methods, many researchers [76, 16, 28, 23, 29, 122, 51] employ localization supervision in the cross-attention module. DreamTuner [52] further refines this approach by designing an attention layer that more effectively integrates features from different parts of the image.

Masks serve as a strong prior that indicates the position and contour of the specified object, which is pivotal for guiding the focus of generative models. Benefitting from advanced segmentation methods, the SoI can be precisely isolated from the background. Based on this strategy, plenty of studies [26, 27, 28, 29, 30, 31, 51, 123] choose to discard the pixels of the background area so that the reconstruction loss can focus on the targeted object and exclude irrelevant disturbances. Also, another technique [124] extends to the background reconstruction for better disentanglement. In addition, as discussed in Section V-A, the layout indicated by the mask can be incorporated into the attention modules as a supervision signal. Moreover, the mask can stitch specific feature maps to construct more informative semantic patterns [39, 125]. Then Subject-Diffusion [23] takes this approach a step further by applying masking to the latent features throughout the diffusion stages. Additionally, there are some other mask integration approaches. AnyDoor [126] employs an extra high-frequency filter to extract detailed features alongside the segmented subject as a condition for the image generation process. DisenBooth [127] defines an identity-irrelevant embedding with a learnable mask. By maximizing the cosine similarity between the identity-preservation embedding and identity-irrelevant embedding, the mask will adaptively exclude the redundant information, and thus the subject appearance can be better preserved. PACGen [128] adds two more additional prompts, an SoI suppression prompt (e.g. sks person) and a diverse prompt (e.g. high quality, color image), which participates in the classifier-free guidance assisted with a binary mask that indicates subject area. Face-Diffuser [129] determines the mask through augmentation from the noise predicted by both a pre-trained text-to-image diffusion model and a learning-based personalized model. Each model makes its own noise prediction, and the final noise output is a composite created through mask-guided concatenation.

Due to limited references, existing methods often struggle to capture complete semantic information of the SoI, resulting in challenges in producing realistic and diverse images. To address this, various techniques employ data augmentation strategies to enrich the diversity of SoI. COTI [130] adopts a scorer network to progressively expand the training set by selecting semantic-relevant samples with high aesthetic quality from a large web-crawled data pool. SVDiff [16] manually constructs mixed images of multiple SoI as new training data, thereby enhancing the model’s exposure to complex scenarios. Such concept composition is also used in [23, 90, 89]. BLIP-Diffusion [34] segments the foreground subject and composes it in a random background so that the original text-image pairs are expanded to an instruction-followed dataset. To create such an instruction-followed dataset for face personalization, DreamIdentity [131] leverages the existing knowledge of celebrities embedded in large-scale pre-trained diffusion model to generate both the source image and the edited face image. PACGen [128] shows that the spatial position also entangles with the identity information. Rescale, center crop, and relocation are effective solutions for this issue. Besides, StyleAdapter [62] chooses to shuffle the patch to break the irrelevant subject and preserve the desired style. Break-A-Scene [28] aims to invert multiple subjects from a single reference image. To achieve this goal, the method samples a random subset of the target subjects and then employs a masking strategy, ensuring that the learning process is specifically focused on the sampled subjects.

Regularization is a kind of method that is used to regularize the weight update to avoid overfitting or preserve a better appearance.

A commonly adopted method is to use an additional dataset composed of images with the same category as SoI [10]. By reconstructing these images, the personalized model is required to preserve the pre-trained knowledge, which is an effective way to alleviate the overfitting issue. Building on this strategy, StyleBoost [132] introduces an auxiliary dataset for the purpose of style personalization. Later, [11] introduces a more delicate construction pipeline for the regularization dataset. This includes detailed prompts that specify the shape, background, color, and texture. To disentangle the subject from the background, the pipeline includes sampling subjects that share the same class noun both within identical and varied background contexts.

Another regularization approach is to utilize the pre-traiend text prior that is trained on large-scale datasets. In an ideal case, the SoI token is able to fluently compose with other text descriptions to generate well-aligned images like a pre-trained word. Therefore, the pre-trained words can be seen as a supervised signal to guide the optimization. For example, Perfusion [15] constrains the key projection towards the embedding of the class noun to inject the text-level knowledge and the value projection towards SoI images to achieve visual fidelity. Moreover, inspired by coached active learning [133, 134], which uses anchor concepts for optimization guidance, Compositional Inversion [119] employs a set of semantically related tokens as anchors to constrain the token embedding search towards areas of high alignment with the SoI. This kind of constraint is applied to the input of the text encoder but also applicable to the output. [135] aims to learn an encoder that produces the offset on the class token embedding to represent the key visual features. By minimizing the offset, the final word embedding denoted by the class token plus the offset is able to achieve better text alignment. Similarly, Cones 2 [120] minimizes the offset by reconstructing the features of 1,000 sentences containing the class noun. And [136] optimizes the learnable token towards the mean textual embedding of 691 well-known names. [56] proposes to use a contrastive loss to guide the SoI text embedding close to its nearest CLIP tokens pre-trained on large-scale samples. [30] aims to minimize the embedding similarity between SoI text and its class noun to improve generalizability. On the other hand, VICO [121] empirically finds the end-of-text token <|EOT|> keeps the semantic consistency of SoI. To leverage this discovery, an L2 loss is leveraged to reduce the difference of attention similarity logits between the SoI token and <|EOT|>.

In addition to these commonly used regularization terms, several studies have introduced novel methods from different aspects. Because the projection kurtosis of a natural image tends to remain constant across various projection directions [137], [138] introduces a loss function that minimizes the difference between the maximum and minimum kurtosis values extracted via Discrete Wavelet Transform [139].

To assess the performance of personalized models, various datasets have been developed:

The primary dataset used in DreamBooth [10] includes 30 subjects such as backpacks, animals, cars, toys, etc. This is expanded to DreamBench-v2 [35], which adds 220 test prompts to the subjects. Custom Diffusion [14] focuses on evaluating 10 subjects, each with 20 specific test prompts, and includes tests for multi-subject composition with 5 pairs of subjects and 8 prompts for each pair. Later, the authors released Custom-101 [14], which comprises 101 subjects to provide a broader scope of evaluation. Additionally, a recent dataset, Stellar [31], specifically targets human-centric evaluation, featuring 20,000 prompts on 400 human identities.

As personalized content synthesis aims to maintain fidelity to the SoI while ensuring alignment with textual conditions, the metrics are designed for text alignment and visual similarity. To measure how well the semantics of the text prompt are represented in the generated image, CLIP similarity score of text features and image features is widely adopted. To determine how closely the generated subject resembles the SoI, visual similarity is assessed based on large-scale pre-trained models like CLIP and DINO. Conventional metrics such as the Fréchet Inception Distance (FID) [140] and Inception Score (IS) [141] are also applicable for evaluating PCS models. These metrics provides insights into the general image quality and coherence. In addition to these commonly adopted metrics, there are some discussions of specialized metrics for PCS system evaluation. [50] suggests evaluating personalized models based on fidelity, controllability, diversity, base model preservation, and image quality. [31] develops specific metrics for human personalization, including soft-penalized CLIP text score, Identity Preservation Score, Attribute Preservation Score, Stability of Identity Score, Grounding Objects Accuracy, and Relation Fidelity Score. These metrics ensure a structured and detailed evaluation of personalized models.

Current personalized content synthesis (PCS) systems face a critical challenge of overfitting, particularly when trained with a limited set of reference images. This overfitting problem manifests in two ways: 1) Loss of SoI editability. The personalized model tends to produce images that rigidly mirror the SoI in the reference, such as consistently depicting a cat in an identical pose. 2) Irrelevant semantic inclusion. the irrelevant elements in the references are generated in the output, such as backgrounds or objects not pertinent to the current context.

To investigate the rationale behind, Compositional Inversion [119] observes that the learned token embedding is located in an out-of-distribution area compared to the center distribution formed by pre-trained words. This is also found in [136] that the pseudo token embeddings deviate significantly from the distribution of the initial embedding. Meanwhile, [119, 125, 121] found that the unique modifier dominates in the cross-attention layers compared to the other context tokens, leading to the absence of other semantic appearances.

To address this issue, many solutions have been proposed. Most methods discussed in Sec. V contribute to the alleviation of the overfitting problem, such as the exclusion of redundant background, attention manipulation, regularization of the learnable parameters, and data augmentation. However, it has not been solved yet, especially in the cases where the SoI has a non-rigid appearance [119] or the context prompt has a similar semantic correlation with the irrelevant elements in the reference [60]. It is clear that addressing overfitting in PCS is not merely a technical challenge but a necessity for ensuring the practical deployment and scalability of these systems in varied and dynamic real-world environments. Therefore, there is an urgent need for an effective strategy and robust evaluation metrics to achieve broader adoption and greater satisfaction in practical uses.

The ultimate goal of personalized content synthesis (PCS) is to create systems that not only render the SoI with high fidelity but also effectively respond to textual prompts. However, achieving excellence in both areas simultaneously presents a notable conflict. In particular, achieving high subject fidelity typically involves capturing and reproducing detailed, specific features of the SoI, which often requires the model to learn and replicate very delicate characteristics. On the other hand, text alignment demands that the system flexibly adapts the SoI according to varying textual descriptions, which might suggest changes in pose, expression, environment, or stylistic alterations that could contradict the reconstruction process during training. Therefore, it is hard to gain flexible adaption in different contexts while pushing the model to capture fine-grained details. To address this inherent conflict, Perfusion [15] proposes to regularize the attention projections by these two items. [142] decouples the conditional guidance into two separate processes, which allows for the distinct handling of subject fidelity and textual alignment. Despite these efforts, there still remains room for further exploration and refinement of this issue. Enhanced model architectures, innovative training methodologies, and more dynamic data handling strategies could potentially provide new pathways to better balance the demands of subject and text fidelity in PCS systems.

Despite the popularity of personalization, there is a noticeable lack of standardized test datasets and robust evaluation metrics for measuring progress and comparing different approaches effectively. Therefore, future efforts could focus on creating comprehensive and widely accepted benchmarks that can test various aspects of PCS models. Additionally, there is a need to develop metrics that can more accurately reflect both the qualitative and quantitative performance of PCS systems.