Creating Language-driven Spatial Variations of Icon Images

Language driven image editing were widely explored, [3], [8], [16], [26], [19], [27], etc. Diffusion models in particular have been widely utilized for language driven natural image editing tasks, showcasing remarkable results. [24] enable text guided generation via a general cross-attention module. [11] inject text control into a diffusion generation process by manipulating the attention layers. [22] utilizes the clip latent space for text conditioned image generation. [2] allow users to specify a region in the image and replace the masked content with a diffusion process. [25] enables text guided generation by fine-tuning a diffusion model with a handful of examples and special tokens. [13] approaches the task by optimizing a latent code towards a target latent code generated by a diffusion model conditioned on the editing text. [4] train an edit diffusion model on editing paired data. More related work in this line of research include but not limited to [14], [28]. However, a shortcoming of these methods is they can typically only change the texture and geometry of objects in the image. They typically fail at fine-grained manipulation, such as editing the geometry or spatial relationships between objects whilst maintaining the identity of the elements in the image.

Related to the modification of icon images is the adjacent problem of conditioned generation of icons images [21, 30, 10, 5, 29]. These works utilize neural networks and learn to generate a sequence of Scalable Vector Graphics (SVG) operations (e.g. points, lines, circles) that can be executed to produce an image. Recent work [7, 12, 31] removes the need for direct SVG supervision by learning to generate icon images directly. The important problem of modifying existing icons to produce spatial variations is not addressed by these methods, as their foremost focus is on generation rather than editing. What’s more, these methods suffer the same problem as natural image editing methods: during generation the identity preservation of the original visual elements in the image is not guaranteed. More closely aligned with our work is Lillicon [9], that tries to solve the icon image editing problem. Lillicon uses transient widgets to select and manipulate features of icons to create their scale variations. However, their method does not take language as input, their system requires considerable user interaction, and the only variations created are limited to scale variations. In contrast, our method doesn’t require much user interaction and can create a wide range of different icon variations driven by language input.

In order to express editing requests as constraints on segments our DSL library is designed to contain two types of functions: (1) Constraint Specifiers for specifying constraints and (2) Compute Operators for accessing segments. The constraint specifiers are functions that directly specify either a motion type constraint or a geometry violation value, they are formatted as:

with $F_{m}\in\mathcal{F}$ represents motion constraint specifier and $F_{v}\in\mathcal{F}$ represents value constraint specifiers, $S_{i}\in\mathcal{S}$ and $S_{j}\in\mathcal{S}$ represents segment as input parameters, $v_{i}\in R$ and $v_{j}\in R$ represents scalar/vector as input parameters. The output scalar value $v_{o}\in R$ indicating how much the constraint is violated.

The compute operators are functions that can get attributes of the segments, or get the spatial relations between segments, they are formatted as:

with $F_{c}\in\mathcal{F}$ represents the compute operators, and $F_{v}\in\mathcal{F}$ represents value constraint specifiers, $S_{i}$ and $S_{j}$ represents segment as input parameters, $v_{i}\in R$ and $v_{j}\in R$ represents scalar/vector as input parameters. In order to make the DSL general enough to handle segments with different geometry, attributes of segments can only be accessed by using the compute operators, so that all the segment specific geometry computations are handled inside compute operators. These computations are implemented to be fully differentiable with respect to the segment transformation variables to enable gradient based optimization. What’s more, the functions are designed in a way so that they can be combined recursively to express constraints with complex structures. Please see table 1 for a partial overview of the functions in the library.

Given user’s edit request prompt $T$ and our DSL library $\mathcal{F}$, A pretrained LLM such as GPT-4 or GPT-4V is asked to convert the editing request $T$ using functions from the DSL library $\mathcal{F}$ into a set of the most critical primary constraints $\mathcal{C}^{1}$ which must be satisfied. To make LLM understand better the context of user’s request, we found that providing the image scene information can be crucial for many editing tasks. We represent the image scene as a list of segments, with each segment represented as (1) the semantic label (2) the four vertex positions of the segment’s axis-aligned bounding box. (3) Connection relations between segments. This information provides the LLM with the location, size, semantics, and interaction information of the segments in the image scene. We hide low-level geometry details from the LLM, such as each segments boundary path vertices, as we found empirically that it did not help with our task. We found that including several manually crafted question and answer examples as part of the prompt provided to the LLM can greatly improve the quality of LLM’s output and bias the LLM to output answers that are similar to the provided examples.

We observed that relying only on the primary constraints from the LLM can frequently result in either an over-constrained or under-constrained problem. We believe this is due to the lack of geometry understanding with current LLMs. To address this issue, we use the LLM to output the most crucial primary constraints (typically under-constraining), and use a geometry constraint detection method to automatically generate additional candidate secondary constraints $\mathcal{C}^{2}$. These candidate secondary constraints are crucial to preserve the logical and physical coherence of the scene. Given the initial arrangement of the segments, many potential relations can be detected. We choose to detect the most dominant spatial relations, namely: Inside, Contain, Overlap(Connect). Each has a weak and strong version. The weak version of each relation constraint is constrained by the function $overlap(S_{i},S_{j})$, $inside(S_{i},S_{j})$ and $contain(S_{i},S_{j})$ respectively, the strong version of each relation additionally includes the $coincide\_on\_point(S_{i},p,S_{j},p)$ constraint to indicate they are joined at a fixed point. Based on the two level hierarchy structure (ObjectName:PartName) of labels $\mathcal{L}$ provided along with the segmentation map, the above mentioned relations can be naturally categorized into intra-object relations $\mathcal{R}_{a}$ and inter-object relations $\mathcal{R}_{i}$. The preservation of intra-object relations ensures the integrity of the structure of objects, the preservation of inter-object relations ensures the perception consistency between input scene and edited scene. The intra-object relation $\mathcal{R}_{a}$ can be further categorized into crucial intra-object relations $\mathcal{R}^{*}_{a}$ and non-crucial intra-object relations $\mathcal{R}^{{}^{\prime}}_{a}$, for example, consider the basket example shown in fig 3, the relations between each handle and the basket body are crucial intra-object relations, the relations between the two handles are non-crucial intra-object relations as they are not real connections but due to the initial spatial arrangements. We use the LLM(GPT-4 Vision) to assign types to these intra-object relations. Note that not all secondary constraints need to be satisfied, as some initial relation constraints will prevent us from satisfying the primary constraints. We therefore design a search method to find a subset of secondary constraints to satisfy.

In this experiment, we compare our editing method against several LLM based editing methods that we implemented. gpt4/gpt4v-dm: Direct Motion parameters prediction , this method takes in the geometry information(boundary vertex positions) of segments, and ask GPT-4 or GPT-4V to output the affine transformation parameters for every segment. gpt4/gpt4v-dc: Direct Constraints prediction, this method takes in the geometry information(boundary vertex positions) of segments, and ask GPT-4 or GPT4Vision to output the all potential constraints expressed in our DSL. The GPT-4V version of these baselines additionally take in the colored segmented image as input to have more visual information. For all the experiments, the GT segmentation map is given. The relative 2D Chamfer Distance is computed as the Chamfer Distance between the boundary points of predicted edited segment and corresponding GT edited segment then divided by the size of the GT edited segment. Please see Table 2, Fig. 5 and Fig. 7 for the comparisons. The results demonstrate our method clearly outperforms all baselines by a large margin.

In this experiment, we compare our editing method against several state-of-the-art diffusion based image editing methods: sdi2i: Stable Diffusion Image2Image  [24] , ip2p: InstrauctPix2Pix  [4], imagic: Imagic  [13], for each of these methods, we first apply the methods (as implemented by Hugging Face) directly on the input icon image to produce results. Additionally, we perform comparisons on variations of these methods. Considered that these methods are mostly trained on natural images and do not take semantic segmentation as input (which our methods have access to), we try to partially address this for a fairer comparison. We designed a lift-and-project variation of these methods, this process uses the labels from the segmentation along with the edge map of the input image to generate a more realistic image using ControlNet [32], then we apply these editing methods to edit the generated images, we then use GroundedSam [23] to detect the shape of the labels and rasterize them back to an icon image for error computation, please see the supplementary material for more details. Method names with an ”L” appended (sdi2iL, ip2pL and imagicL) indicates the lift-and-project version is used. We compute the image space mean-squared-error between the edited image and the GT image. The Chamfer Distance used in the previous experiment cannot be used here because we do not have the paired segments between the original image and the edited image. Please see Table 3 and Fig. 6 for comparisons. The results show that our method outperforms all baselines by a large margin.

We study the effectiveness of our state search components described in Section 6 and report these results in Table 4 and Fig. 4. We find that without relation constraint search, where all the candidate relation constraints exist, the optimization can become over-constrained and prevent us from achieving the editing goal. Without motion search, some segments are unnecessary modified which leads to less preservation of object features.

Please see the Fig. 12 for the editing results of our full automatic pipeline. In this experiment, the only input is the icon image and the editing request, without any GT segmentation or label provided. The segmentation is first automatically predicted by our segmentation tool and then used with the editing pipeline. As you can see, our full automatic pipeline can produce reasonable editing results for some images and its performance can be further improved when segmentation tool pipeline it rely on gets improved. Please see the supplementary material for the discussion of how to evaluate our segmentation tool and how much it can reduce the manual effort required from users.

The main failure modes are associated with some fundamental challenges of the 2D icon editing task: One failure mode occurs when the editing request is too abstract resulting in the LLM failing to output the correct constraints. Another failure mode occurs when the icon image depicts a perspective rendered 3D scene where the spatial relations become too complex. Refer to the supplementary material for extended discussion of limitations.

Please see Fig, 11 for multiple editing strategies(given by LLM) of the same editing request and Fig. 10 for results with segment completion enabled. Please see the supplementary material for more results, such as re-use of the same segmentation for multiple requests, examples of LLM constraint formation, and more qualitative comparison results.

Given the initial arrange of the segments in the scene, some segments $S_{i}$ may be occluded by other segment $S_{j}$, as the consequence of this occlusion, the geometry information of the occluded region in segment $S_{i}$ is lost. During the editing, if the front segment $S_{j}$ is moved to a different location, the occluded region in segment $S_{i}$ will become visible. In such case, user may want the missing geometry to be filled for the naturalness of the scene. In order to do so, we designed a segment completion method and allow user to choose if some guessed geometry should be generated and appended to the occluded segments. The method start by extracting the boundary edge map of the occluded segment $S_{i}$ and feed it into a pretrained ControlNet  [32] together with its semantic label $l_{i}$ to generate multiple images of the guessed complete version of segment $S_{i}$. Then a pre-trained label detection model such as GroundedSam or others  [23, 15, 20, 18, 17] is used to extract the shape of the complete version of $S_{i}$, a voting procedure is then applied to choose the shape that is closet to the mean geometry of all guessed shapes as the final complete shape of $S_{i}$. Please see fig 14 for the illustration of the method.

Please see algo 2 for more details of the optimization process. Two state matrices $\mathbf{M}_{M}$ for motion states and $\mathbf{M}_{R}$ for relation states are initialized according to the LLM’s primary constraint specification. In the relation search stage, the front edges are firstly identified, then the combinations of the front edges are iterated, for each combination, the FlipRelation() flip the relations states as $\mathbf{M}^{(1)}_{R}$ : The inter-object relations are set from Strong(ST) to Weak(WK). $\mathbf{M}^{(2)}_{R}$ : The inter-object relations are set from Weak to Not Exist. $\mathbf{M}^{(3)}_{R}$ : The non-crucial intra-object relations are set from Strong(ST) to Not Exist(N). $\mathbf{M}^{(4)}_{R}$ : The crucial intra-object relations are set from Strong(ST) to Weak(WK). The Solve() is then applied to evaluate the flipped states in order, the score outputs from $Solve()$ is the violation value of all primary constraints and secondary constraints set to the flipped state. Each flipped state inherit the value from the previous state. Optional Early stop can be performed if the score is low enough. If not stopped, the front edges are updated to non-visited neighbor edges and repeat the process. In the motion search stage, the front nodes are firstly identified, then for each segment in the front nodes, the FlipMotion() flips the motion states as : $\mathbf{M}^{(1)}_{M}$ : the motion type is set to Translate+Rotate+Scale(TRS), $\mathbf{M}^{(2)}_{M}$ : the motion type is set to Translate+Rotate(TR), $\mathbf{M}^{(3)}_{M}$ : the motion type is set to Translate+Scale(TS), $\mathbf{M}^{(4)}_{M}$ : the motion type is set to Translate(T), $\mathbf{M}^{(5)}_{M}$ : the motion type is set to Not Exist(N). If not stopped, the front nodes are updated to non-visited neighbor nodes and repeat the process. Once both search stages end, the output optimized segments are the final edited segments.

Under the hood of $Solve()$, the constraints input to the solver will be firstly converted into a constraint tree, with leaf nodes as segments or constant values, with the non-leaf nodes as operations that act on one or multiple segments or values (its children nodes). The root node always corresponds to one of the constraint specifier, this is to ensure the final output value of every constraint indicates how much this constraint is violated. During the optimization, the solver will perform a post-order traversal of the tree to compute the final constraint violation value as a loss value. We use Adam optimizer with a gradually decreasing(starting from lr=10) learning rate for optimizing the affine transformation variables. Each optimization run takes maximum 150 iterations. Additionally, we can choose to create a proxy segment $S^{{}^{\prime}}_{i}$ for each segment $S_{i}$ to run the optimization, the proxy segment is constructed by a ray-shooting process, for each pixel, we shoot K 2D rays(uniformly spaced around the circle centered on the pixel) outward from the pixel, if a majority of these rays(ratio¿=0.9) hit the original segment, the pixel is marked as part of the proxy segment. With this process, we obtained a set of pixels used to create the proxy segment. One benefit of doing so is to make the inside check of segments be consistent with our DSL (so that anything inside the inner holes of an segment is still considered as inside of the element).

As described in the main paper, once the transformation for segments are updated. The final step is to re-render them back to an image. The depth ordering of the segments will be crucial for the re-rendering of the edited segments. We developed an algorithm to decide the ordering relation between two segments given their initial spatial configurations. Given two segments $S_{i}$ and $S_{j}$, Our method starts by extracting the boundary region $b$ around the intersection region between $S_{i}$ and $S_{j}$, then for each segment, we compute the farthest path length $L(b)$ between any two points in the boundary region $b$. The path must be formed inside the segment, for each point pair $(p_{s},p_{e})\in b$, we run a breath-first-search(BFS) from $p_{s}$ to $p_{e}$, the number of search steps will be saved as the path length for $(p_{a},p_{b})$. the longest path length for segment $S_{i}$ is indicated as $Length(b_{i})$ and the longest path length for segment $S_{j}$ is indicated as $Length(b_{j})$. The shorter path indicates that this object is convex and extruding into the other object, that we usually perceive as this object occluded the other object. This rule is summarized as the following, If $Length(b_{i})>Length(b_{j})$ : segment i is occluded by segment j, If $Length(b_{i})<Length(b_{j})$ : segment j is occluded by segment i, If $Length(b_{i})\sim Length(b_{j})$ : The depth order is not determined. Once the depth ordering of segments are determined, we can raster the segments into a color image buffer with the help of a depth image buffer. Please see fig 15 for illustration.

The output labelled segments of the segmentation tool pipeline described in the main paper may not always be well aligned with user’s understanding of the image scene. This is caused by multiple sources: (1) The label generated by GPT4Vision may not be accurate, (2) The ControlNet generated image may not be an accurate reflection of the input image scene, (3) The performance of label grounding models such as GroundedSam or GLIP is still far from ideal. As a result, some regions are over-split while others are under-split, and the assigned labels may also need to be adjusted. Based on these observations, we designed some metrics to measure how much our segmentation tool can close the gap between raw input and GT segmentation, in other words, how much our tool can help the user to spend less manual effort on labeling and segmentation. We designed two metrics for evaluating the amount of effort that user may need to manually spent based on the auto-generated labelled segments from our tool. We assume that a simple segmentation and label fixing tool would allow users to do two operations: (1) Drawing a path in the image indicating the boundary path of a segment, (2) Changing an incorrect label assigned to one segment into a different label. As you can see, in the case of under-split, user can draw additional paths on a segment to reach the desired split. In the case of over-split, user can changing the labels of adjacent segments so that they are merged as one segment. In the case of incorrect label, only changing the label is enough. As the consequences of these two fixing operations, two evaluation metrics are naturally emerged: (1) The length of the additional boundary paths drawn on top of the predicted boundary paths. (2) The number of label correcting operation. Please see fig 16 for the illustration of these metrics. For metric (1), we compute it by measuring how many GT segment boundary pixels are not covered by predicted segment boundary pixels, for each GT segment boundary pixel, we search if there is a predicted boundary pixel within a radius(r = 20) of this GT pixel. For metric (2), we count the number of mismatched labels assigned to predicted segments and GT segments. We also include the numbers of these two metrics computed on the original input (indicates the total manual effort a user need to spent given the original unsegmented image) for comparison. Please see table 5 for this comparison. As you can see, the path need to be drawn(fix) become much shorter after using our segmentation tool, the label correcting numbers are slightly higher but it is a relative cheap operation. In summary, our segmentation tool can greatly reduce the user’s manual effort for creating the labelled segmentation that is required by the editing process.

Another thing to notice is that an edit can be achieved with different segmentation results, please see fig 17 for illustration of the editing results generated by our editing system with different segmentation configuration.

Please see fig 21 for additional comparison results against different GPT based editing methods. Please see fig 22 for additional comparison results against different diffusion based editing methods. Please see fig 23 for additional ablation comparison results with different components removed from our editing system. Please see fig 24 for the editing results of how the same segmentation map can be re-used multiple times for different editing text prompt. Please see fig 25 and 26 for additional results for our segmentation tool. Please see table 6 for examples of what it looks like for the GPT to output constraints using our designed DSL.

We observed several failure modes of our method. The first failure case occur when the editing request is too abstract so that the spatial relation is only implicitly specified. For example in the first editing task in fig 20, the LLM had a hard time associating the ”dig” action with our DSL operators which then leads to inaccurate constraints. The second relative rare failure case is caused by sub-optimal hyper-parameters in the optimization method, and can lead to less desired result (the second task in fig 20). The third case is caused by inaccurate primary constraints generated by LLM, in the third task in fig 20, the LLM thinks the ”length” in the editing request meant to be the vertical length of the handle rather than the principal length of the handle. We expect that as the performance of LLMs advanced in the future, some of these failure modes will be mitigated. Another type of failure modes is that when the input icon image contains many perspective rendered 3D objects so that the spatial relations of the objects can become quite complex and goes beyond the capability of our DSL and scene representation.