Towards Book Cover Design via Layout Graphs

The input of the Layout Generator is a layout graph, which is a directed graph with each object $o$ represented by a node $\mathbf{n}_{o}=(\mathbf{c}_{o},\mathbf{l}_{o})$, where $\mathbf{c}_{o}$ is a class vector and $l_{o}$ is the location vector of the object. The class vector contains a 128-dimensional embedding of the class of the object. The location vector $\mathbf{l}_{o}$ is a 35-dimensional binary vector that includes the location and size of the object. The first 25 bits of $\mathbf{l}_{o}$ describe the location of the object on a $5\times 5$ grid and the last 10 bits indicate the size of the desired object on a scale of 1 to 10.

The edges of the layout graph are the positional relations between the objects. Each edge $\mathbf{e}_{o,p}$ contains a 128-dimensional embedding of six relationships between every possible pairs of nodes $o$ and $p$. The six relationships include “right of”, “left of”, “above”, “below”, “surrounding” and “inside”.

The layout graph is fed to a GCN [Scarselli_2009], $E_{\mathrm{graph}}$, to learn an embedding $\mathbf{m}_{o}$ of each object $o$. Where a traditional Convolutional Neural Network (CNN) [lecun1998gradient] uses a convolution of shared weights across an image, a GCN’s convolutional layers operate on graphs. They do this by traversing the graph and using a common operation on the edges of the graph.

To construct the GCN, we take the same approach as Johnson et al. [Johnson_2018] which constructs a list of all of the nodes and edges in combined vectors $\mathbf{v}$ and then uses a multi-layer perceptron (MLP) on the vectors, as shown in Fig. 3. Vector $\mathbf{v}$ consists of a concatenation of an edge embedding $\mathbf{e}_{o,p}$ and the two adjacent vertices $o$ and $p$ and vertex embeddings $\mathbf{n}_{o}$ and $\mathbf{n}_{p}$. The GCN is consists of two sub-networks. The GCN (Edge) network in Fig. 3a takes in vector $\mathbf{v}$ and then performs the MLP operation. The output is then broken up into temporary object segments $\mathbf{n}^{\prime}_{o}$ and $\mathbf{n}^{\prime}_{p}$ and further processed by individual GCN (Vertex) networks for each object. The result of GCN (Vertex) is a 128-dimensional embedding for each object, which is used by the subsequent Box Regression Network and Mask Generator.

The purpose of the Mask Generator is to generate a mask of each isolated object for the Appearance Generator. The Mask Generator is based on a CNN. The input of the Mask Generator is the object embedding $\mathbf{m}_{o}$ learned from the GCN and the output is a $32\times 32$ shape mask of the target object. This mask is only the shape and does not include size information. Furthermore, since the Mask Generator creates detailed masks, a variety of shapes should be used. To do this, a 64-dimensional random vector $\mathbf{z}_{o}$ is concatenated with the object embedding $\mathbf{m}_{o}$ before being given to the Mask Generator.

In order to produce realistic object masks, an adversarial Mask Discriminator $D_{\mathrm{mask}}$ is used. The Mask Discriminator is based on a conditional Least Squares GAN (LS-GAN) [mao2017least] with the object class $s_{o}$ as the condition. It should be noted that the object class $\mathbf{s}_{o}$ is different than the 128-dimensional class vector $\mathbf{c}_{o}$ in the layout graph. The GAN loss $\mathcal{L}^{D}_{mask}$ is:

where $G_{\mathrm{mask}}$ is the Mask Generator and $\mathbf{T}_{o}$ is a real mask. Accordingly, the Mask Discriminator $D_{\mathrm{mask}}$ is trained to minimize $-\mathcal{L}^{D}_{mask}$.

The Box Regression Network generates a bounding box estimation of where and how big the object should be placed in the layout. Just like the Mask Generator, the Box Regression Network receives the object embedding $\mathbf{m}_{o}$. The Box Regression Network is an MLP that predicts the bounding box $\mathbf{b}_{o}=\{(x_{0},y_{0}),(x_{1},y_{1})\}$ coordinates for each object $o$.

To generate the layout, the outputs of the Mask Generator and the Box Regression Network are combined. In order to accomplish this, the object masks from the Mask Generator are shifted and scaled according to bounding boxes. The shifted and scaled object masks are then concatenated in the channel dimension and used with the Appearance Generator to create a layout feature map $F$ for the Book Cover Generator.

The objects’ appearances that are bound by the mask are provided by the Appearance Generator $G_{\mathrm{app}}$. The Appearance Generator is a CNN that takes real images of cropped objects of ($64\times 64\times 3)$ resolution and encodes the appearance into a 32-dimension appearance vector. The appearance vectors $\mathbf{a}_{o}$ represent objects within the same class and changing the appearance vectors allows the appearance of the objects in the final generated result to be controlled. This gives the network to provide a variety of different object appearances even with the same layout graph. A feature map $\mathbf{F}$ is created by compiling the appearance vectors to fill the masks that were shifted and scaled by the bounding boxes.

The Book Cover Generator $G_{\mathrm{book}}$ is based on a deep Residual Network (ResNet) [he2016deep] and it generates the final output. The network has three parts. The first part is the contracting path made of strided convolutions which encodes the features from the feature map $\mathbf{F}$. The second part is a series of 10 residual blocks and the final part is an expanding path with transposed convolutions to upsample the features to the final output image $\mathbf{I}$.

In order to enhance the quality of output of the Book Cover Generator a Perception Network is used. The Perception Network $P_{\mathrm{content}}$ is a pre-trained very deep convolutional network (VGG) [simonyan2014very] that is only used to establish a perceptual loss $\mathcal{L}^{P}_{\mathrm{content}}$ [johnson2016perceptual]. The perceptual loss:

is the content consistency between the extracted features of the VGG network $P_{\mathrm{content}}$ given the generated layout image $I$ and a real layout image $R$. In Eq. (2), $u$ is a layer in the set of layers $\mathcal{U}$ and $P^{(u)}_{\mathrm{content}}$ is a feature map from $P_{\mathrm{content}}$ at layer $u$.

The Layout Discriminator $D_{\mathrm{layout}}$ is a CNN used to judge whether the layout image $\mathbf{I}$ appears realistic given the layout $\mathbf{F}$. In this way, through the compound adversarial loss $\mathcal{L}_{\mathrm{layout}}$, the generated layout will be trained to be more indistinguishable from images of real layout images $\mathbf{R}$ and real layout feature maps $\mathbf{Q}$. The loss $\mathcal{L}_{\mathrm{layout}}$ is defined as:

where $\mathbf{Q}^{\prime}$ is a second layout with the bounding box, mask, and appearance attributes taken from a different, incorrect ground truth image with the same objects. This is used a poor match despite having the correct objects. The aim of the Layout Discriminator is to help the generated image $\mathbf{I}$ with ground truth layout $\mathbf{Q}$ to be indistinguishable from real image $\mathbf{R}$.

The Book Cover Discriminator is an additional discriminator that is used to make the generated image look more like a book. Unlike the Layout Discriminator, the Book Cover Discriminator only compares the generated image $\mathbf{I}$ to random real book covers $\mathbf{B}$. Specifically, an adversarial loss:

where $D_{\mathrm{book}}$ is the Book Cover Discriminator, is added to the overall loss.

The Object Discriminator $D_{\mathrm{obj}}$ is another CNN used to makes each object images look real. $\mathbf{i}_{o}$ is an object image cut from the generated image by the generated bounding box and $\mathbf{r}_{o}$ is a real crop from the ground truth image. The object loss $\mathcal{L}_{\mathrm{obj}}$ is:

The entire Layout Generator with all the aforementioned networks are trained together end-to-end. This is done using a total loss:

where each $\lambda$ is a weighting factor for each loss. In addition to the previously described losses, Eq. (6) contains a pixel loss $\mathcal{L}_{\mathrm{pixel}}$ and two additional perceptual losses $\mathcal{L}^{P}_{\mathrm{mask}}$ and $\mathcal{L}^{P}_{\mathrm{layout}}$. The pixel loss $\mathcal{L}_{\mathrm{pixel}}$ is the L1 distance between the generated image $\mathbf{I}$ and the ground truth image $\mathbf{R}$. The two perceptual losses $\mathcal{L}^{P}_{\mathrm{mask}}$ and $\mathcal{L}^{P}_{\mathrm{layout}}$ are similar to $\mathcal{L}^{P}_{\mathrm{content}}$ (Eq. 2), except instead of a separate network, the feature maps of all of the layers of discriminators $D_{\mathrm{mask}}$ and $D_{\mathrm{layout}}$ are used, respectively.

Along with the scene objects, the title text and the solid region can be moved on the layout graph. Fig. 8 shows examples of generated book covers with the same layout graph except for the “Solid” nodes. By moving the “Solid” node to have different relationship edges with other nodes, the solid regions can be moved. In addition, multiple “Solid” nodes can be added to the same layout graph to construct multiple solid regions.

Due to each node in the layout graph containing its own appearance vector, different variations of generated book covers can be created from the same layout graph. Fig. 9 shows a layout graph and the effects of changing the appearance vector of individual nodes. In the figure, only one node is changed and all the rest are kept constant. However, even though only one element is being changed in each sub-figure, multiple elements are affected. For example, in Fig. 9 (c) when changing the “Grass” node, the generated grass area changes and the model automatically changes the “Solid” and “Sky” regions to match the appearance of the “Grass” region. As it can be observed from the figure, the solid bar on the left is normally contrasted from the sky and the grass. This happens because each node is not trained in isolation and the discriminators have a global effect on multiple elements and aim to generate more realistic compositions.