Text2Street: Controllable Text-to-image Generation for Street Views

In recent years, many methods [17, 24, 22, 23, 14, 25, 27, 2, 7, 34] have been dedicated to dealing with the task of general text-to-image generation. For example, AlignDRAW [17] iteratively draws patches on a canvas, while attending to the relevant words in the description. GAWWN [24] synthesizes images given instructions describing what content to draw in which location based on generative adversarial networks [9]. DALLE [22] describes a simple approach for this text-to-image task based on a transformer that autoregressively models the text and image tokens as a single stream of data. DALLE2 [23] proposes a two-stage model: a prior that generates a CLIP [21] image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. DDPM [14] presents high quality image synthesis results using diffusion models [29], a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Stable Diffusion [25] applies diffusion models training in the latent space of pretrained autoencoders, and turns diffusion models into powerful and flexible generators for general conditioning inputs by introducing cross-attention layers into the model architecture. These methods have garnered remarkable results in general text-to-image generation. However, their effects in street-view text-to-image tasks is not as commendable.

There has been a recent surge in the study of methods for street-view image generation. For example, SDM [31] processes semantic layout and noisy image differently. It feeds noisy image to the encoder of the U-Net [26] structure while the semantic layout to the decoder by multi-layer spatially-adaptive normalization operators. BEVGen [30] synthesizes a set of realistic and spatially consistent surrounding images that match the bird’s-eye view (BEV) layout of a traffic scenario. BEVGen incorporates a novel cross-view transformation with spatial attention design which learns the relationship between cameras and map views to ensure their consistency. GeoDiffusion [6] translates various geometric conditions into text prompts and empower pre-trained text-to-image diffusion models for high-quality detection data generation and is able to encode not only the bounding boxes but also extra geometric conditions such as camera views in self-driving scenes. BEVControl [32] proposes a two-stage generative method that can generate accurate foreground and background contents. These methods typically require the input of BEV maps, object bounding boxes, or semantic masks to generate images. However, there is almost no research on generating street-view images relying solely on text. In this paper, we primarily focus on resolving the issue of street-view text-to-image generation.

Text2Street takes a street-view description prompt (e.g., “a street-view image with the crossing, 3 lanes, 4 cars and 1 truck driving on a sunny day”) as input and generates a corresponding street-view image. Prior to the main process, the input prompt is parsed by a large language model (e.g., GPT-4 [20]) to extract descriptions of road topology, traffic status, and weather conditions, which are then fed into three main components. The first component is the lane-aware road topology generator, which takes the road topology description (“crossing, 3 lanes”) as input and produces a local semantic map. The second component is the position-based object layout generator, which takes the traffic object description from the traffic status (“4 cars and 1 truck”) as input and generates traffic object layout. The third component is the multiple control image generator, which takes road topology, object layout, and weather condition descriptions (“a sunny day”) as input, and outputs an image that matches the original street-view description prompt.

For Stable Diffusion [25], directly generating images that comply with road topology, including road structure and lane topologies, is difficult. To address this, we introduce the lane-aware road topology generator (LRTG), as shown in Fig. 3. This generator does not directly produce road images; instead, it first creates a local semantic map describing the road structure, representing a complete regional-level road structure, including drivable areas, intersections, sidewalks, zebra crossings, etc. Simultaneously, to ensure the generated lane lines adhere to traffic regulations (i.e., equidistant and parallel lanes), we characterize and generate lane lines on the semantic map, which is easier and more controllable than generating lane lines directly on perspective-view images. Furthermore, to ensure the number of lane lines aligns with the provided text, we incorporate a counting adapter for the precise generation of a specified number of lane lines. In LRTG, we only generate the semantic map, which serves as a crucial intermediary for street-view images, as further detailed in Section 3.4.

When generating local semantic maps, we utilize Stable Diffusion to encode road topology descriptions based on the CLIP [21] text encoder. Subsequently, the encoded input is then fed into cross-attention layers of U-Net [26] to denoise image latents, ultimately outputting the corresponding semantic map. Consistent with Stable Diffusion, the learning objective is as follows:

$\displaystyle\mathcal{L}_{SD}=\mathbb{E}_{\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}(z_{t},t,\tau(y))\right\|^{2}_{2}\right],$

(1)

where $x\in\mathbb{R}^{H\times W\times 3}$ is the given images cropped from labeled semantic maps in RGB space, $\mathcal{E}(\cdot)$ refers to the encoder of pretrained autoencoders [8] and $z=\mathcal{E}(x)$ represents encoded image latents, $z_{t}$ is from the forward diffusion process at the timestep $t$, $y$ is the text prompt and $\tau(\cdot)$ represents the pretrained CLIP text encoder, the term $\epsilon$ denotes the target noise, and $\epsilon_{\theta}(\cdot)$ signifies the time-conditional U-Net used for predicting the noise. This manner ensures the reasonable generation of road structures and lane line shapes within the semantic map.

For achieving precise control over the number of lane lines, the counting adapter $f_{CA}$ gathers attention scores from all cross-attention layers of the U-Net. These scores are subsequently reshaped to match the same resolution and then averaged to yield attention features for all tokens. From these attention features, the ones corresponding to tokens “lane lines” are selected. These selected features $\mathcal{F}_{l}$, undergo further processing through two convolutional layers with the kernel of $3\times 3$, followed by one fully connected layer, which serves to predict the number of lane lines $N_{l}$. The learning objective for achieving precise control over the number of lane lines is as follows:

$\displaystyle\mathcal{L}_{CA}=\left\|N_{l}-f_{CA}(\mathcal{F}_{l})\right\|^{2}_{2}.$

(2)

Based on Eq. 1 and 2, LRTG can be jointly optimized to generate the local semantic map, encompassing both road structure and lane lines as required.

To ensure that generated images can depict diverse traffic conditions, we utilize the large language model to convert traffic status into the number of traffic objects (e.g., car, truck, pedestrian, etc). Then, the position-based object layout generator (POLG) is proposed to create an object layout based on the text description of object quantity, as demonstrated in Fig. 4. To guarantee a specified number of objects are generated, we incorporate an object-level bounding box diffusion strategy to generate positions of object bounding boxes. Simultaneously, to ensure the generated traffic objects comply with traffic rules, we incorporate the local semantic map from the LRTG into the box diffusion process. With POLG, we generate layout information for traffic objects, which also serves as an intermediary for generating the final street-view images, as introduced in Section 3.4.

In the bounding box diffusion strategy, we first represent traffic objects as position vectors $\mathcal{O}_{i}=\left[x_{i},y_{i},z_{i},l_{i},w_{i},h_{i},\zeta_{i},c_{i}\right]$ ($i=1,2,...,N_{o}$, $N_{o}$ is the number of objects), where $x_{i},y_{i},z_{i}$ denote the coordinate of object position, $l_{i},w_{i},h_{i}$ represent the object’s size with length/width/height, $\zeta_{i}$ signifies the object’s yaw angle, and $c_{i}$ indicates the object’s category. Subsequently, the position vector is diffused based on diffusion models DDPM [14]. Furthermore, to ensure objects adhere to traffic regulations (such as cars must be driven on the road and not against traffic), we use ControlNet [33], incorporating the local semantic map from LRTG as a control into the POLG. Ultimately, the learning objective is as follows:

$\displaystyle\mathcal{L}_{POLG}=\mathbb{E}_{o,m,\epsilon,t}\left[\left\|\epsilon-\epsilon_{\theta}(o_{t},t,\mathcal{C}(m))\right\|^{2}_{2}\right],$

(3)

where $o$ represents the position vectors of the objects, $o_{t}$ is from the forward diffusion process at the timestep $t$, $m$ denotes the local semantic map, and $\mathcal{C}(\cdot)$ signifies the ControlNet. And other symbols are consistent with those in Eq. 1.

Based on Eq. 3, the layout information of traffic objects that meet the traffic status can be optimized and generated through POLG based on textual descriptions.

To produce images with realistic weather that align with road topology and traffic status, we introduce the multiple control image generator (MCIG), as depicted in Fig. 5.

To effectively utilize the previously generated local semantic map and traffic object layout, camera pose sampling and image projection are conducted before these two pieces of information enter MCIG. This results in a 2D road semantic mask $\mathcal{M}_{r}$ and traffic object layout map $\mathcal{M}_{o}$ from a perspective view, as shown in Fig. 2. The 2D traffic object layout maps are also represented as 2D traffic object position vectors $\mathcal{P}=\{\mathcal{P}_{i}\}_{i=1}^{N_{o}}=\{\left[x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i},c_{i}\right]\}_{i=1}^{N_{o}}$. The projection uses a conventional method based on intrinsic and extrinsic transformation, where the intrinsic parameters use fixed camera parameters, and the extrinsic parameters are sampled near the prior camera height.

As depicted in Fig. 5, MCIG comprises five modules: object-level position encoder, text encoder, semantic mask ControlNet, object layout ControlNet, and naive Stable Diffusion. The first four modules control image generation based on four different types of information, i.e., 2D traffic position vectors, text describing the weather, 2D road semantic masks, and 2D traffic object layout maps.

The object-level position encoder encodes the 2D traffic object position vectors, including 2D bounding boxes and object categories, represented as:

$\displaystyle\mathcal{PE}(\mathcal{P})=f_{\mathcal{PE}}(\mathcal{BE}\otimes\mathcal{CE}).$

(4)

The box encoder maps object bounding boxes to a higher-dimensional space, ensuring that the network can learn higher-frequency mapping functions and focus on the positions of each object. Specifically, the box encoder is an encoding function based on sine and cosine. The mathematical form of the encoding function is as follows:

$\displaystyle\mathcal{BE}(p)=\left[\cdots,\text{sin}(2^{l}\pi p),\text{cos}(2^{l}\pi p),\cdots\right]^{L-1}_{l=0},$

(5)

where $\mathcal{BE}(\cdot)$ is applied to each component of the box (i.e., $x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i}$) of each object $\mathcal{P}_{i}$, and $L$ is empirically set to 10. Simultaneously, the category encoder $\mathcal{CE}$ employs the CLIP text encoder to encode the object category (e.g., “car”). Subsequently, the box encoding and category encoding are concatenated at the feature embedding dimension of each object. The concatenated features are then mapped to features with the same dimension as the original text encoder’s embedding through a two-layer fully connected network $f_{\mathcal{PE}}(\cdot)$, serving as position embeddings. The text encoder, based on the CLIP text encoder, encodes the weather description text $\mathcal{T}$, resulting in text embeddings.

The object position encodings and weather text embeddings, upon concatenation at the token dimension, are fed into the cross-attention layer of Stable Diffusion, individually controlling the object position and weather during the image generation. Simultaneously, the semantic mask ControlNet and object layout ControlNet employ two similar ControlNets, utilizing images (i.e., semantic masks and layout maps) as inputs to control the road topology and object layout during the street-view image generation. The learning objective function of MCIG is as follows:

		$\displaystyle\mathcal{L}_{MCIG}=$		(6)
		$\displaystyle\mathbb{E}_{\mathbb{P}}\left[\left\|\epsilon-\epsilon_{\theta}(z_{t},t,\mathcal{PE}(\mathcal{P}),\tau(\mathcal{T}),\mathcal{C}(\mathcal{M}_{r}),\mathcal{C}(\mathcal{M}_{o}))\right\|^{2}_{2}\right],$

where $\mathbb{P}$ is the set of $\{\mathcal{E}(x),\mathcal{P},\mathcal{T},\mathcal{M}_{r},\mathcal{M}_{o},\epsilon,t\}$ for convenience of presentation.

Through the optimization of MCIG using Eq. 6, we obtain street-view images that conform to the initial prompt about road topology, traffic status, and weather conditions.

For image-level evaluation, we use Fréchet Inception Distance (FID) $S_{\text{FID}}$ [13] to measure image fidelity, and CLIP score $S_{\text{CLIP}}$ [12] to image-text alignment. Please refer to related works [13, 12] for computation details.

In the attribute-level evaluation, we primarily measure the accuracy of text-to-image street-view generation in four aspects: road structure, lane line counting, traffic object counting, and weather conditions. For these four metrics, we train four neural networks on nuScenes dataset to evaluate scores of generated images. Specifically, a two-class classifier based on ResNet-50 [11] is trained for road structure accuracy $S_{\text{road}}$ to distinguish whether the road structure in street-view RGB images is an “intersection” or “non-intersection”. For the accuracy of lane line counting $S_{\text{lane}}$, a six-class classifier is similarly trained on ResNet-50 to distinguish whether the number of lane lines in street-view RGB images is equal to 0, 1, 2, 3, 4, or $\geq 5$. For the accuracy of traffic object counting $S_{\text{obj}}$, a object detector based on YOLOv5 [15] is trained to evaluate the number of traffic objects in street-view RGB images. For the accuracy of weather conditions $S_{\text{wea}}$, a four-class classifier is also trained on ResNet-50 to distinguish whether the weather conditions in street-view RGB images are sunny day, sunny night, rainy day, or rainy night. All models are trained on nuScenes training dataset and used as evaluation metrics for attribute-level evaluation of street-view image generation.

We compare our approach with several state-of-the-art algorithms in text-to-image generation, including Stable Diffusion [25], Stable Diffusion 2.1 [1] and Attend-and-Excite [5] on nuScenes validation dataset, as listed in Tab. 1. These methods are all finetuned on nuScenes training dataset. Note that we have also listed the performance on the nuScenes validation dataset as the “Reference”.

Comparing the proposed method with state-of-the-art methods, we can see that our method consistently outperforms other methods across almost all metrics from Tab. 1. Significantly, our method outperforms all others on attribute-level metrics (i.e., $S_{\text{road}}$, $S_{\text{lane}}$, $S_{\text{obj}}$ and $S_{\text{wea}}$), demonstrating its superior controllability for fine-grained text-to-image street-view image generation. Specifically, our method shows a obviously 4.50%, 14.91% improvement on metric $S_{\text{lane}}$ and $S_{\text{obj}}$ compared to the second best performance. Additionally, our method also performs better on image-level metrics (i.e., $S_{\text{FID}}$ and $S_{\text{CLIP}}$), reflecting its superior overall generation quality and image-text consistency. Overall, these observations validate the effectiveness of our proposed method for controllable image generation for street views.

Visual examples generated by our method are illustrated in Fig. 6. From Fig. 6, it is evident that our method yields superior results in dealing with varying road structures (1st and 4th rows), different numbers of lane lines (1st and 3rd rows), diverse numbers of traffic objects (1st and 2nd rows), and various weather conditions (2nd and 3rd rows) compared to other methods. This indicates that our method can effectively generate street-view images only based on text, and also implies its controllability and superiority in street-view text-to-image generation.

To assess the effectiveness of individual components, we carry out ablation experiments on nuScenes validation dataset, comparing the performance variations within the proposed approach.

Fistly, to validate the effectiveness of the lane-aware road topology generator (LRTG), we introduced three models for ablation comparison. The first model, termed as “Baseline”, is a naive multiple control image generator (MCIG) only with only the text encoder, which actually is a Stable Diffusion model. The second model, named “A₁”, is based on the “Baseline” with the addition of LRTG excluding lane line control. The third model, “A₂”, adds LRTG with lane line control to the first model. The comparison of these three models is presented in the first three rows of Tab. 2. It can be observed that the introduction of road structure control (“A₁”) significantly improves the $S_{\text{road}}$ metric, and the addition of both road structure and lane lines (“A₂”) controls further enhances both $S_{\text{road}}$ and $S_{\text{lane}}$ metrics. This confirms the effectiveness of LRTG in controlling road topology.

Secondly, to validate the effectiveness of the position-based object layout generator (POLG), we add POLG to “Baseline”, termed as “B”. Comparing the first and fourth rows of Tab. 2, it is evident that the inclusion of POLG significantly improves the metric $S_{\text{obj}}$, demonstrating the control capability of POLG in traffic object generation.

Thirdly, to verify the compatibility of different modules, we also list the model “C” (i.e., Text2Street), which combines all three modules. As can be seen from the last row of Tab. 2, “C” achieves the best performance across all metrics, confirming the compatibility among different modules.

To demonstrate the utility of street-view text-to-image generation for downstream tasks, we select object detection as a representative task. We use the proposed Text2Street to generate 30,000 images based on random prompts as a supplement to the original training data to train YOLOv5 on the nuScenes dataset, as listed in Tab. 3. The results indicate that the images generated by our method are beneficial for downstream street-view tasks, highlighting the potential of the street-view text-to-image generation.

In addition to street-view text-to-image generation, our approach also allows for modifications to local semantic maps, object layouts, or text, enabling the editing of road structures, lane lines, object layouts, and weather conditions in the originally generated RGB images, as depicted in Fig. 7.