Automated Creative Optimization for E-Commerce Advertising

Neural architecture search is firstly proposed in (Zoph and Le, 2017), where reinforcement learning is employed for searching architectures. After then, more search-based algorithms have been proposed (e.g., evolution algorithm (Liu et al., 2018b), greedy search (Liu et al., 2018c)). However, these algorithms cost much search time due to the huge and discrete search space. Recently, the one-shot architecture search algorithms achieve huge success depending on the continuous search space and greatly shorten the time, such as DARTS (Liu et al., 2018a), SNAS (Xie et al., 2018), NASP (Yao et al., 2020b). These algorithms jointly optimize the weights for different operators and model parameters by stochastic gradient descent.

Following SIF (Yao et al., 2020a), we relax the choices among operations as a sparse vector in a continuous space. For each pair of feature fields $i$ and $j$, $\bm{\alpha}_{ij}\in\mathbb{R}^{|\mathcal{O}|}$ denotes the weight vector and the $k$-th element $\alpha^{(k)}_{ij}$ corresponds to the weight of the $k$-th operator $\mathcal{O}_{ij}^{(k)}$. Now the operation $\bar{\mathcal{O}}_{ij}$ between the feature pair $i$ and $j$ is formulated as follows:

The constraint $\mathcal{C}_{1}$ ensures that only one operator would be selected, and the constraint $\mathcal{C}_{2}$ makes the algorithm more stable. Thus, the search problem is defined as:

where $\mathcal{I}[\bm{x},\bm{\alpha},\bm{\Theta}]=\sum_{i}^{L}\sum_{j=i+1}^{L}\sum_{k=1}^{|\mathcal{O}|}\alpha^{(k)}_{ij}\mathcal{O}_{ij}^{(k)}(\bm{e}_{i},\bm{e}_{j})$, $\bm{\alpha}$ denotes the operation weights for all the $|\mathcal{O}|^{K}$ interactions. $\ell$ is the binary cross-entropy loss in this work. $\bm{\Theta}$ represents the parameters for embedding matrices and the fully connected layers. For the sake of simplicity, we temporarily omit the expression of the first-order term and the bias in FMs models. Following previous successful work with proximal steps (Yao et al., 2020a, b), we utilize the same optimization method for efficient searching.

An interesting point proposed in FFM (Juan et al., 2016) and FwFM (Pan et al., 2018), that features from one field often interact differently with that from other fields, prompts us the potential distinct influence of operators on feature embeddings. Noticing that the joint optimization for the operation weights and embedding vectors, we design the operation-aware Embedding module, shorted as OAE, which assigns different embeddings for each operator. Fig. 2 represents the interactions of different operations.

According to the operation-aware Embedding, $\mathcal{I}$ is redefined as:

The embedding $\bm{e}^{(k)}_{i}$ represents the embedding vector corresponding to the $k$-th operation. Observing the original Eq (1), the shared embedding matrices would obtain gradients of different magnitudes with the change of the operator. A comprehensible example is a difference between the plus and multiply operations. The frequent change would affect the computation of the gradient and result in poor convergence. The introduction of the operation-aware embedding removes the interferences of other operations when optimizing the embeddings and may lead to better learning. The final algorithm for interaction function search with the OAE module is described in Alg. 1.

After the model training, the interaction function $O_{ij}$ is selected by $O_{ij}=\arg\max\bm{\alpha}_{ij}$. The expected reward of the candidate creative $c$ is followed as:

where $\sigma(x)=\frac{1}{1+e^{-x}}$ is the sigmoid function. Then, the creative with the highest expected reward will be selected and revealed with a reward.

Following the synthetic experiments in the predictor work (Hill et al., 2017), we produce the synthetic data depending on the assumed CTR predictor and simulate the reward for the proceeding of bandit problems. There are five elements for compositing creatives, including template, picture size, text font, background blur, and background color. We generate the expected reward over 167 products which uses the representation in Equation 3. On average, each product has 67 composited creatives satisfying visual constraints. The operators for different embedding vectors are initialized randomly and the parameters of embedding matrices are sampled from Gaussian distributions. Finally, each creative has an expected reward (CTR), which is the success probability in the Bernoulli trails to simulate the user feedback. The average value of the generated CTR is 0.0885.

At each trial, we sample a product and the algorithm recommends the creative which owns the estimated maximal CTR. Every 10,000 trails are collected for updating and training to simulate the delayed feedback in real production systems. Each algorithm is run multiple times with 20 batches to reduce the bias caused by randomness. The empirical studies are running in a Linux operating system with the Tesla P100 GPU for accelerating.

We choose the following typical algorithms for comparisons where parameters are tuned with the best performance in tems of the overall cumulative CTR.

• •

  Uniform: The creative is selected uniformly at random.

• •

  Egreedy: Recommend a creative at random with probability $0.1$ and recommend the creative with the best statistical reward with probability $0.9$.

• •

  LinUCB (Li et al., 2010): Each candidate creative has the corresponding linear models with the observed contextual information. LinUCB recommends the creative according to the estimated reward with a upper confidence bound.

• •

  LinTS: Similarly with LinUCB, the interactions are constructed by linear models. LinTS utilizes Thompson Sampling for exploration depending on the Bayesian linear regression.

• •

  MVT (Hill et al., 2017): MVT captures the interactions with a linear model and utilizes Thompson Sampling proposed in this work for exploration.

We add another two FM-based algorithms FM and FM-TS. FM recommends the creative which owns the maximal estimated CTR without exploration while FM-TS exploits Thompson Sampling as declared in Section 5.1.

Our proposed method AutoCO automatically searches interaction functions between different feature fields and utilizes the same exploration method as FM-TS.

Generally, bandit algorithms are evaluated in terms of cumulative regret (Sutton and Barto, 2018) which represents the cost before finding the optimum. In practice, the regret is the difference between the expected reward of the optimal strategy and the empirical performance of the conducted algorithm. The lower regret is better. The value of the cumulative reward is complementary to the cumulative regret and higher is better. We also report the normalized cumulative reward, which is actually the CTR in online advertisements.

We perform 20 batches, i.e.2 million impressions to test each algorithm. We report the average performance of 5 repetitions experiments in Fig. 3. There are several findings in our empirical studies.

Finding 1: The interactions between different elements enrich the commonality leverage and lead to better performances for CTR prediction. The linear models (e.g., LinUCB and LinTS) which only build the linear relationship between elements converge to worse creatives although they have comparable performances at the beginning. The expressive representations for the interactions help model the commonality of numerous creatives and improve the performances of the overall CTR.

Finding 2: The proposed AutoCO method shows superior performances during the whole process. The significant improvement comes from two key points: (1) Automated interaction function search; (2) Efficient exploration with variation based Thompson Sampling. To investigate the different influences of these two parts, we conduct two experiments.

First, we compare the AutoCO with the fixed single operation in the search space of AutoCO. Results are shown in Fig. 4. All the algorithms are conducted with Thompson Sampling as the exploration method. Among the five operation functions, the concat operation has the best performance than the other four fixed operations while the AutoTS outperforms the concat interaction function. It can be inferred that better interaction functions for different feature fields are found out via the AutoML method.

Second, different exploration methods are tested to compare the efficiency with the Thompson sampling method utilized in this paper. For the FM model and AutoCO, experiments are run for five times to report the average performance in Table 2. The variation based Thompson Sampling achieves excellent performances compared with Greedy and Egreedy. The trivial exploration method, Egreedy, would always recommend a predefined percentage ($\epsilon$) of creatives randomly although a large number of impressions are collected, causing a waste for displaying bad creatives. Instead, Thompson Sampling recommends the creative proportionally to the probability of being optimum and finally only the optimal creative would be displayed. Another point is that the variational inference for posterior approximation is well conducted in Thompson sampling and this method enables straightforward training in a variety of basic models.

In conclusion, the proposed AutoCO shows the superiority in the accuracy of the CTR predictor and efficiency for exploration on the synthetic data where better expressiveness is obtained through interaction function search.

We conduct an ablation study on the simulated data to analyze the effect of the operation-aware embedding module. We first test the AutoCO and then remove the OAE module. The result is reported in Table 3. The OAE module leads to a slight improvement in terms of CTR compared with the shared embeddings between different operations. This indicates the assistance of the independent embeddings. In the process of training, we also find that OAE contributes to more stable learning.

• •

  Mushroom: The Mushroom dataset (Schlimmer, 1981) contains 22 attributes and 8124 samples. The mushrooms are divided into two categories: safe and poisonous. We following the previous settings (Blundell et al., 2015; Riquelme et al., 2018) to formulate the bandit problem, where the algorithm needs to decide whether to eat the given mushroom. If eating a safe mushroom, a reward +5 will be received. Eating a poisonous mushroom obtains the reward +5 with a probability of 0.5 and the reward -35 otherwise. If the agent does not eat the mushroom, no reward is delivered. We sample 50,000 pieces of data from the original dataset.

• •

  Adult: The Adult dataset (Kohavi, 1996) involves 14 attributes of personal information. The task is to determine if a person makes over $50K a year or not. If the agent makes a right prediction, a reward 1 is delivered. Otherwise, no reward is given. This dataset contains 48842 samples. We pick the 9 categorical features of the mixed 14 original features to conduct our experiments.

Compared with the single interaction function, the AutoCO has the lowest cumulative regret on the two public datasets, as shown in Table 4. All the algorithms utilize the Thompson Sampling for exploration. The good performance of the AutoCO indicates that the complex interaction functions rather than the single operation enhance the performance of modeling relationships between the different attributes.

To verify the efficiency of Thompson sampling via variational inference, we conduct experiments to compare it with greedy strategy. Experiment results are shown in Table 5. We can conclude that Thompson sampling via variational inference can indeed increase the performance.

We exploit AutoCO for the recommendation of the composited creatives in an online advertisement platform. The composited elements include the template, picture size, text font, background blur, and background color. Our experiments are conducted on one display advertising position of a famous e-commerce company. We generate more than 290,000 creatives satisfying visual constraints covering 1,000 popular products. In addition to the element feature mentioned before, we utilize the category feature of products as well as the contextual information of product images. The feature embeddings of images are extracted from ResNet.

We performed the experiments for six consecutive days. Traffic for this experiment was randomly and equally separated to the algorithms. The experiments were conducted on non-holidays to avoid fluctuations in traffic. Each algorithm got more than 150,000 impressions per day. The model was updated every hour using the data collection from the beginning to the latest hour.

Baseline algorithms: (1) Uniform: The composited creatives are delivered to the platform randomly; (2) Context-free Thompson Sampling: The distribution of expected reward is assumed as a Beta distribution. The initialized $\alpha=1$ and $\beta=50$; (3) FM-Egreedy: FM is the CTR predictor and $\epsilon=0.2$; (4) FM-TS: Apply the proposed Thompson method for exploration in the FM model. In our offline experiments, we found that the interaction functions do not change frequently after several epochs, which is also declared in (Yao et al., 2020b). In order to reduce the burden of online updating, we do not frequently search the interaction functions.

The six days online experiment shows the superiority of our approach AutoCO, as shown in Fig. 5. The context-free TS algorithm almost has no improvements compared with the Uniform baseline, which exactly demonstrates the sparsity of feedback. On average, the impression for each creative is less than one per day. The contextual algorithms have better performances where the interaction between different features are paid attention to capture the commonality between different creatives.

On the online A/B test, although the AutoCO is merely comparable in the first two days, the significant improvement is achieved on day 3. In the following consecutive three days, the AutoCO steadily improve the performance. This indicates the quick convergence of the proposed method which is essential for online advertisement. Compared with the FM-based models, our approach still shows competitive performances. It can be inferred that the search of interaction functions help find out more appropriate interaction functions to improve the expressiveness.