Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations

Machine learned recommendation systems are appearing in more and more domains. Examples include systems for search engines, content recommendation, online shopping, online ads, and diagnostic. Deep models, that have grown to dominate and drive such systems, can recommend how to find answers to questions, what movie to watch, what product to buy, which ads are shown, how to fix one’s car, and how to diagnose one’s medical condition. Similar models are the backbone of language translation and understanding, speech recognition, image understanding, video processing, and other applications.

Consider a situation where a user queried for a job recommendations and two attractive results were shown. The user chose one, and deferred the other for later, without saving the details. Later, the user queries again to find the second job recommendation, but the system showes a different recommendation list. The user is unable to find the job they want. This scenario is very realistic and common. An identical query to the same system gives a different set of recommendations, and no matter how hard one tries, they cannot reproduce a result set they had found before. For the user, this can be frustrating. There may be serious consequences in other applications, like medical diagnosis. Irreproducibility in deep models is a critical factor in causing situations like this. Unlike the crisis of lack of replicability (Pineau et al., 2021) in published results due to missing information, much less attention has been given to this problem.

While deep models provide astonishing results on many tasks, they appear to unveil model irreproducibility. The problem manifests as multiple models with the same configuration and training data generate predictions with deviations far beyond classical statistical deviations. The scenario we described is one example. This problem has become very obvious in practical large scale systems. Despite its importance, it received very little attention in academic publications. Only recently, a series of empirical works (Chen et al., 2020; D’Amour et al., 2020; Dusenberry et al., 2020; Shamir and Coviello, 2020a, b; Snapp and Shamir, 2021; Summers and Dinneen, 2021; Yu et al., 2021) demonstrated it. An initial theoretical framework for reproducibility in optimization only appears in very recent work (Ahn et al., 2022) and demonstrates the problem for the much simpler case of convex optimization. Generalizing such results to deep learning, with highly non-convex loss landscape, is even more challenging.

In real systems, two models that are supposedly the same, may behave very differently when deployed in production. For some applications this may be acceptable. However, for some recommendation systems, providing different recommendations for the same request is undesirable, specifically in cases where a typical user would expect equal results. Irreproducibilty is exacerbated if the current recommendations and their user engagements become the training data for future recommendations, as in reinforcement learning systems. Divergence of the training data gradually expands model and future recommendation differences. Practical systems often retrain models and redeploy new versions, use continuous (online) training, update models with fresh new data, or replace models by new generations. Different recommendations may be unavoidable if new generations or new data change model predictions, but one would expect fresh versions or a small amount of new data not to cause substantial deviations.

Irreproducibility affects not only the user, but also the engineering development cycle. It can lead to arbitrary model evaluations, and invalid A/B testing results. Model developers usually use aggregate prediction accuracy metrics on either validation or progressive validation (Blum et al., 1999) (in online models) data to judge experimental models. With huge training datasets, training is expensive, timely, and resource intensive. Model developers attempt to fail fast based on training performance, to only deploy promising models for live recommendations. Without taking reproducibility into consideration, a model can be deployed in experimental stage, show favorable metrics, but then, when retrained to be deployed in production, produce unfavorable results. This can be expensive to diagnose and address. An example of large scale recommendation is click-through-rate (CTR) prediction in sponsored advertising (McMahan et al., 2013). The signal produced by the model is used as part of an auction process to determine ads to show. Shown ads choices affect other system metrics. Two models that are irreproducible tend to show significant divergence in these system metrics, and also end up pushing different examples to the training data. Irreproducibility is thus a source of concern to the engineering development cycle. The problem is not unique to CTR systems, and can interfere with development cycles of other recommendation systems in a similar fashion.

Unlike overfitting, duplicate models that produce different recommendations may have identical average loss or accuracy metrics, but they can exhibit potentially very large Prediction Differences (PDs) on individual examples (Chen et al., 2020; Dusenberry et al., 2020; Yu et al., 2021). PDs do not diminish with more training examples (unlike epistemic/model uncertainty). We may observe PDs in orders of magnitude of 20% or more of the predicted positive engagement rates. Specifically, we observed that PDs are very correlated to system irreproducibility. Thus they can be used to measure the level of irreproducibility (Shamir and Coviello, 2020a) in a training model prior to its deployment, saving resources. In large scale, we also observed that it is sufficient to measure PDs between only two models. This is very suitable for cutting costs in such scale.

We consider a supervised learning problem in recommendation systems. Logs of past recommendation sets constitute the training dataset. A recommendation set can be a stream of past recommendations that a user engaged with, or it can be an outcome set provided for a specific query issued by the user. A recommended item is assigned a label based on the user response or engagement with the item. In the simplest binary case, which we focus on, the label is either positive, i.e., the user engaged with the item, or negative, the user did not engage with the item.

Many such systems are in the sparse regime. There is a huge selection of items the user can engage with, and a huge set of features that can describe the request as well as the items themselves. Features can be properties of the request, the item, the combination of the two, the user, the recommendation user interface, the recommendation rendering, and more. A single recommendation item, represented as a labeled example in the training dataset, will have only an insignificant portion of features present. Using standard notation, the $t$ example is the pair $\{{\mathbf{x}}_{t},y_{t}\}$, with ${\mathbf{x}}_{t}$ being a feature value vector and $y_{t}\in\{0,1\}$ the label. In such sparse problems, most entries in ${\mathbf{x}}_{t}$ are 0. Some features are binary; either the $i$th feature is present, with the respective component $x_{t}^{i}$ of ${\mathbf{x}}_{t}$ taking value $1$, or absent, with $x_{t}^{i}=0$. Some features are categorical, i.e., from a group of features, only one can be present, although for some categories, multiple ones can be in a given example.

Models for such systems can train on the training dataset with multiple epochs, where in each epoch, all or a subset of the currently available training dataset is (re)visited. A trained model is then deployed to provide recommendations to live user requests. These recommendations are ordered by engagement rates predicted by the model, or joined with other signals, such as bids in CTR, and shown to the user based on combination of the predicted engagement rates and the other signals. The requests, their outcomes, and users’ engagements with the outcomes produce more training examples that can be fed into the training dataset, and used to refresh models. Models can be retrained from scratch on refreshed datasets, or trained continuously when additional data is available.

For some recommendation systems, such as device application installation recommendations, except in special cases, current world trends may have smaller effects on engagement rates. However, in others, such trends can affect which content a user is engaged with, or which ads users tend to click on. Trend as well as other temporal nonstationarities justify online learning, where models train with a single visit to each data point, and examples are ordered in a roughly chronological order. With ample training data, which many such systems have, this reduces overfitting to the training data. (Some of the real systems we consider train online, although, others, for which the time is not as critical, may use data shuffling with different schedules.) For efficiency, production systems have to be mini-batched, parallelized and distributed. Thus even with online systems, training is not per-example online, and model update ordering is only roughly chronological. Training accuracy (and other training metrics) can (still) be evaluated with progressive validation (Blum et al., 1999), where prediction on each example is evaluated relative to the true label before the model trains on the example. We focus the exposition on online learning production systems, but the methods described also apply to batch training systems. Specifically, some results reported in Section 6 did not use an online schedule.

Recommender deep neural networks can be fully connected or of other architectures. Due to the extreme sparsity, models can train mainly on individual item engagement label rates, although other losses can also be introduced for various purposes. Numerical input features act as inputs to the network. Categorical features are mapped into embedding vectors, each vector representing a category. All features/embeddings are concatenated into an input layer of the deep network. The embeddings constitute the majority of the model parameters, but as described, only an insignificant portion of the stored embedding vectors is present in any training example. Embeddings are stored in highly distributed systems, and the correct vector must be fetched for any training example. Hence, fetching embeddings can be costly. This cost scales with ensembles that store multiple copies of embedding vectors for the same feature, all of which must be fetched for a given example. This can substantially slow ensemble training (unless model components are shared).

Taking the embedding layer as inputs, a network of several hidden layers is used to produce an engagement rate output. Since the sparse parameter space is dominated (by far) by embeddings, deep networks can be rather shallow with single digit layer count. Let ${\mathbf{a}}^{\ell}$ denote the activation output of layer $\ell$. The embeddings constitute layer $\ell=0$. Starting with layer $\ell=1$, a nonlinearity $f(\cdot)$ is applied on the output of the previous layer. Then, the layer produces an output ${\mathbf{a}}^{\ell}=W^{\ell}\cdot f({\mathbf{a}}^{\ell-1})+{\mathbf{b}}^{\ell}$ where $W^{\ell}$ and ${\mathbf{b}}^{\ell}$ are link weight matrices and bias vectors, respectively, that are learned by the network. For the output layer, $\ell=L$, $W^{L}$ can have a single row (although if multiple losses or outputs are needed, it can have more rows). For binary labeled models, we use logistic regression with logarithmic cross entropy loss. The output of layer $L$ is converted to probability of positive engagement with the logistic (Sigmoid) function $\sigma(x)\stackrel{{\scriptstyle\triangle}}{{=}}1/(1+\exp(-x))$. The positive engagement probability predicted by the model is $\hat{y}_{t}=\sigma(a^{L})$. It is compared against the true label $y_{t}$ producing loss whose gradients propagate to the network. Some form of Stochastic Gradient Descent (SGD) can be used to train the models, including the embeddings, and the weights and biases. Because of the problem sparsity, per-coordinate learning schedules as those in AdaGrad (Duchi et al., 2011) are used, especially for training the embeddings. Actual production models may be a lot more complex with additional architecture, factorization, constraints and components, as well as use more complex optimizers. However, such complications are beyond the scope of this paper, and do not change the results and conclusions.

Hidden layer activations are clipped within some range to ensure numerical stability. Together with capping, some form of normalization should be applied to limit the range of the signal. Without such normalization, an extremely non-smooth objective surface is attained, which at the very least encourages model irreproducbility. Weight normalization (Salimans and Kingma, 2016), layer normalization (Ba et al., 2016), or batch normalization (Ioffe and Szegedy, 2015) can be applied. For shallow networks with sparse embedding inputs, batch normalization may not be ideal. Layer norm and a slightly different form of weight normalization have shown beneficial to our production systems. With this form of weight norm, an $L_{p}$ norm of a row of $W^{\ell}$, constituting the matrix weights incoming to some output neuron of the layer, is kept at a fixed value $v$. Specifically, with $p=2$, let ${\mathbf{w}}_{j}^{\ell}$ denote the $j$th row of $W^{\ell}$. Then, ${\mathbf{w}}_{j}^{\ell}$ is replaced by $\tilde{{\mathbf{w}}}_{j}^{\ell}\stackrel{{\scriptstyle\triangle}}{{=}}v\cdot{\mathbf{w}}_{j}^{\ell}/\|{\mathbf{w}}_{j}^{\ell}\|_{2}$. One advantage of this weight norm is that in deployment, normalization can be computed once for all weights, whereas with layer norm, normalization must be computed on the activations for every inference.

A very important trait of mature large scale production recommendation systems is that they have been highly optimized. From our experience, even very small accuracy improvements of small fractions of a percent of the training loss can make a huge difference to the application system’s overall performance. For binary-labeled models, in addition to training loss, ranking loss, commonly computed as Area Under the Curve (AUC) loss is also important.

where $I(\cdot)$ is the indicator function, and $N$ is the total number of positive-negative label pairs in the validation corpus. Such a loss can also be computed per query (or request) (PQAUC). Training and ranking accuracy tend to be well correlated with many of the practical system’s objectives. Therefore, resources can be saved by eliminating models with bad accuracy metrics without need for live deployment.

Deployed production models may be constrained in complexity, amount of training data, and other system considerations. As such, they may not be able to produce metrics that can be produced without these limitations. Unconstrained teacher models can be trained and used with distillation (Hinton et al., 2015) to transfer such gains to the student to be deployed. Distillation can be advantageous for reproducibility beyond the methods described in Section 1. Some techniques trade off reproducibility to accuracy. We can train more accurate models, with degraded reproducibility, and use distillation to distill the accuracy of these models to more reproducible student models. This approach has been used with gSmeLU to improve production model performance, as we report in Section 6.

Engagement rate Prediction Difference (PD) for individual examples is correlated with system level reproducibility performance metrics. For content recommendations, such differences can lead to different or swapped recommendation sets. When predictions are used for auctions, PDs affect the auction outcomes. Unlike classification applications that focus on the final label, in recommendation systems, the exact predicted engagement probabilities can make a difference. Theoretically, we can use different statistics averaged over multiple models (Chen et al., 2020; Shamir and Coviello, 2020a; Yu et al., 2021) such as standard deviations or KL divergences. In (Shamir and Coviello, 2020a), various $L_{p}$ norms relative to an average prediction of a set of models were considered. However, training multiple models is expensive in large scale production systems. Thus a metric with minimal training costs is desirable. PDs between two models in our large scale systems tend to settle to roughly the same values for different pairs of the same model. Thus it is (usually) sufficient to train one pair of models and measure PD between them. We also want a metric that is not sensitive to actual predicted engagement rates. We define the relative PD for example $t$ as the absolute difference between predictions $\hat{y}_{t,1}$ and $\hat{y}_{t,2}$ of models $1$ and $2$ normalized by their average prediction. Then, the model relative PD, averaged on all examples, is given by

Several forms of smooth activations have been recently proposed, all attempting to preserve ReLU’s shape with a smooth form. Many can be parameterized with a single parameter, denoted as $\beta$. Let $\text{erf}(\cdot)$ be the standard error function and $\Phi(\cdot)$ the standard normal CDF. Functional forms of some smooth activations are:

Activations such as Mish and TanhExp are similar to GELU and Swish, and did not add benefit to accuracy-reproducibilty trade-offs in our systems. Others, such as SELU did not perform as well. GELU can be approximated by Swish with $\text{GELU}_{\beta}(x)\approx x\sigma(\sqrt{8/\pi}\beta x)$. Fig. 1 shows the different activations and their gradients for different values of $\beta$. The parameter $\beta$ controls the width of a transition region from approaching $0$ on the left towards a slope $1$ on the right. For activations in (3)-(5), a smaller $\beta$ gives a wider region, and a larger $\beta$, a narrower one. When $\beta\rightarrow\infty$, the activation resembles ReLU, and when $\beta\rightarrow 0$, it is close to linear. Activations like GELU and Swish are not monotonic, and have a region in which they change the direction of the gradients. Also, they do not have a stop region, i.e, strictly $0$, and a clear slope $1$ region. Activations, such as GELU must be either numerically computed or approximated by other functions. Such activations require more complex hardware implementation, especially with cheaper, simplified hardware that supports only a limited number of operations. These properties can make deployment error-prone, expensive, or slow.

ReLU’s noncontinuous gradient partitions the parameter space into regions with multiple local minima. Some equal, but may provide different model predictions. Training randomness leads the model into a region, and converges to its local minimum. Sudden jumps in gradient, like in ReLU, may lock some hidden units at some states, requiring others to compensate by moving towards their nearest optima. If this happens early in training, it determines the trajectory of the optimization. Different examples and update orders thus lead to different trajectories. Smoother activations give a model fewer opportunities to diverge. Wider transition regions allow gradients to change more slowly, and units in the network to gradually adjust. Wider transition regions are closer to more reproducible linear functions. However, a good activation must also retain good accuracy. Softplus, for example, may not achieve accuracy as good as other activations. Roughly, reproducibility improves by widening activations’ transition regions, but accuracy moves in the opposite direction. However, as observed on our datasets, there exist tuning points where smooth activations outperform ReLU on both accuracy and reproducibility. This is demonstrated in Section 6. Smooth activations thus provide a tuning knob (the parameter $\beta$) to trade off between accuracy and reproducibility. Trade-offs as we described are usually dataset, model architecture, and model configuration dependent.

As mentioned above, there are several disadvantages to activations in (3)-(5) and others alike. In this section, we introduce a new, simple smooth activation SmeLU, that gives competitive accuracy-reproducibility trade-offs. We then generalize the methodology to describe a larger family of activations.

SmeLU was deployed in multiple recommendation production systems for multiple products, with a wide range of architectures and objectives, where irreproducibility was a critical concern. For sponsored advertising CTR prediction, using SmeLU allowed replacing costly ensembles by a single network in several products. In one system, PDs of $17\%$ were observed with single-network ReLU models. A three component ensemble lowered PD to $12\%$. SmeLU with a single component network lowered production PDs to $7.5\%$ ($55\%$ reduction) with no accuracy degradation relative to the ensemble, reducing ensemble complexity and technical debt. Tuning to narrower $\beta$, keeping PD equal to that of the ensemble, gave $0.3\%$ PQAUC loss improvements, giving substantial system metrics improvements. In another system, SmeLU gave over $0.5\%$ PQAUC loss improvements in production, keeping PDs equal to those of an ensemble, but also providing $33\%$ training speed improvements due to elimination of multiple embedding lookups. In a different system, PDs were reduced from $20\%$ to $10\%$. In a content recommendation system, SmeLU provided $6\%$ decrease of recommendation swapping rate, with other additional positive metrics. In an application installation recommendation system, SmeLU gave up to $6\%$ decrease of PDs.

For a clean comparison of accuracy-reproducibility trade-offs of smooth activations and SmeLU, we also tested them on a “toy” simple model on real CTR prediction data for sponsored advertisement. We constructed a simple $6$ layer fully connected network with dimensions $[1024,512,256,128,64,16]$ with $5$ informative features that were learned as embedding vectors, providing a total of $240$ inputs. Models were identically initialized, trained with data-shuffling on billions of examples with the system described in Section 2. Fig. 4 reports relative PD $\Delta_{r}$ against PQAUC ranking loss relative movements. Losses are expressed as percentage change from a baseline ReLU model. Moving left implies better accuracy, and down implies better reproducibility. For smooth activations, the different points are the result of different $\beta$. Moving from the left down to the right described increasing $\beta$ for SmeLU (and decreasing for other activations). ReLU has $12\%$ PD. All smooth activations have points to the left and below ReLU, showing they can be tuned to have both better accuracy and PD. Differences among smooth activations are subtle. Swish can reach better accuracy, but with worse PD, and Softplus can have better PD, but with bad accuracy. SmeLU allows lower PDs ($7\%$ on this example) with improved accuracy over ReLU.

Generalized SmeLU gives an additional degrees of freedom over SmeLU. Training the activation together with the model (per layer or unit) led to prediction accuracy improvements. However, when trained for accuracy, gSmeLU does not give good PDs. Accuracy improvements can still be leveraged on a teacher model in a distillation set up. In one system, gSmeLU gave about $0.15\%$ PQAUC loss reduction to the teacher model, which was leveraged to speed up convergence of the student, reducing training time by $13\%$.

We unveiled the importance of irreproducibility in real world large scale recommendation systems. We demonstrated how ReLU exacerbates this problem, and described using smooth activations as an approach to mitigate it. We presented SmeLU, which is a simple smooth activation that can be used to improve reproducibility, and reported production launches in real recommendation systems that were able to benefit from SmeLU to improve reproducibility, accuracy, and efficiency. We also described a generalization of SmeLU that can improve model accuracy, and reported how it was productionalized, gaining efficiency without harming reproducibility.

The authors acknowledge Sergey Ioffe for early discussions, and Lorenzo Coviello for help in experimentation and launches.