Assessing the Impact of Upselling in Online Fantasy Sports

Let $\mathbf{X}=\{x_{1},x_{2},\ldots,x_{n}\}$ denote the set of user-specific and deposit-specific features, including demographic information, historical transaction data, and engagement metrics. Let $F(\mathbf{X})$ represent the set of deposit amounts for a user, modelled as a multi-class classification model. The problem involves two key components: predicting the expected deposit amount for each user and computing the recommended list of suggested deposit amounts. We want to learn the function $F(\mathbf{X})$ from the historical deposit data $\mathbf{X}$ to predict the expected deposit amounts.

The payments dataset used in this research comes from millions of payment transactions conducted on the Dream11 fantasy sports gaming platform. Whenever the user adds cash to the account, we recommend a set of three values shown in table 2. There are maximum limits and UX constraints on these values. We arrived at these values by analyzing historical deposit amounts and identifying the most frequent deposit amounts and their multiples. This allowed us to cover a range of common deposit behaviours and encourage higher deposits without overwhelming the user.

The control group received a default recommendation, whereas the target groups received various personalized amounts alongside different discrete intensity values [1x, 1.25x, 1.5x, 2x]. This experiment lasted eight weeks, from January 1, 2024, to February 28, 2024. We observed an immediate impact on the per transaction value of about +5.6%, while users were also making fewer transactions on the platform (reduction in infra cost), resulting in a 4.8% fall in the most aggressive variant. While evaluating the long-term sustained impact in the avg_deposit_week_8, Figure [1] illustrates the uplift across the experimental variants shown in Table [3].

Upselling impacted the experiment guardrails, which were the user engagement metrics. The time spent for a successful payment increased, and user conversion dropped by almost 2.1%, mainly impacting new users.

For causal uplift modeling, the dataset consists of three key components: Treatment (T), Response (Y), and Features/Covariates (X). These are defined as follows:

• •

  Treatment (T_i): The intensity of upselling, represented by four levels: {1.0, 1.25, 1.50, 2.0}.

• •

  Response (Y): The primary outcome of interest, measured as the total deposit amount by each user.

• •

  Features (X_i): User-specific characteristics, including demographics, gameplay behavior, and historical transaction data.

To arrive at an optimal upselling policy, we personalize the intensity factor for each user. We estimate the incremental uplift by applying the intensity treatments (T_i) uniformly across all users. This dataset will then be used to model and optimize upselling strategies.

We estimated the Conditional Average Treatment Effect (CATE) $Y(x)$, which represents the change in the probability of completing a deposit $\Pr(Y=1)$ due to varying upselling intensity, based on user features $X$. To achieve this, we tested four algorithms: S-learner, T-learner, X-learner, and R-learner, implemented using Python’s causalml library (Chen et al., 2020), with XGBRegressor as the base model and a fixed propensity score of 0.5.

For model training and optimization, we used the hyperopt package (Bergstra et al., 2013) with SparkTrials to distribute workloads, and MLflow (Zaharia et al., 2018) to manage model runs. The parameter search was guided by Tree-based Parzen Estimators (TPE), with the objective function defined as:

• •

  Define the parameter search space using causalml.

• •

  Train models on the training dataset.

• •

  Apply trained models to both training and test sets.

• •

  Optimize by calculating the mean AUUC (Area Under the Uplift Curve) score on the test set.

We ran 1000 random trials, logging the AUUC for each treatment. Two model configurations were tested:

• •

  Global configuration: Selects the model with the lowest test set loss across all treatments.

• •

  Local configuration: Optimizes the CATE for each treatment group (TG) using the highest test set AUUC.

Table 4 summarizes the XGBRegressor configurations used in the SparkTrials. Each treatment group (TG1, TG2, TG3, TG4) has its own local configuration parameters, while the global model (TG_global) is optimized for overall performance. A treatment is assigned based on the following rule:

Uplift modelling faces performance limitations due to the need for an unbiased evaluation metric. Individual responses to actions and natural responses are unknown simultaneously, preventing explicit labelling of uplift responses. We underwent rigorous evaluations on both the business and model sides to establish confidence in our analysis.

• •

  Percent Treated: Percentage of users distributed across all possible treatment values, including control. In Table [5], the grid-experiment policy is uniformly distributed while the CATE policy is assigned with a non-negative threshold.

  Percent Treated	policy
  Treatment	Grid exp policy	CATE policy($\pi$)
  CG	19.92	3.16
  TG1	20.12	12.71
  TG2	20.05	31.86
  TG3	20.13	37.42
  TG4	19.78	14.86

• •

  ERUPT: Expected Response Under Proposed Treatments: (Zhao et al., 2017), (Hitsch and Misra, 2018) It quantifies a model’s performance at the downstream task of predicting the optimal treatment to assign each member of the population. It is defined as follows

  (1)

  $$\text{ERUPT}=\mathbb{E}(Y\cdot\mathbb{I}(\pi(x)=t)\mid T=t,\pi)$$

  We calculate the expected bootstrapped revenue using a sample size of 5000. After model inference, the CATE policies for both test ($\pi_{\text{test}}$) and train ($\pi_{\text{train}}$) datasets are derived, with the intensity factor yielding the highest uplift being selected. As shown in Figure [2], the CATE policy results in a significant 10.7% uplift across both the training and test datasets.
  

  

• •

  AUUC (Area Under the Uplift Curve): The uplift chart consists of an Uplift Curve and a random baseline curve for each treatment. The charts demonstrate a strong offline fit of the underlying estimators. The cumulative gains are plotted by ranking users based on their inferred uplift scores, showing that most gains are concentrated within the top-ranked users.

  

Our research highlights the intricate trade-offs between upselling intensity and user satisfaction. While increasing upselling intensity leads to immediate revenue gains, it also results in a decline in user recall and satisfaction. This effect was most pronounced in the TG1 group, where a bi-modal distribution of responses revealed polarized reactions to upselling strategies. These findings emphasize the need to carefully balance upselling intensity with user experience to maintain both engagement and revenue growth.

The causal uplift modelling framework allows us to tailor the intensity for each user, effectively managing these trade-offs and ensuring fairness in recommendations. Results from both the test and training datasets show that taking these models online holds significant potential, with an estimated revenue uplift of 10.7%.

Further analysis, as shown in Figure [1], revealed that the guardrails mainly were impacted for new users. The offline CATE policy applied a 1x intensity for 80% of new users, resulting in a 2.1% improvement in conversion rates. While these offline results are promising, further validation is required in an online experimental setting to confirm the optimal trade-offs between revenue and user experience.

Our future work will focus on testing the models online, exploring additional user features, and extending the experimentation to incentivise user behaviour upselling on the platform. Further, developing personalized upselling strategies that dynamically adjust intensity levels based on user behaviour and feedback could enhance revenue and user satisfaction.

The ”Deposit Amount Upselling” platform shown in figure 4 integrates several key components to predict deposit amounts and provide personalized upselling recommendations. Below is a summary of the implementation steps:

• •

  Data Ingestion: The platform ingests data from two primary sources: streaming events and backend data stored in the warehouse.

• •

  Feature Aggregation/pre-processing: We prepare the data by performing the following steps: label creation, gradient features calculation, outlier removal and feed it to the Deposit Amount Predictor model, ensuring the model leverages both real-time and historical data for accurate predictions.

• •

  Deposit Amount Predictor: A machine learning model predicts the likely deposit amounts for users. The model outputs deposit amount classes (e.g., 0-20, 20-40, 1000-25000) and corresponding probability scores.

• •

  Upselling Recommendation: The predicted amounts and scores are processed by an upselling policy, which employs an intensity layer to experiment using the model’s predictions.

• •

  Data Ingestion: Real-time data processing is performed using Kafka on our in-house stream processing platform (Gaikwad, 2023). Backend data is retrieved from SQL databases like Redshift.

• •

  Feature Store: Processed features are stored efficiently in a feature store, supporting high throughput and low-latency retrieval. Delta Lake is used for offline storage, while Redis is employed for online storage.

• •

  Model Training and Prediction: Causal Machine learning models are developed in Python using (Chen et al., 2020) library. Airflow manages the scheduling for both batch and real-time inference scenarios.

• •

  Model Management: ML Flow (Zaharia et al., 2018) provides seamless model tracking and registry management.

• •

  Deployment and Inference: We power and monitor our inference jobs on our in-house machine-learning platform, Darwin

• •

  Backend Integration: Recommendations are served into backend systems for delivery to end-user devices.