OptiGrad: A Fair and more Efficient Price Elasticity Optimization via a Gradient Based Learning

In optimizing commercial pricing, several strategies are deployed with distinctive methodologies. Individual Optimization, for instance, calculates the optimal price for each customer based on their unique risk profile. Alternatively, Indirect Ratebook Optimization applies reverse-engineering on downstream optimized prices, which are frequently the outcomes of Individual Optimization. Each method has its own advantages and limitations, which can vary depending on whether the optimization is conducted in an online or offline environment. Individual Optimization, for example, is noted for its precision, eliminating the potential for compounded errors that may arise from the reverse-engineering process inherent in Indirect Ratebook Optimization. However, its practical application can be complex, with challenges in operationalizing rating prices due to the necessity of direct price calculations. On the other hand, Indirect Ratebook Optimization facilitates easier implementation within the insurance rating system for production, though its offline nature limits market responsiveness.

This study primarily focuses on Direct Ratebook Optimization, which seeks to combine the production deployment advantages of Indirect Optimization with a more straightforward method of price optimization, eliminating the need for reverse engineering. This method strives for an optimal balance between accuracy and operational feasibility.

The quest for fairness in AI systems requires the articulation of a clearly defined fairness criteria. First, there is the concept of information sanitization that restricts the use of sensitive data in predictive model training. A key instance from the European insurance sector is the implementation of the Gender Directive. Although nearly a decade passed before legal guidelines for its application to insurance activities were established, the directive was fundamentally aimed at upholding the principle of equal treatment between men and women in the access and provision of goods and services, including insurance. The result was a prohibition on the use of gender as a rating variable in insurance pricing. The directive enforced gender equality in insurance pricing across the European Union from December 21, 2012, as detailed in (Schmeiser et al., 2014).

However, a major challenge for actuaries is that sensitive variables can be highly correlated with several other features. For example, the combination the driver’s occupation and car’s size, color could inadvertently introduce gender bias in the prediction of car insurance prices. Geographic information has also been noted as dependent to the race or national origin in Insurance (Saxena et al., 2024). An illustrative instance is shown  (Angwin et al., 2017), which revealed that individuals residing in neighborhoods predominantly populated by racial minorities are subject to elevated insurance premiums compared to individuals with equivalent risk profiles residing in other neighborhoods. A significant aspect of fairness in the Fair Machine Learning community is statistical or group fairness. This partitions the world into groups defined by one or several sensitive attributes. It requires that a specific relevant statistic about the classifier is equal across those groups. This is particularly challenging in the context of continuous sensitive attributes, where it is crucial to guarantee distributional independence instead of looking at average expectation between groups. In the following, we outline the most popular definition and mitigating strategy used in recent research.

This section delves into the complexities associated with integrating fairness considerations into commercial premium optimization, with a specific focus on car insurance pricing. To empirically investigate these challenges, we conducted experiments utilizing the Atoti Dataset¹¹1Further details provided in the Experimental Section, specifically designed for elasticity optimization purposes. In this section, we perform an Indirect Ratebook Optimization with an XGBoost model trained on individually optimized prices, wherein we adjusted the hyperparameter $\lambda$ in Eq. 2, to find a balance between Gross Written Margin (GWM) and minimum conversion rates.

We explore various scenarios by incrementally adjusting the $\lambda$ hyperparameter and evaluating the resultant fairness levels, measured with respect to a sensitive attribute—age, which was excluded from the model training but utilized solely for fairness assessment. Our findings, depicted in Figure 1, indicate that $\lambda$ the influences fairness. For example in extreme settings, characterized by either low conversion rates (approximately 23%) or higher conversion rate (approximately 27%)—exhibit the most pronounced biases. Interestingly, bias decreases at intermediate $\lambda$ values, highlighting the intricate interplay between fairness considerations and premium pricing optimization. These results suggest that addressing fairness in commercial premium introduces a layer of complexity extending beyond the simple management of profit or conversion rates.

In this section, we consider the actuarial pure premium model $h_{w_{h}}$ and the conversion model $f_{w_{f}}$ to be differentiable and already trained. Both models are typically represented using General Linear Models (GLMs), which is a common practice in the actuarial field, although it is worth noting that they can also be represented as neural networks.

In this section, the OptiGrad formulation (referenced in Equation 7) is extended to integrate fairness criteria by leveraging the Hirschfeld-Gebelein-Rényi (HGR) Neural Network (HGR_NN) architecture, referenced in section 2.2.2. The choice of the HGR_NN architecture is motivated by its capability to effectively measure dependencies with continuous features, a crucial aspect given that commercial pricing is continuous (Grari et al., 2021b). Integrating an HGR-based differentiable estimation into the framework enables the optimization of three key objectives: maximizing profit margins, guaranteeing a minimum conversion rates, and reducing bias.

The HGR estimation, implemented via two interconnected networks, $\phi_{w_{\phi}}:\mathcal{P}\rightarrow\mathbb{R}$ and $\psi_{w_{\psi}}:\mathcal{S}\rightarrow\mathbb{R}$, quantifies the nonlinear dependence between the adjusted commercial premium, denoted as $c_{w_{c}}(x)*h_{w_{h}}(x)$ (the input for $\phi_{w_{\phi}}$), and the sensitive attribute (the input for $\psi_{w_{\psi}}$). This framework enables the evaluation of fairness at each stage of the training process, facilitating corrective measures to mitigate bias. The goal is to ensure that the pricing strategy optimization does not inadvertently reinforce bias described by the sensitive attribute $S$ (binary or continuous).

Equation 9 introduces the HGR component, it aims to ensure that the non linear dependence between the commercial price and the sensitive attribute $S$ is below a certain threshold $\epsilon^{\prime}\in\mathbb{R}$. This HGR estimation corresponds to the expectation of the product of the optimally standardized outputs of both networks ($\widehat{\phi}_{w_{\phi^{*}}}$ and $\widehat{\psi}_{w_{\psi^{*}}}$) with $w_{\phi}^{*}$ and $w_{\psi}^{*}$ the optimal neural networks parameters maximizing the expectation of the HGR.

By maintaining the same Lagrangian relaxation approach, the overall optimization problem of our Fair Price Elastic Optimization framework (Fair-OptiGrad) can thus finally be formulated as:

In this objective, the third term represents the fairness component. The $\lambda_{S}$ hyper-parameter controls its impact in the optimization. The optimization is transformed in a min-max objective in order to recover at each step the optimal neural network $\widehat{\psi}_{w_{\psi}}$ and $\widehat{\phi}_{w_{\phi}}$. This maximization of the HGR estimation can be updated by multiple steps of gradient ascent for each gradient descent iteration on $w_{c}$ for accurately estimating the HGR. This allows to evaluate the fairness more accurately at each stage of the training process.

The algorithm takes as input a training set from which it samples batches of size $b$ at each iteration. At each iteration, it first standardizes the output scores of networks $\phi_{w_{\phi}}$ and $\psi_{w_{\psi}}$ to ensure 0 mean and a variance of 1 on the batch. Then it computes the adversary objective function to estimate the $HGR$ neural estimate with $n_{a}$ gradient ascent iterations. At the end of each iteration, the algorithm updates the prediction parameters $\omega_{c}$ by one step of gradient descent.

In our first experiment, we compare the OptiGrad framework against conventional state-of-the-art methods, specifically focusing on Individual Optimization (as outlined in Section 2.1), which is constrained to the training data, and an Indirect Ratebook Optimization, which employs an XGBoost model ($300$ trees with a max depth to $5$) trained on the outputs from the Individual Optimization. This offline configuration permits the evaluation of the latter on both the test and dev datasets. Moreover, for our OptiGrad methodology we investigate the effect of model complexity—comparing a simple regression against a deep neural network—on the coefficient $c_{w_{c}}$. For all scenarios we have constrained the upper and lower limits to $1.6$ and $1.2$, respectively. Each algorithm has a tradeoff hyperparameter that allows to varying the minimum conversion rate. We have therefore ensured a thorough exploration of these tradeoffs by sweeping across hyperparameter values for each algorithm.

The results, depicted in Figure 4, indicate that our OptiGrad approach outperform the traditional methodology in performance on the test and dev datasets. On the training dataset, Individual Optimization exhibits comparable effectiveness. The least effective model is the Indirect Optimization, which significantly underperformed due to the loss of performance in attempting to reverse-engineer the downstream individually optimized prices. Furthermore, we observe that increasing the complexity of our coefficient model $c_{w_{x}}$ has no impact.

To analyse the enforcement of fairness of our OptiGrad framework, we investigated two distinct scenarios: (i) with a continuous sensitive attribute: Age, (ii) and a binary sensitive attribute: Employment status. It is important to note that these sensitive attributes are used solely during the training phase and are not employed in the testing phase, which is dedicated to the assessment of the fairness.

Our Fair-OptiGrad algorithm incorporates two hyperparameters to balance the trade-off between achieving a minimum conversion rate and enforcing fairness. Following a similar approach to our initial scenario, we conducted an exhaustive hyperparameter sweep to optimally balance these two components.

In Figure 5, we observe that adjusting the parameter $\lambda_{S}$ effectively promotes fairness across different demographics. Increasing $\lambda_{S}$ from 0 to 1250 results in a significant reduction in the HGR from 0.15 to 0.07 for age, and from 0.14 to 0.06 for employment. This indicates a robust enforcement of fairness as $\lambda_{S}$ increases. Most important observation is that the Pareto front for the different $\lambda_{s}$ remains similar with the exception of the extreme values ($\lambda_{s}=1250$) while unfairness (HGR) is reduced in average.