Reliable Interval Prediction of Minimum Operating Voltage Based on On-chip Monitors via Conformalized Quantile Regression

For the task of $V_{min}$ point estimation, in both the product flow and in-field scenarios, the objective remains consistent: utilizing a set of features to predict a single value. We denote these features as a $D$ dimension vector $\bm{\mathrm{x}}\in\mathbb{R}^{D}$, the $V_{min}$ as a real number $\mathrm{y}\in\mathbb{R}$, and the point predictor as $g_{p}(\cdot;\bm{\theta}):\mathbb{R}^{D}\to\mathbb{R}$, parameterized by $\bm{\theta}$. Given a training dataset of $N$ tested chips $\mathcal{D}=\{(\bm{\mathrm{x}}_{i},\mathrm{y}_{i})\}_{i=1}^{N}$, the predictor is optimized by minimizing the mean of a loss function $\mathcal{L}_{p}$:

where $\bm{\mathrm{X}}=[\bm{\mathrm{x}}_{1},\cdots,\bm{\mathrm{x}}_{N}]^{T}\in\mathbb{R}^{N\times D}$ is a matrix of inputs, and $\bm{\mathrm{y}}=[\mathrm{y}_{1},\cdots,\mathrm{y}_{N}]^{T}\in\mathbb{R}^{N}$ is a vector of true $V_{min}$.

In manufacturing test processes, engineers often face risks of over-kill or under-kill when relying solely on $V_{min}$ point predictions to identify abnormal products due to process variations. In in-field scenarios, point estimation can be highly unreliable due to the presence of numerous environmental uncertainties. Consequently, the utilization of prediction intervals becomes essential for effectively detecting outliers and identifying potential failures.

Unlike point estimation, which only generates a single value for an input example, region prediction provides an interval prediction. A region regressor $g_{r}(\cdot;\bm{\theta}_{lo},\bm{\theta}_{hi}):\mathbb{R}^{D}\to\mathbb{R}^{2}$, consisting of a pair $g_{p}(\cdot;\bm{\theta}_{lo}):\mathbb{R}^{D}\to\mathbb{R}$ and $g_{p}(\cdot;\bm{\theta}_{hi}):\mathbb{R}^{D}\to\mathbb{R}$ of the lower and the upper bound function, maps a sample $\bm{\mathrm{x}}$ to a closed region $C(\bm{\mathrm{x}})$:

Given a coverage rate $1-\alpha$ where $\alpha\in[0,1]$ and the training dataset $\mathcal{D}$, the prediction intervals of a region regressor $g_{r}$ should be able to cover at least $1-\alpha$ labels:

We introduce two well-known region regression methods satisfying Eq. 3: Gaussian process and quantile regression. Their theoretical traits are summarized in Table I.

Our $V_{min}$ prediction framework is depicted in Fig. 1, where four stress read points are drawn for illustration. $V_{min}$ at each stress read point will be predicted. The horizontal dash line (min_spec) stands for the product specification of the minimum operating voltage, i.e., device with $V_{min}$ higher than that threshold will violate the specification and likely become a failure.

We utilize low-cost parametric data and on-chip data to predict $V_{min}$ at time zero and subsequent read points during stress simulated in-field life. Note that stress is done at an elevated voltage such that a much shorter stress duration is equivalent to a much longer in-field life. Specifically, two kinds of $V_{min}$ prediction scenarios are considered: in the production test flow, and in the in-field deployment which is simulated by accelerated stress. In the first case, both production parametric test data and on-chip data are included to build $V_{min}$ predictors. In the second case, however, we make $V_{min}$ degradation prediction based on all accessible features before the $V_{min}$ test timestamp, including production parametric test data at time zero and on-chip data measured at all previous read points during stress. In our industrial dataset, both $V_{min}$ and on-chip data are collected at the same read point, and the total number of read points is relatively small, i.e., less than 10. In this case, time series methods would suffer over-fitting problems. Thus, we treat on-chip data at different read points as different features, and apply CQR to predict $V_{min}$ intervals.

Since CQR is originated from CP, we first briefly summarize how CP works, and then present CQR for $V_{min}$ interval prediction.

Even though the coverage of prediction intervals is guaranteed for the training dataset $\mathcal{D}$ in GP and QR, such characteristic is not held for a testing instance $(\bm{\mathrm{x}}_{N+1},\mathrm{y}_{N+1})$:

The adoption of the aforementioned two region predictors for new examples is risky without the coverage guarantee.

In semiconductor industry, all chips can be viewed as examples from a hidden distribution: $\{(\bm{\mathrm{x}}_{i},\mathrm{y}_{i})\}_{i=1}^{N+1}$ are sampled i.i.d. from a distribution $P_{XY}$. CP can help to calibrate any heuristic interval to meet the coverage guarantee in Eq. 6 [10]. CP has two main versions: full CP and split CP. In regression tasks, full CP needs infinite times of model fitting, rendering it impossible for practical usage. On the contrary, split CP is more computationally efficient with the scarification of splitting the training dataset.

We outline how split CP utilizes a $V_{min}$ point predictor $g_{p}$ to generate a interval $C(\bm{\mathrm{x}})$ for $\bm{\mathrm{x}}$:

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Split the training dataset $\mathcal{D}$ into a new training dataset $\mathcal{D}_{tr}$, and a small calibration dataset $\mathcal{D}_{ca}$ such that $\mathcal{D}_{tr}\cup\mathcal{D}_{ca}=\mathcal{D}$, and $\mathcal{D}_{tr}\cap\mathcal{D}_{ca}=\phi$.

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Fit the point regressor $g_{p}$ in $\mathcal{D}_{tr}$.

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Compute $\hat{q}$ as the $\lceil(M+1)(1-\alpha)\rceil/M\text{-th}$ quantile of the conformal score function $s(\bm{\mathrm{x}},\mathrm{y})$ of absolute residuals in the calibration set $\mathcal{D}_{ca}$:

where $M$ is the number of examples in $\mathcal{D}_{ca}$.

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Construct the interval for $\bm{\mathrm{x}}_{N+1}$, satisfying Eq. 6:

While split CP satisfies the coverage guarantee, the length of predicted intervals is $2\hat{q}$, remaining fixed to different inputs. This property may incur overkill for good products and underkill for defective ones. CQR, however, is a variant interval prediction method combining CP and QR together.

We describe the procedures of split CQR:

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Split the training dataset $\mathcal{D}$ .

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Fit the quantile regressor $g_{r}$ in $\mathcal{D}_{tr}$.

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Compute $\hat{q}$ as the $\lceil(M+1)(1-\alpha)\rceil/M\text{-th}$ quantile of the conformal score function $s(\bm{\mathrm{x}},\mathrm{y})$ in $\mathcal{D}_{ca}$, where

$\mathord{\mathchoice{\vbox{\hbox{\scalebox{0.7}{$\displaystyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\textstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptstyle\bullet$}}}}{\vbox{\hbox{\scalebox{0.7}{$\scriptscriptstyle\bullet$}}}}}$ Construct the interval for $\bm{\mathrm{x}}_{N+1}$ satisfying Eq. 6:

CQR inherits good features of CP and QR, as shown in Table I. It is shown empirically effective in achieving the shortest interval length than CP and QR across 11 datasets while persisting the designed coverage rate [11]. Herein, we adopt it for reliable $V_{min}$ interval prediction.

Our experiments use 156 5nm automotive chips to demonstrate the effectiveness of the proposed $V_{min}$ prediction framework. As shown in Fig. 1, parametric data and on-chip monitor data are considered for $V_{min}$ prediction. We describe how the input features and the output $V_{min}$ are collected.

All 156 chips go through the dynamic Dhrystone stress at elevated voltage in Burn-In (BI) oven for 1008 hours to simulate in-field long-term aging degradation. At specific stress read points, i.e., 0, 24, 48, 168, 504, and 1008 hours, we pause the stress process and 1) test SCAN $V_{min}$, 2) perform the parametric tests, and 3) collect on-chip monitor data. SCAN $V_{min}$ is tested on Automatic Test Equipment (ATE) tester, at temperatures of -45°C, 25°C, and 125°C. The parametric tests are also performed on ATE tester, including IDDQ, trip IDD, leakage, etc., across all three temperatures. The chip has two types of on-chip monitors: domain sensors which include Ring Oscillator Delay (ROD) sensors and in-situ Critical Path Delay (CPD) sensors. In our experiment, due to hardware and logistic process limitations, ROD is measured on ATE at room temperature (25°C) only while CPD is measured in-situ in BI oven at 80C. We summarize the traits of input features in Table II.

We illustrate the features used for $V_{min}$ prediction at each read point and the evaluation metrics for point prediction and interval regression. As shown in Fig. 1, for the prediction of $V_{min}$ at time 0, both parametric test data and on-chip monitor data collected at time 0 are utilized to predict $V_{min}$; For the prediction of $V_{min}$ at the subsequent read points to enable in-field failure prediction, we use on-chip monitor data collected at all previous read points and parametric data collected at time 0, because parametric tests are no longer possible once chips are shipped to customers and deployed in-field.

For $V_{min}$ point prediction, the performance criteria are the coefficient of determination ($R^{2}$) and Root Mean Square Error (RMSE); For $V_{min}$ region prediction, the metrics are the average interval length and the coverage of true $V_{min}$ of the testing data.

To reduce the influence of randomization, a 4-fold cross-validation is adopted. We report the average score of each metric across the 4 testing folds. In CQR, 75% training data are used to train predictors while the remaining 25% chips are held for calibration. To ensure a fair comparison, we use the same random seed for all $V_{min}$ interval predictors.

ML models with fewer learnable parameters and simpler structures are more favorable for our high-dimensional small data scenario. Moreover, feature selection is an essential dimension reduction technique for some ML models to avoid overfitting problems.

Firstly, we demonstrate model selection for $V_{min}$ point prediction. 5 regressors are considered: Linear Regression (LR), Gaussian Process (GP) [6], XGBoost [12], CatBoost [13], and a 2-layer Neural Network (NN). The detailed configurations of each regressor except LR are provided below:

The $R^{2}$ of $V_{min}$ point predictions of regression models are depicted in Fig. 2 For SCAN $V_{min}$ tested at time 0, while CatBoost is the best method across all three temperatures, linear regression is also performing well with a small drop of $R^{2}$, which is less than 0.03. For all methods except GP, the RMSE for $V_{min}$ point predictions are within $2.5mV$ to $7mV$ ($12mV$ to $22mV$ for GP) for all scenarios, and exhibiting similar comparison as $R^{2}$ among different models, i.e., CatBoost performs best for time 0 prediction while linear regression performs reasonably well overall. As linear regression is straightforward to implement by either software or hardware, it is a sufficiently good option for $V_{min}$ time 0 prediction in industrial production tests.

For $V_{min}$ degradation prediction, no regression model is outperforming the rest across all temperatures and stress read points, in terms of $R^{2}$ and RMSE. We note that linear regression is still performing reasonably well, and even the best one for predicting SCAN $V_{min}$ at 25°C and 125°C, for both $R^{2}$ and RMSE. With its simplicity, implementing a linear regression model with an on-chip hardware accelerator seems to be a viable option for in-field $V_{min}$ degradation prediction.

In addition, an interesting observation is that there is no clear reduction of $R^{2}$ in SCAN $V_{min}$ degradation prediction accuracy from 0 to 1008 hours. It demonstrates that our design of on-chip monitors captures informative gate-level features that exhibit a strong correlation with system-level $V_{min}$.

We consider three interval prediction methods: GP, QR, and CQR. QR and CQR are built on 4 point regressors: LR, NN, XGBoost, and CatBoost. The configurations of these models are the same as those in Section IV-C. We set $\alpha=0.1$ and let predictors generate an interval with 5% to 95% coverage.

The average length of prediction intervals of SCAN $V_{min}$ and coverage rates are shown in Table III. Both GP and QR underestimate the interval for testing chips, failing to meet the designed coverage rate. CQR, in contrast, successfully calibrates the undercovered interval predictions of QR across all stress read points and temperatures, underscoring the importance of applying conformal prediction for reliable region predictions.

CQR performs differently with different point regression models. The best variant is CQR CatBoost, achieving the shortest intervals with around 90% coverage rate. While LR is competitive for point prediction in Section IV-D, its CQR version predicts larger intervals than CQR CatBoost, especially for SCAN $V_{min}$ at -45°C and 25°C.

We present evidence supporting the value of on-chip monitor data in the prediction of $V_{min}$ intervals. Fig. 3 illustrates the interval length of CQR CatBoost with three types of feature sets: 1) parametric test data and on-chip monitor data (same to Section IV-F), 2) parametric test data only, and 3) on-chip monitor data only. In addition, Table IV summarizes the average length across all read points of SCAN $V_{min}$ during stress.

Compared to utilizing parametric data only, the inclusion of on-chip monitor data results in a reduction of 21.01% in the average interval length. Intriguingly, a CQR CatBoost model relying solely on on-chip monitor data outperforms the same model using only parametric test data, despite the much larger number of parametric data (Table II). This implies the on-chip monitor data could contain more information that facilitates $V_{min}$ estimation.