Calibration and Correctness of
Language Models for Code

Code LLMs are perhaps mostly widely-used for code suggestion/completion; other tasks include code synthesis and program repair.

In general, the levels of alignment between the intrinsic confidence (viz., directly provided by the LLM) and correctness is poor. This indicates need for better approaches to calibration. We then explore several engineering responses to the problem, listed in the following research questions. First, we consider the standard approach of confidence rescaling, using Platt scaling.

Next, we investigate the possibility that the model is able to better calibrate upon reflection. We ask the model (using a separate reflective prompt) to consider its own generated code and judge its confidence in the quality.

This concept originates in prediction problems like weather forecasting. Consider a weather model predicting a 70% confidence (probability) of rain the next day. If we ran this model for a while, and observed rain in 70% of the days where a forecast with 70% confidence was made, then we call it a well-calibrated model. A well-calibrated model’s confidence in a given output, is quite close to the empirical relative frequency (likelihood) with which the output is actually correct.

With well-calibrated rain-forecasting, a user has options for a rational response: at 20% confidence of rain, one might take a hat; at higher confidences, one might take an umbrella; if even higher, one might take the bus rather than walk, etc. From an earlier work by [7]: given a model $M$, an input $X$ and true, expected correct output $Y$, a model output $M(X)=\hat{Y}$, (note that we won’t always have $Y=\hat{Y}$) and a output probability $P_{M}(\hat{Y}\,|\,X)$ provided by the model, a perfectly calibrated model satisfies the following condition

In other words, if we have perfectly-calibrated confidence ${p}$ (the model’s calculated probability of its prediction that the output is $\hat{Y}$), then this value equals the empirical fraction p of the cases where the actual output $Y$ correctly matches the prediction $\hat{Y}$. Usually these probabilities don’t perfectly match; there are various measures of the deviation, including Brier Score [8] and $ECE$ (Expected Calibration Error) [9]

Even powerful LLMs can make mistakes [1], potentially leading users to accept incorrect code. A well-calibrated confidence signal could help developers manage [4, 10] this risk. Consider the confidence ${p}$ associated with generated code. A well-calibrated high-value of ${p}$ would indicate a high empirical probabilty p that the code is correct, and so it could be simply accepted; a low value would indicate higher risk that the code is incorrect, and so should be rejected. A poorly calibrated model may lead to either unnecessary rejection of likely correct code, or ill-advised acceptance of likely incorrect code. We note that good calibration allows more nuanced, effective quality-control (Q-C) decisions, beyond simple binary decisions e.g., carefully review each token of generated code, perhaps by several people, vs. just use it. Such a well-calibrated quality-control process has been used in medicine e.g., for elder-care [11], and for decision-making in cancer-care [12]. Given the cost & consequences of properly addressing software quality, and the potential benefits of LLM-generated code, a well-calibrated confidence signal is highly desirable.

When evaluating calibration, we need a notion of correctness. For (non-generative) models outputting labels, classes, True/False, etc. correctness is simply an exact-match with ground-truth correct label. Exact-match could also be viewed as a notion of correctness for code, e.g., with defect repair, where there is a known, incorrect “buggy” version, and a known “fixed” version. Generated code is correct only if it matches exactly the fixed version, given an appropriate prompt. However, this approach is overly strict; the generated code might match exactly, but still pass all tests. Other notions of correctness exist: code-review, formal verification, etc. We use test cases provided with the code as our preferred indication of correctness, as tests are widely used, and are easily automated.

While test-passing correctness offers the advantage of admitting different semantically identical forms, test cases maybe insufficient or incorrect³³3e.g., [13] report gaps in the test sets of HumanEval.; tests can also be “flaky” [14]: the same test, on the same code, might pass, or fail.

We usually calculate a confidence measure (or probability) $p$, associated with generated output code $C$. We consider two categories of measures: intrinsic probability, which is calculated by the generative LLM per se, and reflective probability, obtained by re-invoking the model, instructing it to estimate its confidence in the correctness of the code just generated (see Figure A1 for prompts). Our measures include:

Average Token Probability

(Intrinsic, $p_{avg}$) For an output sequence $\mathrm{T}$ of tokens $\tau_{i},i=1\ldots n$, we collect the associated model probabilities $p(\tau_{i})$, and then compute the mean $p_{\text{avg}}(\mathrm{T})=\frac{1}{n}\displaystyle\sum_{i=1}^{n}p(\tau_{i})$.

Generated Sequence Probability

(Intrinsic, $p_{tot}$) The full generated sequence confidence is calculated as the product of probabilities, $p_{tot}({\mathrm{T}})=\displaystyle\prod_{i=1}^{n}p(\tau_{i})$.

Verbalized Self-Ask

[15, 16, 17] (Reflective, $p_{v}$) We instruct the model to reflect jointly on the prompt, and the model-generated code, and then output a numeric value of its confidence in the generated code.

Extra logic is implemented for when the model fails to output a probability (discussed further in Understanding effects of verbalized retry failures).

Question Answering Logit

[18] (Reflective, $p_{B}$ and $p_{NB}$) In this case, rather than prompting for a numerical score (as above), we ask for a TRUE or FALSE answer. The probability associated with the TRUE token is taken to be the confidence measure. Additionally, we extend this approach using normalization: the model can assign probability mass to multiple possible expressions of TRUE or FALSE or even other variations (e.g, “ True”, “ true”, “ ”, ). Thus when extending to the normalized form (Ask T/F N), we take the fraction of probability mass “True” assigned between only “True” and “False”.

Our experiments consider the four above: average token probability, generated sequence probability, verbalized self-evaluation, and question answering logit. In addition, as a baseline, we also used the length of the generated sequence. The length baseline is calculated based on the number of characters in the generated sequence, scaled such that $0$ is the shortest value in the dataset, and $1$ is the longest.

For reflective measures, we expected that a code generation model should perform well (and provide well-calibrated confidence scores for correctness): first, for synthesis, given just a good natural language description (without tests), of the desired function; second, for completion & bug-fixing, given the surrounding context. In both settings, we measure correctness using available hidden test cases. We also note that including hidden or failing tests in an LLM prompt is not common experimental practice for the tasks we analyze. [19, 20, 21].

Using model’s confidence in its generated output, and a way of determining correctness, one can compute measures of calibration. Calibration measures conceptually arise from the Reliability Plot, which plots correctness vs. confidence

Two reliability plots illustrating this method are shown in Figure 1. 1(a) is for token-level code completions from Codex; it shows the observed proportion of exact-match correct tokens (y-axis) vs. the predicted probability i.e., confidence as per the language model, based on bucketing observations into subsets $S_{1},S_{2},\ldots S_{n}$. Here, there are $n=10$ buckets, equally spaced by confidence measure. Each bucket has an associated bar whose height indicates the proportion (value $\in[0,1]$) of correct samples in the bucket. The closer the bars in each $S_{i}$ are to the diagonal line, the better the calibration.

We note here that Codex is a large model, well-trained on the task of token-level completion; thus, it is both well-calibrated and generally correct on the simple token-level completion task. However, for notions of correctness farther from the training objective, such as line-level code completion and test-passing, Codex’s intrinsic probability may not be as well-calibrated. An example (1(b)) where the confidence is not well-calibrated is Gpt-3.5 for line completion using verbalized confidence (i.e., asking it to write its confidence; see Section III-B).

We study two measures of calibration: Brier Score [8] and $\mathit{ECE}$ [9]. Both measure the deviation from perfect calibration. As before (following [22]), we assume a model $M$, input $X$, actual desired output $Y$, and model prediction $M(X)=\hat{Y}$. In our case, both $Y$ and $\hat{Y}$ are code, rather than a single label. Calibration measures indicate the extent to which the deviations of $\hat{Y}$ from the desired $Y$ are actually aligned with the model’s confidence in its output, $\hat{Y}$.

From the calibration plot, with the evaluation set $T$ bucketed into subsets $S_{i},i=1\ldots m\quad\text{s.t.}$ $\quad\bigcup_{i}S_{i}=T,\quad\forall i\in 1\ldots m$. We estimate correctness in each bucket as the fraction of predictions in that bucket which are “correct” (as discussed in § III-A); confidence is the average estimated probability from the model in that bucket:

where the model generates the code $M(x_{i})$ with confidence (probability) $\hat{p}_{i}$, and $\checkmark M(x_{i})$ indicates that the generated code is correct, as per the operative experimental definition. For a perfectly calibrated $M$, we would have $\text{corr}(S_{i})=\text{conf}(S_{i})\quad\forall i\in 1\ldots m$. In practice, we observe deviations from this ideal. Expected Calibration Error ($\mathit{ECE}$) [9] is a typical measure of calibration, calculated as the weighted average of these deviations.

$\mathit{ECE}$ is intuitive, but can mislead; as seen below, a naïve predictor whose confidence is always the base rate would yield a deceptively low $\mathit{ECE}$ value. An alternative measure, the Brier score, ${\mathcal{B}}$ [8] avoids this issue; it is calculated as follows: \useshortskip

One can achieve an optimal Brier of $\mathcal{B}\approx 0$ when confidently estimating $\hat{p}_{i}\approx 1$ when the code is correct, and estimating $\hat{p}_{i}\approx 0$ when the code is incorrect for each sample.

For comparison, consider using the Unskilled Reference Brier Score, $\mathcal{B}_{ref}$, attainable by a naïve, “unskilled” model, which simply assigns the base-rate $p_{r}$ as its confidence for every prediction. Here, all prediction confidence values are in one bin, the value $p_{r}$; and the empirical correctness in this single bin is the base rate $p_{r}$; so $\mathit{ECE}\approx 0$ which is misleading (thus exemplifying one of the weaknesses of $\mathit{ECE}$). The closed-form Brier Score for this unskilled predictor is: \useshortskip

In a 50-50 coinflip scenario (assuming heads is correct), a naïve predictor that randomly guesses correct with 50% confidence receives $\mathcal{B}_{ref}=0.5*(1-0.5)\approx 0.25$. Higher base rates yield lower $\mathcal{B}_{ref}$; e.g., for the MBPP dataset [20], Gpt-3.5 generates test-passing solutions for about 72% of the programming problems; here always guessing correct with 72% confidence results in $\mathcal{B}_{ref}=0.72*0.28\approx 0.20$. With well-calibrated confidence scores, a “skilled” model can achieve Brier Scores lower than this unskilled $\mathcal{B}_{ref}$ value; if a model does worse, it is indicative of poor calibration. Thus, one commonly reports a Skill Score ($SS$), calculated thus: \useshortskip

Positive $SS$ (perfect score = 1.0) indicates improvement over baseline $\mathcal{B}_{ref}$; negative indicates worse calibration than the baseline. Small positive values of $SS$ can sometimes indicate good skill. For example, the Deutsche Wetterdienst (German weather forecasting service) considers 0.05 Skill Score to be a minimum threshold for a good forecast quality⁴⁴4www.dwd.de/EN/ourservices/seasonals_forecasts/forecast_reliability.htm. As another data point, the American data journalism site 538 reports a skill of around 0.13 in forecasting World Cup games⁵⁵5projects.fivethirtyeight.com/checking-our-work/, which is in the range of what we observe in our experiments for best case code generation by LLMs; but these LLM performance numbers are just a starting point, and can be expected to improve in the future. $\mathit{ECE}$ (Equation 4) and Brier Score (Equation 5) serve slightly different purposes: the Brier Score is calculated for each sample, and measures both the ability to correctly discriminate output categories, and calibration of the output probability. The $\mathit{ECE}$ measures just calibration; but it can be misleadingly low, as noted above for the unskilled predictor. Additionally binning must be done carefully, since it can affect $\mathit{ECE}$ scores [23].

Machine learning models are not always calibrated. [22] discuss ways of rescaling probability estimates to better match observations. A common approach is Platt scaling [24], where a logistic regression is fit to the logit values of the prediction i.e., the $\ln$ of the measured confidence probability. This optimizes two parameters, a linear scaling multiplier and a bias i.e., intercept, that shifts the value.

To reduce the likelihood that the scaling overfits & skews our results, we rescale over five folds; i.e., we fit a logistic regression on a random $\nicefrac{{4}}{{5}}$ of data and apply it to $\nicefrac{{1}}{{5}}$ of data, before sliding over and doing each combination of $\nicefrac{{4}}{{5}}$.

Besides Platt scaling, temperature rescaling has also been used [25, 18, 22]: this approach applies a scalar multiplier on the logits representing each class e.g., a multiclass image classifier. In our binary confidence case, this has similar expressivity to Platt scaling without an intercept. Other approaches include histogram binning [26], isotonic regression [27], inter alia [22, 28]. These approaches are more parameterized; given the data limitations in our experimental setup e.g., a few hundred examples in function synthesis, they pose higher risk of overfitting.

As we discuss in Section IV, Platt scaling does improve calibration, with some caveats.

We explore confidence calibration for three code generation models. These include OpenAI Gpt-3.5 ⁹⁹9The gpt-3.5-turbo-instruct model, https://platform.openai.com/docs/models/gpt-3-5-turbo, OpenAI Codex [19], and CodeGen2-16B [43]. We sample from the models with temperature of 0, consistent with the reality that busy developers typically look at just the first suggestion in the completion [44]. For function synthesis task, temperature zero is most accurate and is fairly standard practice when doing pass@1 with only one solution to generate and present  [19, 13].

Table IV shows the results after applying Platt scaling to all measures (See § III-D). 2(a) shows a reliability plot before rescaling, and its equivalent after rescaling in 2(b). Rescaling can improve calibration. Considering all values, $ECE$ improves from an average of 0.32 to 0.03. For just those measures with a post-scaling $SS$ of at least 0.05, $ECE$ improves from 0.38 to 0.05.

The two logit-based reflective measures $p_{B}$ & $p_{NB}$ are usually strongly positively correlated with one another, since they are calculated from similar numbers. The reflective verbalized self-ask confidence measure $p_{v}$, and the two logit-based reflective confidence measures have no consistent relationships.

For function synthesis with raw measures, $p_{NB}$ shows best calibration for Gpt-3.5, slightly better than Unskilled, for HumanEval. For program repair, we observe the strongest best-case performance with regards to the metrics. Both Gpt-3.5 and Codex show positive $SS$ and low $ECE$ for $p_{B}$, but they are inconsistent after normalization ($p_{NB}$). These metrics suggest with reflection, these models’ confidence is calibrated regarding repair correctness; however, further analysis (see Figure A3) does not indicate good calibration on this task, from any confidence measure. We find that in general, the intrinsic vs. reflective measure values have no consistent relationship, even for a given model, dataset and task.

This lack of relationship may not necessarily be negative: e.g., perhaps the model’s reflective, prompted confidence may be better calibrated, as suggested by prior work [50, 20]. Without rescaling or few-shot prompting, reflective results are inconsistent. In some cases, such as Gpt-3.5 HumanEval and Defects4J, there are signs of calibration with slightly positive $SS$ values and $ECE$ values less than 0.2. Normalizing the T/F values induces some difference; but there are inconsistencies vis-à-vis tasks and models. For nonscaled Gpt-3.5 results, $p_{NB}$ improves calibration in line completion and function synthesis by an average of -0.34 $SS$ and 0.19 $ECE$, but not for program repair or for CodeGen2. Rescaling generally removes any sign of a normalization trend. For the alternative reflective approach of verbalization, the probability is not well calibrated for these models on the studied SE tasks.

In some cases, the reflective approaches are best calibrated without rescaling (see Table III), and show signs of being more robust when reusing learned rescaling parameters on unseen tasks (see Figure A7).

We investigated the impact of few-shotting viz. providing a model completion and correctness as part of the $p_{NB}$ prompt, on calibration [18]. To effectively perform few-shotting, we need a model that is instruction tuned and sufficiently large, which is best matched by Gpt-3.5. We explore few-shotting for the widely-used line completion task.

We perform the experiment with 5-shots consisting of prior completions from the same experiments presented in Table IV, the reflective question, and the ground truth True/False. We try two variants of this experiment, one where the examples are randomly selected, and one where they are chosen based on the similarity to the unanswered prompt. In both cases, we exclude the ground truth result for the unanswered prompt. We focus on Line Completion for this experiment as it represents widespread use and has a large number of examples available.

For line completion, the non-scaled results using random examples did not result in improved calibration over the baseline $p_{NB}$, however using BM25 [6] to select similar examples yielded a positive $SS$ of 0.08, which could be improved further by rescaling, up to 0.15. This result notably exceeds any other measure for DyPyBench, and significantly improves over the baseline $p_{NB}$ $SS$ of 0.

While random few-shotting requires limited extra data, BM25 is more similar to rescaling, in that it is dependent on a larger set of ground truths. This could be actualized by logging user completions, and building up ground truths (on if the completion was correct) based off the test case runs or acceptance of completions.

Alternative and improved ways of prompting (e.g., different verbalization formats [15, 17], fine-tuning [18, 15], chain-of-thought [51, 17], etc.) may alter these findings and are areas for future work.