Evaluating real-time probabilistic forecasts with application to National Basketball Association outcome prediction

Probabilistic predictions and forecasts are ubiquitous in modern society, and many individuals consider, and make decisions based on, such forecasts on a routine basis. For example, in the United States probability of precipitation forecasts became widely publicly available starting in the late 1960’s, and are now a critical factor in countless people’s daily decisions (National Research Council,, 2006; Murphy,, 1998). Over time the number and scope of probabilistic forecasts readily accessible to the public has increased at a steady pace, and now covers prediction of phenomena ranging from sports (Silver et al.,, 2019), to politics (Erikson and Wlezien,, 2012), to medicine (Spiegelhalter,, 1986), to geology (Gomberg,, 2015), among many other, some more exotic (Rowe and Beard,, 2018), areas.

Many such forecasts are made initially well before the event in question occurs, and are then continuously updated as new information becomes available. The example that we focus on throughout this paper is basketball game outcome prediction in the National Basketball Association (NBA). Websites like espn.com, the main web page of the United States-based multinational sports network, ESPN, publish and update in real-time probabilistic forecasts of the home team winning for each NBA game played. Although the method by which ESPN produces these forecasts is largely proprietary, ostensibly initial probability forecasts of the home team winning are constructed based on information that is available before the game starts, e.g. the usual home court advantage in the NBA, relative team strength, player injuries, etc., and then after the game commences and progresses these forecasts are updated with new information such as the score, game time remaining, ball possession, fouls, in game player injuries, etc. The resulting probabilistic forecasts and their fluctuations may be viewed as a curve that is a function of the in-game time; see Figure 1 for an example. Such curves arise in any similar continuously updated probabilistic forecasting task, and are evidently not unique to basketball game outcome prediction; see e.g. Silver, (2020).

Natural questions to ask when faced with any probabilistic forecast, including those that are continuously updated, are “are these forecasts accurate?” and “could these forecasts be improved upon?”. Following the seminal work of Murphy and Winkler, (1987) and Murphy and Winkler, (1992) on evaluating the quality of probabilistic forecasts in meteorology, evaluating a method for producing probabilistic forecasts is often broken into the tasks of measuring its calibration and skill. A model is deemed well-calibrated if its forecasts are compatible with the observed outcomes. In other words, a model that predicts an outcome with a given probability is well-calibrated if the relative frequency that the outcome occurs matches the probability forecast in the long run. A model is deemed to have higher skill than a competing model if its predictions are “sharper” or “more concentrated” than its competitor. For example, a model that forecasts the probability of a rainy day in New York City on a given day with the long run background rate of rainy days (which happens to be about 33.1% for New York City), will in the long run be well-calibrated, but has less skill than a model, perhaps based on more complete weather data, that correctly predicts rainy days with probabilistic forecasts of zero and one. Excellent reviews and more in-depth discussions of these concepts can be found in Gneiting et al., (2007) and Gneiting and Katzfuss, (2014).

The goal of this paper is to develop simple, easily interpretable, tools for evaluating the calibration and skill of methods to produce continuously updated probabilistic forecasts, and to apply these methods to evaluate the probabilistic forecasts pertaining to NBA basketball game outcome prediction published on ESPN. In terms of evaluating model calibration of probabilistic forecasts, standard tools are reliability diagrams and calibration plots, in which outcome frequencies are plotted against binned forecast probabilities; see Gneiting et al., (2007). Below we show how such curves can be extended in the continuously updated case to calibration surfaces, and how such surfaces can be summarized to show at a glance whether a given method is well-calibrated. In order to evaluate the relative skill of one continuously updated forecasting model against another, we employ the method of Lai et al., (2011) to construct confidence intervals for the average loss difference measured by the Brier score (Brier,, 1950) between the two models at given time points throughout the updating process. Estimating these intervals pointwise can be used to construct a simple graphical summary of the relative skill of one model versus another. In order to measure the cumulative statistical significance of differences observed in such a plot, we develop a new significance test for the skill-differences aggregated across time based on a novel large sample result for estimating continuous loss difference curves.

For the purpose of demonstrating these methods and evaluating ESPN’s forecasts, we introduce a number of “competing” continuously updated forecasting methods for basketball outcome prediction. Some are designed to be “straw men” for the purpose of demonstration, whereas others are based on logistic or probit generalized linear models making use of in-game information such as the score difference. We show using our methods that ESPN’s model is generally well-calibrated, and exhibits significantly better skill than some naïve models, although it does not demonstrate superiority over relatively simple logistic regression models based on the score difference and relative team strength alone.

The rest of the paper is organized as follows. Section 2 introduces the details of the ESPN forecasting data that we consider, as well as some competing forecasting methods that we develop and use for the purpose of comparison. Section 3 discusses the construction of calibration surfaces for such forecasts, as well as simple graphical summaries of these surfaces. Section 4 explains the proposed methods to evaluate the relative skill of two sets of real-time probabilistic forecasts. A Monte Carlo simulation study of these methods is given in Section 5. A detailed comparison of the ESPN forecasts as well as those of the proposed models is given in Section 6. Technical details are provided in Appendix A following these sections.

The specific data that we consider and that motivates this work are play-by-play records and real-time probabilistic forecasts of NBA regular season games downloaded from espn.com/nba (ESPN, (2020)). The NBA is a major professional basketball league, which is often referred to as one of the “Big Four” professional sports leagues in North America. Since 2004, except for the lockout season in 2011 and the COVID-19-influenced season of 2020, the NBA is comprised of $30$ teams, with each team playing a schedule of $82$ games in the regular season.

Starting in the 2017 - 2018 NBA season, ESPN Analytics began providing real-time in-game probabilistic forecasts of the home team winning for each NBA game played; an example of the forecasts from one game is shown in Figure 1. The data available from ESPN are quite rich, including real-time information about details such as substitutions, fouls, and ball possession. We consider here only a subset of these data that includes the real-time probabilistic forecasts provided by ESPN, as well as the evolution of the score throughout the game, for the 2017-2018 and 2018-2019 seasons. These data are updated each time there is an “event” in the game, which includes primarily score changes, fouls, and changes of possession. A typical game features between 460-480 events.

We excluded a small portion of these data from our analysis due to two issues. Quite often, multiple events will occur at the same instant in a game. One of the main examples that contributes to this is multiple players substituting at the same time. Although these events are all logged at the same time point, they occur in the dataset in an ordered sequence. The forecasted probabilities published by ESPN during such an event are typically contingent on this order. Therefore, we simply average the forecasts together in such a scenario to produce a probabilistic forecast at that instant. We also tried a number of other ways to handle this situation, such as using the first or last probabilistic forecast among the events recorded, and the difference in the results was negligible.

The second issue is due to games that go to overtime. If two teams’ scores are tied at the end of the 48-minute regulation game time, the teams will play an extra five-minute overtime period. For such games we remove the overtime period from the analysis, and only consider probabilistic forecasts up to the end of the game so that they are comparable to those that arise from games that did not go to overtime. Overtime games represent slightly less than 10% of the total games. Additionally, a small number of data points were discarded due to evident defects or excessive missing values.

The remaining data that we analyze are summarized in Table 1, and in each season there are more than $1100$ games with a total of over 350,000 play-by-play records available. Below we use the data from the 2017-2018 season as training data for our own models, and then we produce and evaluate forecasts for the 2018-2019 season.

Letting $N$ denote the total number of games with forecasts that we wish to evaluate, so $N=1213$ when we consider the 2018-2019 forecasts, the data may then be denoted as $\hat{p}_{i}^{ESPN}(t)$, $1\leq i\leq N$, $t\in[0,1]$ representing the probabilistic forecasts of the home team winning in the $i^{\prime}th$ game at intra-game time $t$. We assume that the game time parameter $t$ is normalized to be between zero and one so that it represents the proportion of the game complete. Although these forecasts are only available when events occur, due to the fact that events are very dense throughout the game, we simply interpolate these forecasts linearly to produce full probability forecast curves over $[0,1]$, which also makes them more comparable from one game to the next. This is illustrated in Figure 1. We also consider the data $H_{i}(t)$ and $A_{i}(t)$, $1\leq i\leq N$, $t\in[0,1]$, denoting the home team score and away team score in the $i^{\prime}th$ game, respectively, at proportion $t$ of the game. In our analysis below we frequently make use of the score difference $ScD_{i}(t)=H_{i}(t)-A_{i}(t)$, $1\leq i\leq N$, $t\in[0,1]$.

The goal of the methods that we develop below is to evaluate the quality of the forecasts $\hat{p}_{i}^{ESPN}(t)$, $1\leq i\leq N$, $t\in[0,1]$. To do this we also develop a number of benchmark models that are used for the purpose of comparison.

We now turn to the task of evaluating the calibration for a given set of continuously updated forecasts $\hat{p}_{i}(t)$ with realized outcomes $Y_{i}$, $i=1,...,N$. As mentioned in the introduction, traditionally when evaluating such forecasts one often considers what are called calibration plots or reliability diagrams. A calibration plot is a plot of binned probabilistic forecasts against the conditional event frequency associated to forecasts in a given bin. Since well-calibrated forecasts should have that the event frequency match the forecast probability, calibration may then be measured by comparing these points against a 45-degree diagonal reference line. Large departures from this line thus indicate poor calibration (cf. (Dawid,, 1986), (Murphy and Winkler,, 1992) and (Ranjan and Gneiting,, 2010)). We refer the reader to Gneiting and Katzfuss, (2014) and the references therein for a more comprehensive discussion.

One clear method then to check for calibration of continuously updated forecasts is to produce a calibration plot for each $t\in[0,1]$ based on the pairs $(\hat{p}_{i}(t),Y_{i})$. While this is in essence what we propose, there are two main challenges in doing so. (1) Traditionally when producing calibration plots, the forecasted probabilities are binned into fixed bins, commonly by deciles. For example, often the event frequency corresponding to all forecasted probabilities between $[0,0.1)$ are compared to 0.05, similarly for [0.1,0.2) to $0.15$, and so on. With continuously updated forecasts, and as with the ESPN forecasts, it is typical that the forecasts fluctuate so that for certain time points $t$ the forecasts cluster around some fixed values, and are hence far from being uniformly distributed in such fixed bins. In the case of the ESPN forecasts, near the end of game the majority of forecasts are clustered around 0 and 1. Fixed bins will often have the problem that the forecasts within them are not uniformly distributed within the bin. (2) Having constructed calibration plots for each $t$, one must examine a large number of such plots to pinpoint if a method appears to be well-calibrated, or to diagnose if there are some subset of times $t$ at which the method is more or less calibrated than others. A simple summary of the many calibration plots produced would be useful.

In order to address (1), we propose to use adaptive bins in constructing the calibration plots. Specifically, for each $t$, suppose we wish to construct $M$ bins for the forecasts $\hat{p}_{i}(t)$. By calculating the ranked forecasts $\hat{p}_{(i)}(t)$, $i=1,...,N$, we may group them into $M$ bins so that $\hat{p}_{(i)}(t)$ is in bin $j$ if $\left(\lfloor N/M\rfloor(j-1)+1\right)\leq i<\lfloor N/M\rfloor j$. We denote the collection of $\hat{p}_{i}(t)^{\prime}s$ in the $j$’th bin as $Bin_{j}$. In simpler terms, the forecasts at a given time $t$ are grouped into $M$ bins based on their rank. As a reference point or summary of the forecasts in the $j$’th bin, we use $\tilde{p}_{j}(t)=\mbox{Median}\left(\hat{p}_{(i)}(t),\;\;\left(\lfloor N/M\rfloor(j-1)+1\right)\leq i<\lfloor N/M\rfloor j\right)$. A calibration plot at time $t$ is constructed then by comparing $\tilde{p}_{j}(t)$ to $\bar{Y}_{j}(t)=Average(Y_{i},\mbox{ such that }\hat{p}_{i}(t)\in Bin_{j})$. Letting $n_{j}$ denote the number of forecasts in $Bin_{j}$, a 95% confidence interval for the mean of the events in $Bin_{j}$ could be constructed as

where $z_{\beta}$ denotes the $\beta$ quantile of the standard normal distribution, and here we have applied a Bonferroni correction to the significance level according to the number of bins used $M$. $\alpha$ is typically taken to be 5% so that 95% confidence intervals are computed.

The above interval is the standard confidence interval for the binomial proportion based on a normal approximation to the binomial random variable, as seen in most elementary statistics text books, and is often used to measure the uncertainty in calibration plots. Often though in such a continuously-updated setting, as with the ESPN forecast data, the standard interval is poor since the event frequency may be very close to zero or one in some bins at a subset of time points $t$. As discussed at length in Brown et al., (2001), a confidence interval that is in general recommended as a replacement to the standard interval, and is more robust to event frequencies close to zero and one, is the Wilson interval (Wilson,, 1927), which takes the form

where $\kappa=z_{\alpha/(2M)}$. We have noticed significant improvements by using the Wilson interval in this setting due to event frequencies that approach zero and one near the end of the game, and so we use these intervals in calibration plot construction below. Additionally, so that no bins contain predominately zero or one probability forecasts, we discard all forecasts to produce a calibration plot at a given $t$, and corresponding events, if $\hat{p}_{i}(t)<0.005$ or $\hat{p}_{i}(t)>0.995$, and we analyze those forecasts for calibration separately; see Table 2.

A calibration plot may then be made by plotting the bin references $\tilde{p}_{j}(t)$ against the above confidence intervals, and comparing these intervals to the reference line $y=x$. The method is deemed well-calibrated at the given significance level if the reference line generally goes through each interval. An example of this is shown based on the ESPN forecasts for $t=0.5$ using $M=10$ bins in the left hand panel of Figure 3(b) at the 95% confidence level. By linearly interpolating the upper bounds of these intervals across both the reference proportions $\tilde{p}_{j}(t)$ as well as $t$, we may construct an “upper calibration surface”, $U_{1-\alpha}(t,p)$. The upper 95% calibration surface $U_{0.95}(t,p)$ is displayed in the right hand panel of Figure 3(b) for the ESPN forecasts from the 2018-2019 season. A lower surface $L_{1-\alpha}(t,p)$ may be similarly constructed by linearly interpolating the lower bounds. A continuously updated forecasting method may then be deemed well-calibrated at a given significance level if the reference plane $f(t,p)=p$, for $t,p\in[0,1]$ is contained between both surfaces.

Although such surface plots are informative, it can be challenging to infer quickly based on these plots whether a given method appears to be calibrated. In order to produce a more easily interpretable summary of such calibration surface plots, we instead consider a plot for each $t$ of the minimum distance over $p$ between the reference plane $f(t,p)=p$, and the upper and lower calibration surfaces. Specifically, we consider plots of the functions $U^{min}_{1-\alpha}(t)=\min_{1\leq j\leq M}U_{1-\alpha}(t,\tilde{p}_{j}(t))-\tilde{p}_{j}(t)$ and $L^{max}_{1-\alpha}(t)=\max_{1\leq j\leq M}L_{1-\alpha}(t,\tilde{p}_{j}(t))-\tilde{p}_{j}(t)$ against $t$. If the upper and lower confidence surfaces contain the reference plane $f(t,p)=p$, then $U^{min}_{1-\alpha}(t)$ should always lie above zero, and $L^{max}_{1-\alpha}(t)$ should always lie below zero. Points with respect to $t$ at which this does not hold can be used to identify times for which a given method is not well-calibrated. We note here that in this case, due to the high degree of fluctuations in continuously updated probabilistic forecasts for basketball prediction, we have found it also useful to aid in the interpretation of these plots to smooth them with respect to $t$ using a simple moving average smoother over 5% of the game times. The conclusions drawn from these plots change little for different values of the window width.

These summary plots calculated based on the ESPN forecasts as well as from the benchmark models HomeWP, ScDnoInt, and PgRSScD using the logit link function are shown in Figure 4. From these we see that the naïve model HomeWP, which predicts that the home team will win each game at all times $t$ with the prior 10-year historic win rate of home teams in the NBA, is well-calibrated, as expected. Similar plots (not shown) for the method CF, which simply predicts that the home team will win with probability 50%, show that this method is not well-calibrated. Considering this plot for the method ScDnoInt (Panel (b) in Figure 4), we see that the forecasts are poorly calibrated at the beginning of the game, but the calibration improves towards the end of the game. This is expected since the corresponding logit model is taken to be free of an intercept term, and so the model is unable to capture the home team advantage which should force the forecast probabilities to favour the home team at the beginning of the game. Both the ESPN forecasts and those from the model PgRSScD, which incorporates team strength as well as the score difference, demonstrated generally good calibration for all game times.

A summary of the outcomes corresponding to games in which a probability forecast at some intra-game time exceeded 0.995 or was less than 0.005 is given in Table 2 for the ESPN forecasts as well as for the forecasts based on the models PgRSScD and ScDnoInt. The empirical home team win rate closely matched these thresholds in each case suggesting that the forecasts for each of these methods are reasonably well-calibrated at these extremal probability forecast levels.

Since many of the above methods are new when applied to continuously updated probabilistic forecasts, in this section we present the results of a small simulation study to investigate their respective performances, and illustrate their application in some controlled examples.

In this section, we apply the methods described in Section 4.1 to evaluate the skill of ESPN’s continuously updated probabilistic forecasts. In this case all GLM benchmark models are fit using the logit link function.

Figure 7 shows plots $\hat{\Delta}_{1213}(t)$ with conservative 95% confidence intervals, as well as approximate $p$-values of the test of $H_{0}$ for comparisons of the ESPN forecasts with the naïve models PgRS, ScD, LS, and PgRSLS. These plots suggest that, in aggregate, the ESPN forecasts significantly out perform these models. The specific points in the game at which the ESPN forecasts exhibit higher skill compared to these benchmarks is also clear in the plots. For the models that use relative team strength as encoded by ESPN’s initial home win probability as a covariate, the relative skill is similar to ESPN’s forecasts early in the game, and similarly those that make use of the score difference improve relative to the ESPN forecasts towards the end of the game. For example, the model based on the score difference alone is strongly outperformed by the ESPN forecasts early in the game, but their forecasts have indistinguishable skill towards the end of the game.

We also compared the ESPN forecasts to those of the somewhat less naïve model PgRSScD using a logit link. Figure 8 shows a plot $\hat{\Delta}_{1213}(t)$ based on the 2018-2019 season with conservative 95% confidence intervals, as well as approximate $p$-values of the test of $H_{0}$ for this comparison. In absolute terms, the estimated skill as measured by the Brier score generally favoured the simple logit model PgRSScD, with the exception of the last moments in the game. However, we do see from this analysis that the difference is apparently not statistically significant at the 5% level at any game time point based on the conservative confidence interval estimates, nor is it significant when the difference is aggregated across time points. We found it interesting that the ESPN’s sophisticated proprietary model, which ostensibly makes use of more nuanced information about the game status and more sophisticated models, did not significantly outperform a simple Logistic regression model. One might draw the conclusion based on this that whatever additional information used by ESPN’s model in producing these forecasts is not clearly beneficial for the purpose of forecasting, except for in the final moments of the game.

Motivated by evaluating forecasts of NBA basketball games, we have developed graphical tools and statistical tests for assessing the calibration and relative skill of continuously updated probabilistic forecasts. These were studied via a simulation study of synthetic “basketball games”, and applied to evaluating and comparing the forecasts published on ESPN and a number of competing models. In terms of calibration, the ESPN forecasts, as well as forecasts produced from simple logistic regression models using the in-game score difference and/or pre-game relative strength of teams as covariates, appear reasonably well-calibrated. In terms of skill, the ESPN forecasts exhibited significantly higher skill over naïve models, but did not demonstrate superiority over simple logistic regression models based on the score difference and relative team strength.

We conclude with a few remarks about some ideas that we considered but chose not to include in the paper, and avenues for future work. It is noteworthy that the confidence intervals defined in (6) may be made narrower and less conservative by using auxiliary information to improve the approximation of replacing $p_{i}(t)(1-p_{i}(t))$ with the upper bound $1/4$. We considered a number of methods to achieve this, including employing auxiliary models and covariates to estimate $p_{i}(t)(1-p_{i}(t))$ solely in the variance estimation step, but found both that this changed the intervals little, and lead to poor performance near the end of the game. For a discussion of similar methods, see Sections 3.4 and 3.5 in (Lai et al.,, 2011).

The basic reason why nested models often cannot be compared by means of the confidence intervals introduced and the test of $H_{0}$ derived using Theorem 1 is that nested models may lead to a covariance kernel $\hat{C}_{cons}$ that is approximately (and asymptotically) degenerate. It might be interesting to adapt this result, and the corresponding intervals and test, to this case, for instance using methods similar to those described in Clark and McCracken, (2015).

Lastly, we arrived after experimenting with many ways of constructing Figure 3 in Section 3 and the calibration plots in Figure 4 in Section 4.1 on using the Wilson interval with a bin-size of $10\%$. There are many other reasonable candidates to construct such intervals, including the typical normal approximation interval, and the Agresti and Coull interval (Agresti and Coull,, 1998), among several others, and we found that these performed similarly well with differences only appearing in the more extremal (with centers closer to 0 or 1) bins. Data driven approaches for deciding on an interval type as well as bin-size for these plots is worthy of further study.