How many labelers do you have?
A closer look at gold-standard labels

Chen Cheng Department of Statistics, Stanford University; email: [email protected]. Hilal Asi Department of Electrical Engineering, Stanford University; email: [email protected]. John Duchi Departments of Statistics and Electrical Engineering, Stanford University; email: [email protected].

Abstract

The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of “gold-standard”. We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models more feasible than it is with gold-standard labels. The entire story, however, is subtle, and the contrasts between aggregated and fuller label information depend on the particulars of the problem, where estimators that use aggregated information exhibit robust but slower rates of convergence, while estimators that can effectively leverage all labels converge more quickly if they have fidelity to (or can learn) the true labeling process. The theory makes several predictions for real-world datasets, including when non-aggregate labels should improve learning performance, which we test to corroborate the validity of our predictions.

1 Introduction

The centrality of data collection to the development of statistical machine learning is evident [12], with numerous challenge datasets driving advances [27, 25, 1, 22, 11, 37, 38]. Essential to these is the collection of labeled data. While in the past, experts could provide reliable labels for reasonably sized datasets, the cost and size of modern datasets often precludes this expert annotation, motivating a growing literature on crowdsourcing and other sophisticated dataset generation strategies that aggregate expert and non-expert feedback or collect internet-based loosely supervised and multimodal data [10, 20, 48, 37, 34, 38, 13]. By aggregating multiple labels, one typically hopes to obtain clean, true, “gold-standard” data. Yet most statistical machine learning development—theoretical or methodological—does not investigate this full data generating process, assuming only that data comes in the form of $(X,Y)$ pairs of covariates $X$ and targets (labels) $Y$ [45, 5, 2, 17]. Here, we argue for a more holistic perspective: broadly, that analysis and algorithmic development should focus on the more complete machine learning pipeline, from dataset construction to model output; and more narrowly, questioning such aggregation strategies and the extent to which such cleaned data is essential or even useful.

To that end, we develop a stylized theoretical model to capture uncertainties in the labeling process, allowing us to understand the contrasts, limitations and possible improvements of using aggregated or non-aggregated data in a statistical learning pipeline. We model each example as a pair $(X_{i},(Y_{i1},\dots,Y_{im}))$ where $X_{i}$ is a data point and $Y_{ij}$ are noisy labels. In the most basic formulation of our results, we compare two methods: empirical risk minimization using all the labels, and empirical risk minimization using cleaned labels $Y^{+}$ based on majority vote. While this grossly simplifies modern crowdsourcing and other label aggregation strategies [10, 37, 34], the simplicity allows (i) us to understand fundamental limitations of algorithms based on majority-vote aggregation, (ii) circumventing these limits by using full, non-aggregated information, and (iii) provides a potential base for future work: the purpose of the simplicity is to allow application of classical statistical tools. By carefully analyzing these models, our main results show that the error of models fit using non-aggregated label information outperforms that of “standard” estimators that use aggregated (cleaned, majority-vote) labels, so long as the model has fidelity to the data, while majority-vote estimators provide robust (but slower) convergence, and these tradeoffs are fundamental. We develop several extensions to these basic results, including misspecified models, semiparametric scenarios where one must learn link functions, and simple models of learned annotator reliability.

While our models are stylized, they also make several concrete and testable predictions for real datasets; if our approach provides a useful abstraction, it must suggest improvements in learning even for more complex and challenging to analyze scenarios. Our theory predicts that methods that fit predictive models on non-aggregated data should both make better-calibrated predictions and, in general, have lower classification error than models that use aggregated clean labels. To that end, we consider two real datasets as well as one large scale semisynthetic dataset, and they corroborate the predictions and implications of our theory even beyond logistic models. In particular, majority-vote based algorithms yield uncalibrated models in all experiments, whereas the algorithms that use full-label information train (more) calibrated models. Moreover, the former algorithms exhibit worse classification error in our experiments, with the error gap depending on parameters—such as inherent label noise—that we can also address in our theoretical models.

1.1 Problem formulation

To situate our results, we begin by providing the key ingredients in the paper.

The model with multiple labels.

Consider a binary classification problem with data points $X_{1},\dots,X_{n}\stackrel{{\scriptstyle\textup{iid}}}{{\sim}}\mathbb{P}_{X}$ , $X_{i}\in\mathbb{R}^{d}$ , with $m$ labelers. We assume each labeler annotates data points independently through a generalized linear model, and the labelers use $m$ possibly different link functions $\sigma_{1}^{\star},\dots,\sigma_{m}^{\star}\in\mathcal{F}_{\mathsf{link}}$ , where

\displaystyle\mathcal{F}_{\mathsf{link}}:=\left\{{\sigma:\mathbb{R}\to[0,1]\mid\sigma(0)=1/2,\mathsf{sign}(\sigma(t)-1/2)=\mathsf{sign}(t)}\right\}.

Here $\mathsf{sign}(t)=-1$ for $t<0$ , $\mathsf{sign}(t)=1$ for $t>0$ and $\mathsf{sign}(0)=0$ . If $\sigma(t)+\sigma(-t)=1$ for all $t\in\mathbb{R}$ , we say the link function is symmetric and denote the class of symmetric functions by $\mathcal{F}_{\mathsf{link}}^{0}\subset\mathcal{F}_{\mathsf{link}}$ . The link functions generate labels via the distribution

\displaystyle\mathbb{P}_{\sigma,\theta}(Y=y\mid X=x)

\displaystyle=\sigma(y\langle\theta,x\rangle)~{}~{}~{}\mbox{for~{}}y\in\left\{\pm 1\right\},~{}x,\theta\in\mathbb{R}^{d}.

(1)

Key to our stylized model, and what allows our analysis, is that we assume labelers use the same linear classifier $\theta^{\star}\in\mathbb{R}^{d}$ —though each labeler $j$ may have a distinct link $\sigma_{j}^{\star}$ —so we obtain conditionally independent labels $Y_{ij}\sim\mathbb{P}_{\sigma_{j}^{\star},\theta^{\star}}(\cdot\mid X_{i})$ . For example, in the logistic model where labelers have identical link, $\sigma_{j}^{\star}(t)=1/\left({1+e^{-t}}\right)$ . We seek to recover $\theta^{\star}$ or the direction $u^{\star}:=\theta^{\star}/\left\|{\theta^{\star}}\right\|_{2}$ from the observations $(X_{i},Y_{ij})$ .

Classification and calibration.

For an estimator $\widehat{\theta}$ and associated direction $\widehat{u}\coloneqq\widehat{\theta}/\|{\widehat{\theta}}\|_{2}$ , we measure performance through

	(i)	$\displaystyle\quad\text{The classification error:}~{}~{}~{}\\|u^{\star}-\widehat{u}\\|_{2}=\sqrt{2(1-\langle u^{\star},\widehat{u}\rangle)}.$
	(ii)	$\displaystyle\quad\text{The calibration error:}~{}~{}~{}~{}~{}~{}\\|\theta^{\star}-\widehat{\theta}\\|_{2}.$

We term these classification and calibration from the rationale that for classification, we only need to control the difference between the directions $\widehat{u}$ and $u^{\star}$ , while calibration—that for a new data point $X$ , the value $\sigma_{j}^{\star}(\langle\widehat{\theta},X\rangle)$ is close to $\mathbb{P}_{\sigma_{j}^{\star},\theta^{\star}}(Y=1\mid X)=\sigma_{j}^{\star}(\langle\theta^{\star},X\rangle)$ —requires controlling the error in $\widehat{\theta}$ as an estimate of $\theta^{\star}$ . As another brief motivation for calling (i) the classification error, note that if $X$ has rotationally symmetric distribution, then for any unit vectors $u,u^{\star}$ we have $\mathbb{P}(\mathsf{sign}(\langle X,u\rangle)\neq\mathsf{sign}(\langle X,u^{\star}\rangle))=\frac{1}{\pi}\cos^{-1}\langle u,u^{\star}\rangle$ ; because $\cos^{-1}t=\sqrt{2(1-t)}\cdot(1+o(1))$ as $t\uparrow 1$ , the error $\left\|{u^{\star}-u}\right\|_{2}=\sqrt{2(1-\langle u,u^{\star}\rangle)}$ is asymptotically equivalent to the angle between $u$ and $u^{\star}$ .

Estimators.

We consider two types of estimators: one using aggregated labels and the other using each label from different annotators. At the highest level, the aggregated estimator depends on processed labels $Y^{+}_{i}$ for each example $X_{i}$ , while the non-aggregated estimator uses all labels $Y_{i1},\ldots,Y_{im}$ . To center the discussion, we instantiate this via logistic regression (with generalizations in the sequel). For the logistic link $\sigma^{\textup{lr}}(t)=\frac{1}{1+e^{-t}}$ , define the logistic loss

\displaystyle\ell_{\theta}^{\textup{lr}}(y\mid x)=-\log\mathbb{P}_{\sigma^{\textup{lr}},\theta}(y\mid x)=\log(1+e^{-y\langle x,\theta\rangle}).

In the non-aggregated model, we let let $\mathbb{P}_{n,m}$ be the empirical measure on $\{(X_{i},(Y_{i1},\dots,Y_{im}))\}$ and consider the logistic regression estimator

\displaystyle\widehat{\theta}^{\textup{lr}}_{n,m}=\operatorname*{argmin}_{\theta}\left\{\mathbb{P}_{n,m}\ell_{\theta}^{\textup{lr}}=\frac{1}{nm}\sum_{i=1}^{n}\sum_{j=1}^{m}\ell_{\theta}^{\textup{lr}}(Y_{ij}\mid X_{i})\right\},

(2)

which is the maximum likelihood estimator (MLE) assuming the logistic model is true. We focus on the simplest aggregation strategy, where example $i$ has majority vote label

Y^{+}_{i}=\mathsf{maj}(Y_{i1},\ldots,Y_{im}).

Then letting $\overline{\mathbb{P}}_{n,m}$ be the empirical measure on $\{(X_{i},Y^{+}_{i})\}$ , the majority-vote estimator solves

\displaystyle\widehat{\theta}^{\textup{mv}}_{n,m}=\operatorname*{argmin}_{\theta}\left\{{\overline{\mathbb{P}}_{n,m}\ell_{\theta}^{\textup{lr}}=\frac{1}{n}\sum_{i=1}^{n}\ell_{\theta}^{\textup{lr}}(Y^{+}_{i}\mid X_{i})}\right\}.

(3)

Method (3) acts as our proxy for the “standard” data analysis pipeline, with cleaned labels, while method (2) is our proxy for non-aggregated methods using all labels. A more general formulation than the majority vote (3) could allow complex aggregation strategies, e.g., crowdsourcing, but we abstract away details to capture what we view as the essentiae for statistical learning problems (e.g. CIFAR [22] or ImageNet [37]) where only aggregated label information is available.

Our main technical approaches characterize the estimators $\widehat{\theta}^{\textup{mv}}_{n,m}$ and $\widehat{\theta}^{\textup{lr}}_{n,m}$ via asymptotic calculations. Under appropriate assumptions on the data generating mechanisms (1), which will include misspecification, we both (i) provide consistency results that elucidate the infinite sample limits for $\widehat{\theta}^{\textup{mv}}_{n,m}$ , $\widehat{\theta}^{\textup{lr}}_{n,m}$ , and a few more general estimators, and (ii) carefully evaluate their limit distributions via asymptotic normality calculations. The latter allows direct comparisons between the different estimators through their limiting covariances, which exhibit (to us) interestingly varying dependence on $m$ and the scaling of the true parameter $\theta^{\star}$ .

1.2 Summary of theoretical results and implications

We obtain several results clarifying the distinctions between estimators that use aggregate labels from those that treat them individually.

Improved performance using multiple labels for well-specified models

As in our discussion above, our main approach to highlighting the import of multiple labels is through asymptotic characterization of the estimators (2)–(3) and similar estimators. We begin this in Section 2 by focusing on the particular case that the labelers follow a logistic model. As specializations of our major results to come, we show that the multi-label MLE (2) is calibrated and enjoys faster rates of convergence (in $m$ ) than the majority-vote estimator (3). The improvements depend in subtle ways on the particulars of the underlying distribution, and we connect them to Mammen-Tsybakov-type noise conditions in Propositions 1 and 2. In “standard” cases (say, Gaussian features), the improvement scales as $\sqrt{m}$ ; for problems with little classification noise, the majority vote estimator becomes brittle (and relatively slower), while the convergence rate gap decreases for noisier problems.

Robustness of majority-vote estimators

Nonetheless, our results also evidence support for majority vote estimators of the form (3). Indeed, in Section 3 we provide master results and consequences that hold for both well- and mis-specified losses, which highlight the robustness of the majority vote estimator. While MLE-type estimators (2) enjoy faster rates of convergence when the model is correct, these rates break down when the model is mis-specified in ways we make precise; in contrast, majority-vote-type estimators (3) maintain their (not quite optimal) convergence guarantees, yielding $\sqrt{m}$ -rate improvements even under mis-specification, suggesting practical value of cleaning data when modeling uncertainty is impossible.

Fundamental limits of majority vote

In Section 4, we develop fundamental limits for estimation given only majority vote labels $(X,Y^{+})$ . We start with a simple result that any estimator based on majority vote aggregation cannot generally be calibrated. Leveraging local asymptotic minimax theory, we also provide lower bounds for estimating the direction $u^{\star}$ and parameter $\theta^{\star}$ given majority vote labels, which coincide with our results in Section 2. These results highlight both (i) that cleaned labels necessarily force slower convergence in the number of labelers $m$ and (ii) the robustness of majority-vote-based estimators: even with a mis-specified link function, they achieve rate-optimal convergence for procedures receiving only cleaned labels $Y^{+}$ .

Semi-parametric approaches

The final theoretical component of the paper (Section 5) provides an exemplar approach that puts the pieces together and achieves efficiency via semi-parametric approaches. While not the main focus of the paper, we highlight two applications. In the first, we use an initial estimator to fit a link function, then produce a refined estimator minimizing the induced semiparametric loss and recovering efficient estimation. In the second, we highlight how our results provide potential insights into existing crowdsourcing techniques by leveraging a blackbox crowdsourcing algorithm providing measures of labeler reliability to achieve (optimal) estimates.

Experimental evaluation

Our theoretical results predict two outcomes: first, that if the model has fidelity to the labeling process, then using all noisy labels should yield better estimates than procedures using cleaned labels, and second, that majority vote estimators should be more robust. In Section 6, we therefore provide real and semi-synthetic experiments that corroborate the theoretical predictions; the experiments also highlight a need, which some researchers are now beginning to address [13], for more applied development of actual datasets, so that we can more carefully evaluate the downstream consequences of filtering, cleaning, and otherwise manipulating the data we use to fit our models.

1.3 Related work

We briefly overview related work, acknowledging that to make our theoretical model tractable, we capture only a few of the complexities inherent in dataset construction. Label aggregation strategies often attempt to evaluate labeler uncertainty, dating to Dawid and Skene [10], who study labeler uncertainty estimation to overcome noise in clinical patient measurements. With the rise of crowdsourcing, such reliability models have attracted substantial recent interest, with approaches for optimal budget allocation [21], addressing untrustworthy or malicious labelers [8], and more broadly an intensive line of work studying crowd labeling and aggregation [48, 47, 42, 34], with substantial applications [11, 37].

The focus in many of these, however, is to obtain a single clean and trustworthy label for each example. Thus, while these aggregation techniques have been successful and important, our work takes a different perspective. First, we target statistical analysis for the full learning pipeline—to understand the theoretical landscape of the learning problem with multiple labels for each example—as opposed to obtaining only clean labels. Moreover, we argue for an increased focus on calibration of the resulting predictive model, which aggregated (clean) labels necessarily impede. We therefore adopt a perspective similar to Peterson et al. and Platanios et al.’s applied work [29, 32], which highlight ways that incorporating human uncertainty into learning pipelines can make classification “more robust” [29]. Instead of splitting the learning procedure into two phases, where the first aggregates labels and the second trains, we simply use non-aggregated labels throughout learning. Platanios et al. [32] propose a richer model than our stylized scenarios, directly modeling labeler reliability, but the simpler approaches we investigate allow us to be theoretically precise about the limiting behavior and performance of the methods.

Our results complement a new strain of experimental and applied work in machine learning. The Neural Information Processing Systems conference, as of 2021, includes a Datasets and Benchmarks track, recognizing that they “are crucial for the development of machine learning methods” [3]. Gadre et al. [13] introduce DataComp, a testbed for experimental work on datasets, building off the long history of dataset-driven development in machine learning [16]. We take a more theoretical approach, attempting to lay mathematical foundations for this line of work.

We mention in passing that our precise consistency results rely on distributional assumptions on the covariates $X$ , for example, that they are Gaussian. That such technical conditions appear may be familiar from work, for example, in single-index or nonlinear measurement models [31, 30]. In such settings, one assumes $\mathbb{E}[Y\mid X]=f(\langle\theta^{\star},X\rangle)$ for an unknown increasing $f$ , and it is essential that $\mathbb{E}[YX]\propto\theta^{\star}$ to allow estimation of $\theta^{\star}$ ; we leverage similar techniques.

Notation

We use $\left\|{x}\right\|_{p}$ to denote the $\ell_{p}$ norm of a vector $x$ . For a matrix $M$ , $\left\|{M}\right\|$ is its spectral norm, and $M^{\dagger}$ is its Moore-Penrose pseudo inverse. For a unit vector $u\in\mathbb{R}^{d}$ , the projection operator to the orthogonal space of $\mathrm{span}\{u\}$ is $\mathsf{P}_{u}^{\perp}=I-uu^{\top}$ . We use the notation $f(n)\asymp g(n)$ for $n\in\mathbb{N}$ and $f(x)\asymp g(x)$ for $x\in\mathbb{R}_{+}$ if there exist numerical constants $c_{1},c_{2}$ and $n_{0},x_{0}\geq 0$ such that $c_{1}|g(n)|\leq|f(n)|\leq c_{2}|g(n)|$ and $c_{2}|g(x)|\leq|f(x)|\leq c_{2}|g(x)|$ for $n\geq n_{0}$ and $x\geq x_{0}$ . We also use the empirical process notation $\mathbb{P}Z=\int zd\mathbb{P}(z)$ . We let $c=o_{m}(1)$ denote that $c\to 0$ as $m\to\infty$ .

2 The well-specified logistic model

We begin in a setting that allows the cleanest and most precise comparisons between a method using aggregated labels and one without by considering the logistic model for the link (1),

\displaystyle\sigma^{\textup{lr}}(t)=\frac{1}{1+e^{-t}}\in\mathcal{F}_{\mathsf{link}}^{0}.

This simplicity allows results that highlight many of the conclusions we draw, and so we present initial, relatively clean, results for the estimators (2) and (3) here. In particular, we assume identical links $\sigma_{1}^{\star}=\dots=\sigma_{m}^{\star}=\sigma^{\textup{lr}}$ , have an i.i.d. sample $X_{1},\ldots,X_{n}$ , where for each $i$ we draw $(Y_{i1},\ldots,Y_{im})\stackrel{{\scriptstyle\textup{iid}}}{{\sim}}\mathbb{P}_{\sigma^{\textup{lr}},\theta^{\star}}(\cdot\mid X_{i})$ for a true vector $\theta^{\star}$ .

2.1 The isotropic Gaussian case

To deliver the general taste of our results on the performance of the full information (2) and majority vote (3) approaches, we start by studying the simplest case when $X\sim\mathsf{N}(0,I_{d})$ .

Performance with non-aggregated data.

The non-aggregated MLE estimator $\widehat{\theta}^{\textup{lr}}_{n,m}$ in Eq. (2) admits a standard analysis [43], which we state as a corollary of a result where $X$ has more general distribution in Proposition 1 to come.

Corollary 1.

Let $X\sim\mathsf{N}(0,I_{d})$ and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . The maximum likelihood estimator $\widehat{\theta}^{\textup{lr}}_{n,m}$ is consistent, with $\widehat{\theta}^{\textup{lr}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}\theta^{\star}$ , and for $\mathsf{P}_{u^{\star}}^{\perp}=I_{d}-u^{\star}{u^{\star}}^{\top}$ ,

\displaystyle\sqrt{n}(\widehat{u}^{\textup{lr}}_{n,m}-u^{\star})

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1}\cdot(t^{\star})^{-2}\cdot\mathbb{E}[\sigma^{\textup{lr}}(\langle\theta^{\star},X\rangle)(1-\sigma^{\textup{lr}}(\langle\theta^{\star},X\rangle))]^{-1}\mathsf{P}_{u^{\star}}^{\perp}\right).

The first part of Corollary 1 demonstrates that the non-aggregated MLE classifier is calibrated: it recovers both the direction and scale of $\theta^{\star}$ . Moreover, the second part shows that this classifier enjoys convergence rates that roughly scale as $O(1)/\sqrt{nm}$ , so that a linear increase in the label size $m$ roughly yields a linear increase in convergence rate.

Performance with majority-vote aggregation.

The analysis of the majority vote estimator (3) requires more care, though the assumption that $X\sim\mathsf{N}(0,I_{d})$ allows us to calculate the limits explicitly. In brief, we show that when $X$ is Gaussian, the estimator is not calibrated and has slower convergence rates in $m$ for classification error than the non-aggregated classifier. The basic idea is that a classifier fit using majority vote labels $Y^{+}_{i}$ should still point in the direction of $\theta^{\star}$ , but it should be (roughly) “calibrated” to the probability of a majority of $m$ labels being correct.

We follow this idea and sketch the derivation here, as it is central to all of our coming theorems, and then state the companion corollary to Corollary 1. Each result depends on the probability

\mathbb{P}(Y^{+}=\mathsf{sign}(\langle x,\theta^{\star}\rangle)\mid x)=\rho_{m}(|\langle x,\theta^{\star}\rangle|)

of obtaining a correct label using majority vote, where $\rho_{m}$ defines the binomial probability function

\displaystyle\rho_{m}(t)=\mathbb{P}\left(\mathsf{Binomial}\Big{(}m,\frac{1}{1+e^{-|t|}}\Big{)}\geq\frac{m}{2}\right)=\sum_{i=\lceil m/2\rceil}^{m}\binom{m}{i}\left(\frac{1}{1+e^{-|t|}}\right)^{i}\left(\frac{e^{-|t|}}{1+e^{-|t|}}\right)^{m-i},

(4)

when $m$ is odd. (When $m$ is even the final sum has the additional additive term $\frac{1}{2}\binom{m}{m/2}\frac{e^{-m|t|/2}}{(1+e^{-|t|})^{m}}$ , which is asymptotically negligible.) Key to the coming result is choosing a parameter to roughly equalize binomial (majority vote) and logistic (Bernoulli) probabilities, and so for $Z\sim\mathsf{N}(0,1)$ we define the function

\displaystyle h_{m}(t)

\displaystyle=\mathbb{E}\left[{|Z|(1-\rho_{m}(t^{\star}|Z|))}\right]-\mathbb{E}\left[{\frac{|Z|}{1+e^{t|Z|}}}\right].

(5)

We use $h$ to find the minimizer of population loss $L^{\textup{mv}}_{m}(\theta)=\mathbb{E}[\ell_{\theta}^{\textup{lr}}(Y^{+}\mid X)]$ by considering the ansatz that $\theta=tu^{\star}$ for some $t>0$ . Using the definition (4) of $\rho_{m}$ , we can write

\displaystyle L^{\textup{mv}}_{m}(\theta)

\displaystyle=\mathbb{E}\left[\log(1+\exp(-S\langle X,\theta\rangle))\cdot\rho_{m}(t^{\star}|\langle X,u^{\star}\rangle|)\right]+\mathbb{E}\left[\log(1+\exp(S\langle X,\theta\rangle))\cdot(1-\rho_{m}(t^{\star}|\langle X,u^{\star}\rangle|))\right],

where $S=\mathsf{sign}(\langle X,\theta^{\star}\rangle)$ , and compute the gradient

\displaystyle\nabla L^{\textup{mv}}_{m}(\theta)

\displaystyle=-\mathbb{E}\left[\frac{S}{1+\exp(S\langle X,\theta\rangle)}\rho_{m}(t^{\star}|\langle X,u^{\star}\rangle|)X\right]+\mathbb{E}\left[\frac{S\exp(S\langle X,\theta\rangle)}{1+\exp(S\langle X,\theta\rangle)}(1-\rho_{m}(t^{\star}|\langle X,u^{\star}\rangle|))X\right].

We set $Z=\langle X,u^{\star}\rangle$ and decompose $X$ into the independent sum $X=(X-u^{\star}Z)+u^{\star}Z$ . Substituting in $\theta=tu^{\star}$ yields

$\displaystyle\nabla L^{\textup{mv}}_{m}(tu^{\star})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{(i)}}}{{=}}-\mathbb{E}\left[\frac{\mathsf{sign}(Z)}{1+\exp(t\|Z\|)}\rho_{m}(t^{\star}\|Z\|)X\right]+\mathbb{E}\left[\frac{\mathsf{sign}(Z)\exp(t\|Z\|)}{1+\exp(t\|Z\|)}(1-\rho_{m}(t^{\star}\|Z\|))X\right]$
	$\displaystyle=-\mathbb{E}\left[\frac{\mathsf{sign}(Z)Z}{1+\exp(t\|Z\|)}\rho_{m}(t^{\star}\|Z\|)\right]u^{\star}+\mathbb{E}\left[\frac{\mathsf{sign}(Z)Z\exp(t\|Z\|)}{1+\exp(t\|Z\|)}(1-\rho_{m}(t^{\star}\|Z\|))\right]u^{\star}$
	$\displaystyle=\left(\mathbb{E}\left[\frac{\|Z\|\exp(t\|Z\|)}{1+\exp(t\|Z\|)}\right]-\mathbb{E}\left[\|Z\|\rho_{m}(t^{\star}\|Z\|)\right]\right)u^{\star}$
	$\displaystyle=\left(\mathbb{E}\left[{\|Z\|(1-\rho_{m}(t^{\star}\|Z\|))}\right]-\mathbb{E}\left[{\frac{\|Z\|}{1+e^{t\|Z\|}}}\right]\right)u^{\star}=h_{m}(t)u^{\star},$	(6)

where in (i) we substitute $S=\mathsf{sign}(Z)$ . As we will present in Corollary 2, $h_{m}(t)=0$ has a unique solution $t_{m}\asymp\sqrt{m}$ , and the global minimizer of the population loss $L^{\textup{mv}}_{m}$ is thus exactly $t_{m}u^{\star}$ .

By completing the calculations for the precise value of $t_{m}$ above and a performing few asymptotic normality calculations, we have the following result, a special case of Proposition 2 to come.

Corollary 2.

Let $X\sim\mathsf{N}(0,I_{d})$ and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . There are numerical constants $a,b>0$ such that the following hold: for the function $h=h_{m}$ in (5), there is a unique $t_{m}\geq t_{1}=t^{\star}$ solving $h(t_{m})=0$ and

\widehat{\theta}^{\textup{mv}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}t_{m}u^{\star}~{}~{}\mbox{and}~{}~{}\lim_{m\to\infty}\frac{t_{m}}{t^{\star}\sqrt{m}}=a.

Moreover, there exists a function $C_{m}(t)=\frac{b}{t\sqrt{m}}(1+o_{m}(1))$ as $m\to\infty$ such that $\widehat{u}^{\textup{mv}}_{n,m}=\widehat{\theta}^{\textup{mv}}_{n,m}/\|{\widehat{\theta}^{\textup{mv}}_{n,m}}\|_{2}$ satisfies

\displaystyle\sqrt{n}\left(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star}\right)

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,C_{m}(t^{\star}){\mathsf{P}_{u^{\star}}^{\perp}}\right).

It is instructive to compare the rates of this estimator to the rates for the non-aggregated MLE in Corollary 1. First, the non-aggregated estimator is calibrated in that $\widehat{\theta}^{\textup{lr}}_{n,m}\to\theta^{\star}$ , in contrast to the majority-vote estimator, which roughly “calibrates” to the probability majority vote is correct (cf. (6)) via the convergence $\widehat{\theta}^{\textup{mv}}_{n,m}\to c\sqrt{m}\theta^{\star}$ as $n\to\infty$ . The scaling of $C_{m}$ in Corollary 2 is also important: the majority-vote estimator exhibits worse convergence rates by a factor of $\sqrt{m}$ than the estimator $\widehat{\theta}^{\textup{lr}}_{n,m}$ : for constants $c^{\textup{lr}}$ and $c^{\textup{mv}}$ that depend only on $t^{\star}=\|{\theta^{\star}}\|_{2}$ and $\Sigma=I-u^{\star}{u^{\star}}^{\top}$ , we have asymptotic variances differing by $\sqrt{m}$ :

\sqrt{n}(\widehat{u}^{\textup{lr}}_{n,m}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1}c^{\textup{lr}}\Sigma\cdot(1+o_{m}(1))\right)~{}~{}\mbox{while}~{}~{}\sqrt{n}(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1/2}c^{\textup{mv}}\Sigma\cdot(1+o_{m}(1))\right).

To obtain intuition for this result via comparison with Corollary 1, consider the Fisher Information $\mathbb{E}[\sigma^{\textup{lr}}(\langle\theta^{\star},X\rangle)(1-\sigma^{\textup{lr}}(\langle\theta^{\star},X\rangle))XX^{\top}]$ for logistic regression. Letting $Z^{\star}=\langle\theta^{\star},X\rangle$ be the predicted margin, there is only “information” available for an estimator when $\sigma^{\textup{lr}}(Z^{\star})$ is near $\frac{1}{2}$ , as otherwise $\sigma^{\textup{lr}}(Z^{\star})(1-\sigma^{\textup{lr}}(Z^{\star}))\approx 0$ ; under majority vote, we analogously require $\mathbb{P}(\mathsf{Binomial}(m,\sigma^{\textup{lr}}(Z^{\star}))>\frac{m}{2})\approx\frac{1}{2}$ , which in turn means that we need $\sigma^{\textup{lr}}(Z^{\star})\in\frac{1}{2}\pm O(\frac{1}{\sqrt{m}})$ by standard binomial concentration, or $Z^{\star}=\langle\theta^{\star},X\rangle\in\pm O(\frac{1}{\sqrt{m}})$ . This occurs on about $1/\sqrt{m}$ fraction of the sample, so of $n\cdot m$ total observations, majority vote “loses” a factor $\sqrt{m}$ .

2.2 Comparisons for more general feature distributions

The key to the preceding results—and an indication that they are stylized—is that the covariates $X$ decompose into components aligned with $\theta^{\star}$ and independent noise. Here, we abstract away the Gaussianity assumptions to allow a more general and nuanced development carefully tracking label noise, as margin and noise conditions play a strong role in the relative merits of maximum-likelihood-type (full label information) estimators versus those using cleaned majority-vote labels. The results, as in Sec. 2.1, are consequences of the master theorems to come in the sequel.

We first make our independence assumption.

Assumption A1.

The covariates $X$ have non-singular covariance $\Sigma$ and decompose as a sum of independent random vectors in the span of $u^{\star}$ and its complement

X=W+Zu^{\star},~{}~{}~{}\mbox{where}~{}~{}~{}W\perp\!\!\!\!\perp Z,~{}~{}\langle W,u^{\star}\rangle=0,~{}\mathbb{E}[W]=0,~{}\mathbb{E}[Z]=0.

Under these assumptions, we develop a characterization of the limiting behavior of the majority vote and non-aggregated models based on classification difficulty, adopting Mammen and Tsybakov’s perspective [26] and measuring difficulty of classification through the proximity of the probability $\mathbb{P}(Y=1\mid X=x)$ to $1/2$ . Thus, for a noise exponent $\beta\in(0,\infty)$ , we consider the condition

\mathbb{P}\left(\left|\mathbb{P}(Y=1\mid X)-\frac{1}{2}\right|\leq\epsilon\right)=O(\epsilon^{\beta}).

(

\textsc{M}_{\beta}

)

We see that as $\beta\uparrow\infty$ the problem becomes “easier” as it is less likely to have a small margin—in particular, $\beta=\infty$ gives a hard margin that $|\mathbb{P}(Y=1\mid X)-\frac{1}{2}|\geq\epsilon$ for all small $\epsilon$ . Under the independent decomposition Assumption A1, the noise condition ( $\textsc{M}_{\beta}$ ) solely depends on the covariate’s projection onto the signal $Z$ . We therefore consider the following assumption on $Z$ .

Assumption A2.

For a given $\beta>0$ , $Z$ is $(\beta,c_{Z})$ -regular, meaning that the absolute value $|Z|$ has density $p(z)$ on $(0,\infty)$ , no point mass at $0$ , and satisfies

\displaystyle\sup_{z\in(0,\infty)}z^{1-\beta}p(z)<\infty,\qquad\lim_{z\to 0}z^{1-\beta}p(z)=c_{Z}\in(0,\infty).

As the logistic function $\sigma^{\textup{lr}}(t)=1/(1+e^{-t})$ satisfies ${\sigma^{\textup{lr}}}^{\prime}(0)=1/4$ , for $t^{\star}=\|{\theta^{\star}}\|_{2}$ in our logistic model (1) we have $\mathbb{P}(Y=1\mid X=W+u^{\star}Z,Z=z)=\sigma^{\textup{lr}}(t^{\star}z)=1/(1+e^{-t^{\star}z})$ . More generally, for any link function $\sigma$ differentiable at $0$ with $\sigma^{\prime}(0)>0$ , we have $\mathbb{P}_{\sigma}(Y=1\mid Z=z)=\sigma(t^{\star}z)=\frac{1}{2}+\sigma^{\prime}(0)t^{\star}z+o(t^{\star}z)$ , so that the Mammen-Tsybakov noise condition ( $\textsc{M}_{\beta}$ ) is equivalent to

\displaystyle\mathbb{P}\left(\left|t^{\star}Z\right|\leq\epsilon\right)=O(\epsilon^{\beta}).

Thus, under Assumption A2, condition ( $\textsc{M}_{\beta}$ ) holds, as by dominated convergence we have

\displaystyle\mathbb{P}\left(|t^{\star}Z|\leq\epsilon\right)=\int_{0}^{\epsilon/t^{\star}}p(z)dz=\int_{0}^{\epsilon/t^{\star}}c_{Z}(1+o_{\epsilon}(1))z^{\beta-1}dz=\frac{c_{Z}}{\beta}\epsilon^{\beta}\cdot(1+o_{\epsilon}(1)),

As a concrete case, when the features $X$ are isotropic Gaussian and so $Z\sim\mathsf{N}(0,1)$ , $\beta=1$ . We provide extensions of Corollaries 1 and 2 in the more general cases the noise exponent $\beta$ allows.

The maximum likelihood estimator retains its convergence guarantees in this setting, and we can be more precise for the analogue of the final claim of Corollary 1 (see Appendix E.1 for a proof):

Proposition 1.

Let Assumptions A1 and A2 hold for some $\beta>0$ and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . Let $L^{\textup{lr}}(\theta)=\mathbb{E}[\ell_{\theta}^{\textup{lr}}(Y\mid X)]$ be the population logistic loss. Then the maximum likelihood estimator (2) satisfies

\sqrt{n}\left(\widehat{\theta}^{\textup{lr}}_{n,m}-\theta^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}(0,m^{-1}\nabla^{2}L^{\textup{lr}}(\theta^{\star})^{-1}).

Moreover,

\sqrt{n}\left(\widehat{u}^{\textup{lr}}_{n,m}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1}\cdot(t^{\star})^{-2}\mathsf{P}_{u^{\star}}^{\perp}\nabla^{2}L^{\textup{lr}}(\theta^{\star})^{-1}\mathsf{P}_{u^{\star}}^{\perp}\right),

and there exists $C(t)$ such that $\lim_{t\to\infty}C(t)t^{2-\beta}$ exists and is finite such that

\sqrt{n}\left(\widehat{u}^{\textup{lr}}_{n,m}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1}\cdot C(t^{\star})\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}\right).

For majority-vote aggregation, we can in turn generalize Corollary 2. In this case we still have $t_{m}\asymp\sqrt{m}$ . However, the interesting factor here is that the convergence rate now depends on the noise exponent $\beta$ .

Proposition 2.

Let Assumptions A1 and A2 hold for some $\beta\in(0,\infty)$ , and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . Suppose $h=h_{m}$ is the function (5) with $Z$ defined in Assumption A1. There are constants $a,b>0$ , depending only on $\beta$ and $c_{Z}$ , such that the following hold: there is a unique $t_{m}\geq t_{1}=t^{\star}$ solving $h(t_{m})=0$ and for this $t_{m}$ we have both

\widehat{\theta}^{\textup{mv}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}t_{m}u^{\star}~{}~{}\mbox{and}~{}~{}\lim_{m\to\infty}\frac{t_{m}}{t^{\star}\sqrt{m}}=a.

Moreover, there exists a function $C_{m}(t)=\frac{b}{(t\sqrt{m})^{2-\beta}}(1+o_{m}(1))$ as $m\to\infty$ such that

\displaystyle\sqrt{n}\left(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star}\right)

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,C_{m}(t^{\star})\left({\mathsf{P}_{u^{\star}}\Sigma\mathsf{P}_{u^{\star}}}\right)^{\dagger}\right).

We defer the proof to Appendix E.2.

Paralleling the discussion in Section 2.1, we may compare the performance of the MLE $\widehat{\theta}^{\textup{lr}}_{n,m}$ , which uses all labels, and the majority-vote estimator $\widehat{\theta}^{\textup{mv}}_{n,m}$ using only the cleaned labels. When the classification problem is hard—meaning that $\beta$ in Condition ( $\textsc{M}_{\beta}$ ) is near 0 so that that classifying most examples is nearly random chance—we see that the aggregation in the majority vote estimator still allows convergence (nearly) as quickly as the non-aggregated estimator; the problem is so noisy that data “cleaning” by aggregation is helpful. Yet for easier problems, where $\beta\gg 0$ , the gap between them grows substantially; this is sensible, as aggregation is likely to force a dataset to be separable, thus making fitting methods unstable (and indeed, a minimizer may fail to exist).

3 Label aggregation and misspecified model

The logistic link provides clean interpretation and results, but we can move beyond it to more realistic cases where labelers use distinct links, although, to allow precise statements, we still assume the same linear term $x\mapsto\langle\theta^{\star},x\rangle$ for each labeler’s generalized linear model. We study generalizations of the maximum likelihood and majority vote estimators (2) and (3), highlighting dependence on link fidelity. In this setting, there are $m$ (unknown and possibly distinct) link functions $\sigma_{i}^{\star}$ , $i=1,2,\ldots,m$ . We show that the majority-vote estimator $\widehat{\theta}^{\textup{mv}}_{n,m}$ enjoys better robustness to model mis-specification than the non-aggregated estimator $\widehat{\theta}^{\textup{lr}}_{n,m}$ , though both use identical losses. In particular, our main result in this section implies

\sqrt{n}(\widehat{u}^{\textup{lr}}_{n,m}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,c\Sigma\cdot(1+o_{m}(1))\right)~{}~{}\mbox{while}~{}~{}\sqrt{n}(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,m^{-1/2}c^{\textup{mv}}\Sigma\cdot(1+o_{m}(1))\right),

where $c$ and $c^{\textup{mv}}$ are constants that depend only on the links $\sigma$ , $t^{\star}=\|{\theta^{\star}}\|_{2}$ , and $\Sigma=I-u^{\star}{u^{\star}}^{\top}$ when $X\sim\mathsf{N}(0,I_{d})$ . In contrast to the previous section, the majority-vote estimator enjoys roughly $\sqrt{m}$ -faster rates than the non-aggregated estimator, maintaining its (slow) improvement with $m$ , which the MLE loses to misspecification.

To set the stage for our results, we define the general link-based loss

\ell_{\sigma,\theta}(y\mid x)\coloneqq-\int_{0}^{y\langle\theta,x\rangle}\sigma(-v)dv.

We then consider the general multi-label estimator and the majority-vote estimator based on the loss $\ell_{\sigma,\theta}$ ,

\widehat{\theta}_{n,m}(\sigma)\coloneqq\operatorname*{argmin}_{\theta}\mathbb{P}_{n,m}\ell_{\sigma,\theta},\qquad\widehat{\theta}^{\textup{mv}}_{n,m}(\sigma)\coloneqq\operatorname*{argmin}_{\theta}\overline{\mathbb{P}}_{n,m}\ell_{\sigma,\theta}.

(7)

When $\sigma=\sigma^{\textup{lr}}$ is the logistic link, we recover the logistic loss $\ell_{\sigma^{\textup{lr}},\theta}(y\mid x)=\ell_{\theta}^{\textup{lr}}(y\mid x)$ , and thus we recover the results in Section 2. For both the estimators, we suppress the dependence on the link $\sigma$ to write $\widehat{\theta}_{n,m},\widehat{\theta}^{\textup{mv}}_{n,m}$ when the context is clear.

3.1 Master results

To characterize the behavior of multiple label estimators versus majority vote, we provide master results as a foundation for our convergence rate analyses throughout. By a bit of notational chicanery, we consider both the cases that $Y^{+}$ is a majority vote and that we use multiple (non-aggregated) labels simultaneously. In the case that the estimator uses the majority vote $Y^{+}$ , let

\varphi_{m}(t)=\rho_{m}(t)\mathds{1}\{t\geq 0\}+(1-\rho_{m}(t))\mathds{1}\{t<0\},~{}~{}\mbox{where}~{}~{}\rho_{m}(t)\coloneqq\mathbb{P}(Y^{+}=\mathsf{sign}(\langle X,\theta^{\star}\rangle)\mid\langle X,\theta^{\star}\rangle=t),

and in the case that the estimator uses each label from the $m$ labelers, let

\varphi_{m}(t)=\frac{1}{m}\sum_{j=1}^{m}\sigma_{j}^{\star}(t)=\frac{1}{m}\sum_{j=1}^{m}\mathbb{P}(Y_{j}=1\mid\langle X,\theta^{\star}\rangle=t).

In either case, we then see that the population loss with the link-based loss $\ell_{\sigma,\theta}$ becomes

L(\theta,\sigma)=\mathbb{E}\left[\ell_{\sigma,\theta}(1\mid X)\varphi_{m}(\langle X,\theta^{\star}\rangle)+\ell_{\sigma,\theta}(-1\mid X)(1-\varphi_{m}(\langle X,\theta^{\star}\rangle))\right],

(8)

where we have taken a conditional expectation given $X$ . We assume Assumption A1 holds, that $X$ decomposes into the independent sum $X=Zu^{\star}+W$ with $W\perp u^{\star}$ , and the true link functions $\sigma_{j}^{\star}\in\mathcal{F}_{\mathsf{link}}$ . We further impose the following assumption for the model link.

Assumption A3.

For each sign $s\in\{-1,1\}$ , the model link function $\sigma$ satisfies $\lim_{t\to s\cdot\infty}\sigma(t)=1/2+sc$ for a constant $0<c\leq 1/2$ and is a.e. differentiable.

Minimizer of the population loss.

We begin by characterizing—at a somewhat abstract level—the (unique) solutions to the problem of minimizing the population loss (8). To characterize the minimizer $\theta^{\star}_{L}\coloneqq\operatorname*{argmin}_{\theta}L(\theta,\sigma)$ , we hypothesize that it aligns with $u^{\star}=\theta^{\star}/\left\|{\theta^{\star}}\right\|_{2}$ , using the familiar ansatz that $\theta$ has the form $\theta=tu^{\star}$ . Using the formulation (8), we see that for $t^{\star}\coloneqq\left\|{\theta^{\star}}\right\|_{2}$ ,

$\displaystyle\nabla L(\theta,\sigma)$	$\displaystyle=-\mathbb{E}[\sigma(-\langle\theta,X\rangle)X\varphi_{m}(\langle X,\theta^{\star}\rangle)]+\mathbb{E}[\sigma(\langle\theta,X\rangle)X(1-\varphi_{m}(\langle X,\theta^{\star}\rangle))]$
	$\displaystyle=-\mathbb{E}[\sigma(-tZ)X\varphi_{m}(t^{\star}Z)]+\mathbb{E}[\sigma(tZ)X(1-\varphi_{m}(t^{\star}Z))]$
	$\displaystyle=\left(-\mathbb{E}[\sigma(-tZ)Z\varphi_{m}(t^{\star}Z)]+\mathbb{E}[\sigma(tZ)Z(1-\varphi_{m}(t^{\star}Z))]\right)u^{\star}=h_{m}(t)u^{\star},$	(9)

where the final line uses the decomposition $X=Zu^{\star}+W$ for the random vector $W\perp u^{\star}$ independent of $Z$ , and we recall expression (5) to define the calibration function

h_{t^{\star},m}(t)\coloneqq\mathbb{E}[\sigma(tZ)Z(1-\varphi_{m}(t^{\star}Z))]-\mathbb{E}[\sigma(-tZ)Z\varphi_{m}(t^{\star}Z)].

(10)

The function $h$ measures the gap between the hypothesized link function $\sigma$ and the label probabilities $\varphi_{m}$ , functioning the approximately “calibrate” $\sigma$ to the observed probabilities. If we presume that a solution to $h_{t^{\star},m}(t)=0$ exists, then evidently $tu^{\star}$ is a minimizer of $L(\theta,\sigma)$ . In fact, such a solution exists and is unique (see Appendix B for a proof):

Lemma 3.1.

Let Assumption A1 hold and $h=h_{t^{\star},m}$ be the gap function (10). Then there is a unique solution $t_{m}>0$ to $h(t)=0$ , and the generic loss (8) has unique minimizer $\theta^{\star}_{L}=t_{m}u^{\star}$ . Define the matrix

\begin{split}H_{L}(t)&\coloneqq\mathbb{E}[(\sigma^{\prime}(-tZ)\varphi_{m}(t^{\star}Z)+\sigma^{\prime}(tZ)(1-\varphi_{m}(t^{\star}Z)))Z^{2}]u^{\star}{u^{\star}}^{\top}\\ &\qquad~{}+\mathbb{E}[\sigma^{\prime}(-tZ)\varphi_{m}(t^{\star}Z)+\sigma^{\prime}(tZ)(1-\varphi_{m}(t^{\star}Z))]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.\end{split}

(11)

Then the Hessian is $\nabla^{2}L(\theta^{\star}_{L},\sigma)=H_{L}(t_{m})$ .

Asymptotic normality with multiple labels.

With the existence of minimizers assured, we turn to their asymptotics. For each of these, we require slightly different calculations, as the resulting covariances are slightly different. To state the result when we have multiple labels, we define the average link function $\overline{\sigma}^{\star}=\frac{1}{m}\sum_{j=1}^{m}\sigma_{j}^{\star}$ and the three functions

$\displaystyle\mathsf{le}(Z)$	$\displaystyle\coloneqq\sigma(t_{m}Z)(1-\overline{\sigma}^{\star}(t^{\star}Z))-\sigma(-t_{m}Z)\overline{\sigma}^{\star}(t^{\star}Z),$
$\displaystyle\mathsf{he}(Z)$	$\displaystyle\coloneqq\sigma^{\prime}(-t_{m}Z)\varphi_{m}(t^{\star}Z)+\sigma^{\prime}(t_{m}Z)(1-\varphi_{m}(t^{\star}Z)),$	(12)
$\displaystyle v_{j}(Z)$	$\displaystyle\coloneqq\sigma_{j}^{\star}(t^{\star}Z)(1-\sigma_{j}^{\star}(t^{\star}Z))(\sigma(t_{m}Z)+\sigma(-t_{m}Z))^{2}.$

The first, the link error $\mathsf{le}$ , measures the mis-specification of the link $\sigma$ relative to the average link $\overline{\sigma}^{\star}$ . The second function, $\mathsf{he}$ , is a Hessian term, as $H_{L}(t_{m})=\mathbb{E}[\mathsf{he}(Z)Z^{2}]u^{\star}{u^{\star}}^{\top}+\mathbb{E}[\mathsf{he}(Z)]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}$ , and the third is a variance term for each labeler $j$ . We have the following theorem, which we prove in Appendix C.

Theorem 1.

Let Assumptions A1 and A3 hold, and let $\widehat{\theta}_{n,m}$ be the multilabel estimator (7). Define the shorthand $\overline{v}=\frac{1}{m}\sum_{j=1}^{m}v_{j}$ . Then $\widehat{\theta}_{n,m}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\theta_{L}^{\star}$ , and

\sqrt{n}(\widehat{\theta}_{n}-\theta_{L}^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{\mathbb{E}[\mathsf{le}(Z)^{2}Z^{2}]+m^{-1}\mathbb{E}[\overline{v}(Z)Z^{2}]}{\mathbb{E}[\mathsf{he}(Z)Z^{2}]^{2}}u^{\star}{u^{\star}}^{\top}+\frac{\mathbb{E}[\mathsf{le}(Z)^{2}]+m^{-1}\mathbb{E}[\overline{v}(Z)]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}\right).

Additionally, if $\widehat{u}_{n}=\widehat{\theta}_{n,m}/\|{\widehat{\theta}_{n,m}}\|_{2}$ and $t_{m}$ is the unique zero of the gap function $h_{t^{\star},m}(t)=0$ , then

\sqrt{n}(\widehat{u}_{n}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{1}{t_{m}^{2}}\frac{\mathbb{E}[\mathsf{le}(Z)^{2}]+m^{-1}\mathbb{E}[\overline{v}(Z)]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left(\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}\right)^{\dagger}\right).

Theorem 1 exhibits two dependencies: the first on the link error terms $\mathbb{E}[\mathsf{le}(Z)^{2}]$ —essentially, a bias term—and the second on the rescaled average variance $\frac{1}{m}\mathbb{E}[\overline{v}(Z)]$ . So the multi-label estimator recovers an optimal $O(1/m)$ covariance if the link errors are negligible, but if they are not, then it necessarily has $O(1)$ asymptotic covariance. The next corollary highlights how things simplify. In the well-specified case that $\sigma$ is symmetric and $\sigma=\overline{\sigma}^{\star}$ , the zero of the gap function (10) is evidently $t_{m}=t^{\star}=\left\|{\theta^{\star}}\right\|_{2}$ , the error term $\mathsf{le}(Z)=0$ , and $v_{j}(Z)=\sigma^{\star}(t^{\star}Z)(1-\sigma^{\star}(t^{\star}Z))$ , and by symmetry, $\sigma^{\prime}(t)=\sigma^{\prime}(-t)$ so that $\mathsf{he}(Z)=\sigma^{\prime}(t^{\star}Z)$ :

Corollary 3 (The well-specified case).

Let the conditions above hold. Then

\sqrt{n}(\widehat{u}_{n}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{1}{m}\cdot\frac{1}{\left\|{\theta^{\star}}\right\|_{2}^{2}}\frac{\mathbb{E}[\sigma(t^{\star}Z)(1-\sigma(t^{\star}Z))]}{\mathbb{E}[\sigma^{\prime}(t^{\star}Z)]^{2}}\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}\right).

Asymptotic normality with majority vote.

When we use the majority vote estimators, the asymptotics differ: there is no averaging to reduce variance as $m$ increases, even in a “well-specified” case. The asymptotic variance does (typically) decrease as $m$ grows, but at a slower rate, roughly related to the fraction of data where there is “signal”, as we discuss briefly following Corollary 2.

Theorem 2.

Let Assumptions A1 and A3 hold, and let $\widehat{\theta}_{n}=\widehat{\theta}^{\textup{mv}}_{n,m}$ be the general majority vote estimator (7). Let $t_{m}$ be the zero of the gap function (10), solving $h_{t^{\star},m}(t)=0$ . Then $\widehat{\theta}_{n}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\theta^{\star}_{L}=t_{m}u^{\star}$ , and $\widehat{u}_{n}=\widehat{\theta}_{n}/\|{\widehat{\theta}_{n}}\|_{2}$ satisfies

\sqrt{n}\left(\widehat{u}_{n}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{1}{t_{m}^{2}}\frac{\mathbb{E}[\sigma(-t_{m}|Z|)^{2}\rho_{m}(t^{\star}Z)+\sigma(t_{m}|Z|)^{2}(1-\rho_{m}(t^{\star}Z))]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}\right).

We defer the proof to Appendix D. In most cases, we will take the link function $\sigma$ to be symmetric, so that $\sigma(t)=1-\sigma(-t)$ , and thus $\sigma^{\prime}(t)=\sigma^{\prime}(-t)$ , so that $\mathsf{he}(z)=\sigma^{\prime}(t_{m}z)\geq 0$ . This simplifies the denominator in Theorem 2 to $\mathbb{E}[\sigma^{\prime}(t_{m}Z)]^{2}$ . Written differently, we may define a (scalar) variance-characterizing function $C_{m}$ implicitly as follows: let $t_{m}=t_{m}(t)$ be a zero of $h_{t,m}(s)=\mathbb{E}[\sigma(sZ)Z(1-\varphi_{m}(tZ))]-\mathbb{E}[\sigma(-sZ)Z\varphi_{m}(tZ)]=0$ in $s$ , that is, $h_{t,m}(t_{m}(t))=0$ so that $t_{m}$ is a function of the size $t$ (recall the gap (10)), and then define

C_{m}(t)\coloneqq\frac{1}{t_{m}^{2}}\frac{\mathbb{E}[\sigma(-t_{m}|Z|)^{2}\rho_{m}(tZ)+\sigma(t_{m}|Z|)^{2}(1-\rho_{m}(tZ))]}{\mathbb{E}[\sigma^{\prime}(t_{m}Z)]^{2}}

(13)

where $t_{m}=t_{m}(t)$ above is implicitly defined. Then

\sqrt{n}\left(\widehat{u}_{n}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,C_{m}(t^{\star})\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}\right).

Each of our main results, including those on well-specified models previously, then follows by characterizing the behavior of $C_{m}(t)$ in the asymptotics as $m\to\infty$ and the scaling of the solution norm $t_{m}=\|{\theta_{L}^{\star}}\|_{2}$ , which the calibration gap (10) determines. The key is that the scaling with $m$ varies depending on the fidelity of the model, behavior of the links $\sigma$ , and the noise exponent ( $\textsc{M}_{\beta}$ ), and our coming consequences of the master theorems 1 and 2 help to reify this scaling.

3.2 Robustness to model mis-specification

Having established the general convergence results for the multi-label estimator $\widehat{\theta}_{n,m}$ and the majority vote estimator $\widehat{\theta}^{\textup{mv}}_{n,m}(\sigma)$ , we further explicate their performance when we have a mis-specified model—the link $\sigma$ is incorrect—by leveraging Theorems 1 and 2 to precisely characterize their asymptotics and show the majority-vote estimator can be more robust to model mis-specification.

Multi-label estimator.

As our focus here is descriptive, to make interpretable statements about the multi-label estimator $\widehat{\theta}_{n,m}$ in (7), we simplify by assuming that each link $\sigma_{j}^{\star}\equiv\sigma^{\star}\in\mathcal{F}_{\mathsf{link}}$ is identical. Then an immediate corollary of Theorem 1 follows:

Corollary 4.

Let Assumptions A1 and A2 hold for some $\beta\in(0,\infty)$ , and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . Then the calibration gap function (10) has unique positive zero $h_{t^{\star},m}(t_{\sigma^{\star}})=0$ , and the multilabel estimator (7) satisfies

\widehat{\theta}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}t_{\sigma^{\star}}u^{\star}.

Additionally, the normalized estimate $\widehat{u}_{n,m}=\widehat{\theta}_{n,m}/\|{\widehat{\theta}_{n,m}}\|_{2}$ satisfies

\sqrt{n}\left(\widehat{u}_{n,m}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{\mathbb{E}[\mathsf{le}(Z)^{2}]+m^{-1}\mathbb{E}[\overline{v}(Z)]}{t_{\sigma^{\star}}^{2}\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}\Sigma\mathsf{P}_{u^{\star}}}\right)^{\dagger}\right).

So in this simplified case, the asymptotic covariance remains of constant order in $m$ unless $\mathbb{E}[\mathsf{le}(Z)^{2}]=0$ . In contrast, as we now show, the majority vote estimator exhibits more robustness; this is perhaps expected, as Corollary 2 shows that in the logistic link case, which is a fortiori misspecified for majority vote labels, has covariance scaling as $1/\sqrt{m}$ , though the generality of the behavior and its distinction from Corollary 4 is interesting.

Majority vote estimator.

For the majority-vote estimator, we relax our assumptions and allow $\sigma_{j}^{\star}$ to be different, showing how the broad conclusions Corollary 4 suggests continue to hold in some generality: majority vote estimators achieve slower convergence than well-specified (maximum likelihood) estimators using each label, but exhibit more robustness. To characterize the large $m$ behavior, we require the following regularity conditions on the average link $\overline{\sigma}_{m}^{\star}=\frac{1}{m}\sum_{j=1}^{m}\sigma_{j}^{\star}$ , which we require has a limiting derivative at $0$ .

Assumption A4.

For the sequence of link functions $\{\sigma_{j}\mid j\in\mathbb{N}\}\subset\mathcal{F}_{\mathsf{link}}$ , let $\overline{\sigma}_{m}^{\star}=\frac{1}{m}\sum_{j=1}^{m}\sigma_{j}^{\star}$ . There exists ${\overline{\sigma}^{\star}}^{\prime}(0)>0$ such that


$\displaystyle\bm{\mathrm{(i)}}\qquad$	$\displaystyle\lim_{m\to\infty}\sqrt{m}\left({\overline{\sigma}_{m}^{\star}\left({\frac{t}{\sqrt{m}}}\right)-\frac{1}{2}}\right)={\overline{\sigma}^{\star}}^{\prime}(0)t,\quad\textrm{for each }t\in\mathbb{R};$	(14a)
$\displaystyle\bm{\mathrm{(ii)}}\qquad$	$\displaystyle\liminf_{m\to\infty}\inf_{t\neq 0}\frac{\left\|\overline{\sigma}_{m}^{\star}(t)-\frac{1}{2}\right\|}{\min\left\{\|t\|,1\right\}}>0;$	(14b)
$\displaystyle\bm{\mathrm{(iii)}}\qquad$	$\displaystyle\lim_{t\to 0}\sup_{j\in\mathbb{N}}\left\|\sigma_{j}^{\star}(t)-\frac{1}{2}\right\|=0.$	(14c)

These assumptions simplify if the links are identical: if $\sigma_{j}^{\star}\equiv\sigma^{\star}$ , we only require $\sigma^{\star}$ is differentiable around $0$ with ${\sigma^{\star}}^{\prime}(0)>0$ and $|\sigma^{\star}(t)-\frac{1}{2}|\gtrsim\min\{|t|,1\}$ .

We can apply Theorem 2 to obtain asymptotic normality for the majority vote estimator (7). We recall the probability

\rho_{m}(t)\coloneqq\mathbb{P}(Y^{+}=\mathsf{sign}(\langle X,\theta^{\star}\rangle)\mid\langle X,\theta^{\star}\rangle=t)

(15)

of the majority vote being correct given the margin $\langle X,\theta^{\star}\rangle=t$ and the calibration gap function (10), which by a calculation case resolves to the more convenient form

\displaystyle h(t)

\displaystyle=h_{t^{\star},m}(t)=\mathbb{E}[\sigma(t|Z|)|Z|(1-\rho_{m}(t^{\star}Z))]-\mathbb{E}[\sigma(-t|Z|)|Z|\rho_{m}(t^{\star}Z)].

The main technical challenge is to characterize the large $m$ behavior for the asymptotic covariance function $C_{m}(t)$ defined implicitly in the quantity (13). We postpone the details to Appendix E.3 and state the result below, which is a consequence of Theorem 2 and a careful asymptotic expansion of the covariance function (13).

Proposition 3.

Let Assumptions A1 and A2 hold for some $\beta\in(0,\infty)$ with $\int_{0}^{\infty}z^{\beta-1}\sigma^{\prime}(z)dz<\infty$ and $t^{\star}=\|{\theta^{\star}}\|_{2}$ , and in addition that Assumption A4 holds and $\sigma$ is symmetric. Then there are constants $a,b>0$ , depending only on $\beta$ , $c_{Z}$ , $\sigma$ , and ${\overline{\sigma}^{\star}}^{\prime}(0)$ , such that there is a unique $t_{m}\geq t_{1}=t^{\star}$ solving $h(t_{m})=0$ , and for this $t_{m}$ we have both

\widehat{\theta}^{\textup{mv}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}t_{m}u^{\star}~{}~{}\mbox{and}~{}~{}\lim_{m\to\infty}\frac{t_{m}}{t^{\star}\sqrt{m}}=a.

Moreover, the covariance (13) has the form $C_{m}(t)=\frac{b}{(t\sqrt{m})^{2-\beta}}(1+o_{m}(1))$ , and

\displaystyle\sqrt{n}\left(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star}\right)

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,C_{m}(t^{\star})\left({\mathsf{P}_{u^{\star}}\Sigma\mathsf{P}_{u^{\star}}}\right)^{\dagger}\right).

Proposition 3 highlights the robustness of the majority vote estimator: even when the link $\sigma$ is (more or less) arbitrarily incorrect, the asymptotic covariance still exhibits reasonable scaling. The noise parameter $\beta$ in Assumption A2, roughly equivalent to the Mammen-Tsybakov noise exponent ( $\textsc{M}_{\beta}$ ), also plays an important role. In typical cases with $\beta=1$ (e.g., when $X\sim\mathsf{N}(0,I_{d})$ ), we see $C_{m}(t)\asymp\frac{1}{t\sqrt{m}}$ . In noisier cases, corresponding to $\beta\downarrow 0$ , majority vote provides substantial benefit approaching a well-specified model; conversely, in “easy” cases where $\beta>2$ , majority vote estimators become more unstable, as they make the data (very nearly) separable, which causes logistic-regression and other margin-type estimators to be unstable [6].

4 Fundamental estimation limits of majority vote labels

We complement our convergence guarantees via lower bounds for labelers with identical logistic links $\sigma_{1}^{\star}=\dots=\sigma_{m}^{\star}=\sigma^{\textup{lr}}$ . We begin with a somewhat trivial observation that unless we know the number of labels $m$ and link ahead of time, consistently estimating $\theta^{\star}$ from aggregated labels is impossible, which contrasts with algorithms using non-aggregated data. Section 4.2 presents the more precise local asymptotic efficiency limits for estimation from majority labels $Y^{+}$ .

4.1 Inconsistency from aggregate labels

To make clear the impossibility of estimation without some knowledge of the link and number $m$ of labelers, we show that there exist distributions with different linear classifiers $\theta^{\star}$ and $\overline{\theta}$ generating the same data distribution. Consider a generalized linear model for binary classification, with link function $\sigma^{\star}\in\mathcal{F}_{\mathsf{link}}^{0}$ , $\theta^{\star}\in\mathbb{R}^{d}$ , define the model (1) with $\mathbb{P}_{\sigma^{\star},\theta^{\star}}(Y=y\mid X=x)=\sigma^{\star}(y\langle\theta^{\star},x\rangle)$ , and let $X_{1},X_{2},\ldots,X_{n}\stackrel{{\scriptstyle\textup{iid}}}{{\sim}}\mathsf{N}(0,I_{d})$ , where for each data point $X_{i}$ we generate $m$ labels $Y_{ij}\stackrel{{\scriptstyle\textup{iid}}}{{\sim}}\mathbb{P}_{\sigma^{\star},\theta^{\star}}(\cdot\mid X_{i})$ , $j=1,\ldots,m$ . As usual, let $Y^{+}_{i}=\mathsf{maj}(Y_{i1},\ldots,Y_{im})$ denote the majority-vote (breaking ties randomly). Then Proposition 4 shows there always exists a calibration function $\overline{\sigma}$ and label size $\overline{m}$ such that $Y^{+}_{i}$ has identical distribution under both the model (1) and also with $\overline{\sigma},\overline{m}$ replacing $\sigma^{\star}$ and $m$ . Letting $\mathbb{P}_{(X,Y^{+})}^{\sigma^{\star},\theta^{\star},m}$ denote the induced distribution on $(X_{i},Y^{+}_{i})$ , we have the following formal statement, which we prove in Appendix F.1.

Proposition 4.

Suppose $\sigma^{\star}\in\mathcal{F}_{\mathsf{link}}^{0}$ satisfies $\sigma^{\star}(t)>0$ for all $t>0$ . For any $\overline{\theta}\in\mathbb{R}^{d}$ such that $\overline{\theta}/\left\|{\overline{\theta}}\right\|_{2}=\theta^{\star}/\left\|{\theta^{\star}}\right\|_{2}$ and any positive integer $\overline{m}$ , there is another link function $\overline{\sigma}$ such that

\displaystyle\mathbb{P}_{(X,Y^{+})}^{\sigma^{\star},\theta^{\star},m}\stackrel{{\scriptstyle\textup{dist}}}{{=}}\mathbb{P}_{(X,Y^{+})}^{\overline{\sigma},\overline{\theta},\overline{m}}.

4.2 Fundamental asymptotic limits in the logistic case

As most of our results repose on classical asymptotics, we provide lower bounds in the same setting, using Hajek-Le Cam local asymptotic (minimax) theory. We recall a typical result, which combines Le Cam and Yang [23, Lemma 6.6.5] and van der Vaart and Wellner [44, Theorem 3.11.5]:

Corollary 5.

Let $\{\mathbb{P}_{\theta}\}$ be a family of distributions indexed by $\theta$ with continuous Fisher Information $\mathsf{I}(\theta)$ in a neighborhood of $\theta^{\star}$ . Then there exist probability densities measures $\pi_{c,n}$ supported on $\{\theta\in\mathbb{R}^{d}\mid\left\|{\theta^{\star}-\theta}\right\|_{2}\leq c/\sqrt{n}\}$ such that for any quasi-convex, symmetric, and bounded loss $\varphi$ ,

\liminf_{c\to\infty}\liminf_{n}\inf_{\widehat{\theta}_{n}}\int\mathbb{E}_{\theta}[\varphi(\sqrt{n}(\widehat{\theta}_{n}-\theta))]\pi_{c,n}(\theta)d\theta\geq\mathbb{E}[\varphi(W)],

where $W\sim\mathsf{N}(0,\mathsf{I}(\theta^{\star})^{-1})$ . If $T:\mathbb{R}^{d}\to\mathbb{R}^{k}$ is differentiable at $\theta^{\star}$ with derivative matrix $\dot{T}(\theta^{\star})\in\mathbb{R}^{k\times d}$ , then additionally

\liminf_{c\to\infty}\liminf_{n}\inf_{\widehat{T}_{n}}\int\mathbb{E}_{\theta}[\varphi(\sqrt{n}(\widehat{T}_{n}-T(\theta)))]\pi_{c,n}(\theta)d\theta\geq\mathbb{E}[\varphi(\dot{T}(\theta^{\star})W)].

The infima above are taken over all estimators $\widehat{\theta}_{n}$ and $\widehat{T}_{n}$ based on $n$ i.i.d. observations from $\mathbb{P}_{\theta}$ .

This corollary is the sense in which estimators, such as the MLE, achieving asymptotic variance equal to the inverse Fisher Information are efficient. It also makes clear that (except perhaps at a Lebesgue measure-zero set of parameters $\theta$ ) any estimator $\widehat{\theta}_{n}$ with asymptotic variance $\Sigma_{\theta}$ necessarily satisfies $\Sigma_{\theta}\succeq\mathsf{I}(\theta)^{-1}$ .

We therefore complete our theoretical analysis of fundamental limits by giving the Fisher Information matrix for binary classification models with aggregated majority vote data. We assume the links are identical, $\sigma_{1}^{\star}=\dots=\sigma_{m}^{\star}=\sigma^{\textup{lr}}$ . Recall (see Prop. 2) that in the case where $X$ is normal, the majority vote estimator $\widehat{u}^{\textup{mv}}_{n,m}$ has asymptotic variance of order $m^{-\frac{1}{2}}$ . As we will present momentarily in Theorem 3, $\widehat{u}^{\textup{mv}}_{n,m}$ is indeed rate optimal for classification as $m\to\infty$ .

As usual, letting Assumption A1 hold, we write $X=Zu^{\star}+W$ with $W\perp u^{\star}$ . Let $t^{\star}=\|{\theta^{\star}}\|_{2}$ . In this case, we may explicitly decompose the Fisher information.

Lemma 4.1.

The Fisher information matrix $\mathsf{I}^{\textup{mv}}_{m}(\theta)$ for the majority vote model $\{\mathbb{P}_{(X,Y^{+})}^{\sigma^{\textup{lr}},\theta,m}\}_{\theta\in\mathbb{R}^{d}}$ satisfies

\displaystyle\mathsf{I}^{\textup{mv}}_{m}(\theta^{\star})=\mathbb{E}\left[{\frac{\rho_{m}^{\prime}(t^{\star}|Z|)^{2}Z^{2}}{\rho_{m}(t^{\star}|Z|)(1-\rho_{m}(t^{\star}|Z|))}}\right]\cdot u^{\star}{u^{\star}}^{\top}+\mathbb{E}\left[{\frac{\rho_{m}^{\prime}(t^{\star}|Z|)^{2}}{\rho_{m}(t^{\star}|Z|)(1-\rho_{m}(t^{\star}|Z|))}}\right]\cdot\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.

Proof.

For $p_{\theta}(x,y)=\mathbb{P}_{\theta}(Y^{+}=y\mid X=x)q(x)$ , where $q$ is the $d$ -dimensional centered density of $X$ , the Fisher information matrix at $\theta$ is

\displaystyle\mathsf{I}^{\textup{mv}}_{m}(\theta)=\mathbb{E}\left[{\nabla_{\theta}\log p_{\theta}(X,Y^{+})\cdot\nabla_{\theta}\log p_{\theta}(X,Y^{+})^{\top}}\right].

Substituting $\rho_{m}(|\langle x,\theta^{\star}\rangle|)=\mathbb{P}(Y^{+}=\mathsf{sign}(\langle x,\theta^{\star}\rangle)\mid X=x)$ , we then have

	$\displaystyle\mathbb{E}\left[{\nabla_{\theta}\log p_{\theta^{\star}}(X,Y^{+})\cdot\nabla_{\theta}\log p_{\theta^{\star}}(X,Y^{+})^{\top}\mid X=x}\right]$
	$\displaystyle=\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\frac{\nabla_{\theta}\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\nabla_{\theta}\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{\top}}{\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{2}}$
	$\displaystyle\qquad+(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\frac{\left({\nabla_{\theta}(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)}\right)\cdot\left({\nabla_{\theta}(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)}\right)^{\top}}{(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{2}}$
	$\displaystyle=\frac{\rho_{m}^{\prime}(\|\langle x,\theta^{\star}\rangle\|)^{2}}{\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|))}\cdot xx^{\top}$

as desired. ∎

By characterizing the large $m$ behavior of the Fisher information, we can then apply Corollary 5 to understand the optimal efficiency of any estimator given aggregate (majority vote) data.

Theorem 3.

Let Assumption A1 hold, Assumption A2 hold with constants $c_{Z}$ and $\beta$ , and $t^{\star}=\|{\theta^{\star}}\|_{2}$ . Then defining the constants $a=\frac{c_{Z}}{\pi}\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta+1}}{\Phi(-z)\Phi(z)}dz$ and $b=\frac{c_{Z}}{4\pi}\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta-1}}{\Phi(-z)\Phi(z)}dz$ , there are functions

\displaystyle A_{m}(t)=\frac{a}{t^{2}}\left(\frac{1}{t\sqrt{m}}\right)^{\beta}(1+o_{m}(1))~{}~{}\mbox{and}~{}~{}B_{m}(t)=\frac{b}{t^{2}}\left(\frac{1}{t\sqrt{m}}\right)^{\beta-2}(1+o_{m}(1))

for which

\mathsf{I}^{\textup{mv}}_{m}(\theta^{\star})=A_{m}(t^{\star})\cdot u^{\star}{u^{\star}}^{\top}+B_{m}(t^{\star})\cdot\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.

See Appendix F.2 for the proof.

By combining Theorem 3 with the classical local asymptotic minimax bounds, we can obtain optimality for estimators of both $\theta^{\star}$ and the direction $u^{\star}$ as $m$ gets large. Computing the inverse Fisher information for $\theta^{\star}$ and for the normalized quantity $u=\phi(\theta)\coloneqq\theta/\left\|{\theta}\right\|_{2}$ , which satisfies $\dot{\phi}(\theta)=\left\|{\theta}\right\|_{2}^{-1}(I_{d}-\phi(\theta)\phi(\theta)^{\top})$ , we thus have the efficient limiting covariances

	$\displaystyle\Sigma_{\theta^{\star}}$	$\displaystyle\coloneqq\mathsf{I}^{\textup{mv}}_{m}(\theta^{\star})^{-1}\asymp{t^{\star}}^{\beta+2}m^{\frac{\beta}{2}}\cdot u^{\star}{u^{\star}}^{\top}+{t^{\star}}^{\beta}m^{\frac{\beta-2}{2}}\cdot\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger},$
	$\displaystyle\Sigma_{u^{\star}}$	$\displaystyle\coloneqq\dot{\phi}(\theta^{\star})\mathsf{I}^{\textup{mv}}_{m}(\theta^{\star})^{-1}\dot{\phi}(\theta^{\star})\asymp{t^{\star}}^{\beta-2}m^{\frac{\beta-2}{2}}\cdot\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger},$

where we recall the shorthand $t^{\star}=\left\|{\theta^{\star}}\right\|_{2}$ . (Here we use that if $A,B$ are symmetric matrices with $AB=BA=0$ , then $(A+B)^{-1}=A^{\dagger}+B^{\dagger}$ ; see Lemma A.1 in Appendix A.) Then combining Theorem 3 with Corollary 5 yields the following efficiency lower bound. (Other lower bounds arising from, e.g., the convolution and variants of the local asymptotic minimax theorem are possible; we present only this to give the flavor of such results.)

Corollary 6.

Let $\widehat{\theta}_{n}$ and $\widehat{u}_{n}$ be arbitrary estimators of $\theta^{\star}$ and $u^{\star}=\phi(\theta^{\star})$ using the aggregated data $\{(X_{i},Y^{+}_{i})\}_{i=1}^{n}$ . For any bounded and symmetric quasi-convex loss $\varphi:\mathbb{R}^{d}\to\mathbb{R}$ ,

\lim_{c\to\infty}\liminf_{n}\sup_{\left\|{\theta-\theta^{\star}}\right\|_{2}\leq c/\sqrt{n}}\mathbb{E}_{\theta}[\varphi(\sqrt{n}(\widehat{u}_{n}-\phi(\theta)))]\geq\mathbb{E}[\varphi(W)],~{}~{}\mbox{where}~{}~{}W\sim\mathsf{N}(0,\Sigma_{u^{\star}}).

If there exist random variables $T_{\theta}$ or $U_{\theta}$ such that $\sqrt{n}(\widehat{\theta}_{n}-\theta^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}T_{\theta^{\star}}$ (respectively, $\sqrt{n}(\widehat{u}_{n}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}U_{\theta^{\star}}$ ), then $\mathsf{Cov}(T_{\theta})\succeq\Sigma_{\theta^{\star}}$ and $\mathsf{Cov}(U_{\theta})\succeq\Sigma_{u^{\star}}$ for Lebesgue almost every $\theta^{\star}\in\mathbb{R}^{d}$ .

This result carries several implications. First, the scaling in both $t^{\star}=\left\|{\theta^{\star}}\right\|_{2}$ and $m$ for the asymptotic (achieved) covariance in Proposition 2 are rate-optimal: we have $C_{m}(t)\asymp t^{\beta-2}m^{\frac{\beta-2}{2}}$ for the majority vote estimator $\widehat{\theta}^{\textup{mv}}_{n,m}$ . Notably, this optimal scaling holds even though the model may be mis-specified: estimators using the aggregated majority vote labels $Y^{+}$ are rate-optimal in the scale $\left\|{\theta^{\star}}\right\|_{2}$ and the number of labelers $m$ (and sample size $n$ ) no matter the margin-based loss. (Achieving the optimal constant requires a well-specified loss, of course.) This robustness of estimators using aggregated data may motivate the use of strong data-cleaning measures in dataset construction when it is difficult to model the particulars of the data generation process.

A perhaps less intuitive result is that for any noise scale $\beta>0$ on the margin variable $Z=\langle X,u^{\star}\rangle$ arising from Assumption A2, the inverse information $\Sigma_{\theta^{\star}}\to\infty$ as $m\to\infty$ , meaning calibration becomes fundamentally harder for estimators using only aggregated labels, even with a (known) logistic link $\sigma^{\textup{lr}}$ . For example, in the Gaussian case when $\beta=1$ , then $\Sigma_{\theta^{\star}}\asymp O(m^{\frac{1}{2}})$ and $\Sigma_{u^{\star}}\asymp O(m^{-\frac{1}{2}})$ , matching the scaling in Corollary 2. The same phenomenon holds even in the “very low noise” regime where $\beta>2$ , meaning that the margin variable $Z$ is typically far from 0 (recall the exponent ( $\textsc{M}_{\beta}$ )). In this case, $\Sigma_{u^{\star}}\to\infty$ as $m\to\infty$ , as a paucity of data align well with the true margin $u^{\star}$ ; estimators (nearly) perfectly predict on training data and become non-robust.

5 Semi-parametric approaches

The preceding analysis highlights distinctions between a fuller likelihood-based approach—which uses all the labels, as in (2)—and the robust but somewhat slower rates that majority vote estimators enjoy (as in Proposition 3). That full-label estimators’ performance so strongly depends on the fidelity of the link (recall Corollary 4) suggests that we target estimators achieving the best of both worlds: learn both a link function (or collection thereof) and refit the model using all the labels. While we mainly focus on efficiency bounds for procedures we view as closer to practice—fitting a model based on available data—in this section, we provide a few results targeting more efficient estimation schemes through semiparametric estimation approaches.

As developing full semi-parametric efficiency bounds would unnecessarily lengthen and dilute the paper, we develop a few relatively general and simple convergence results into which we can essentially plug in semiparametric estimators. In distinction from standard results in semiparametric theory (e.g. [43, Ch. 25] or [4]), our results require little more than consistent estimation of the links $\sigma_{j}^{\star}$ to recover $1/m$ (optimal) scaling in the asymptotic covariance, as the special structure of our classification problem allows more nuanced calculations; we assume each labeler (link function) generates labels for each of the $n$ datapoints $X_{1},\ldots,X_{n}$ , but we could relax the assumption at the expense of extraordinarily cumbersome notation. We give two example applications of the general theory: the first (Sec. 5.2) analyzing a full pipeline for a single index model setting, which robustly estimates direction $u^{\star}$ , the link $\sigma^{\star}$ , and then re-estimates $\theta^{\star}$ ; the second assuming a stylized black-box crowdsourcing mechanism that provides estimates of labeler reliability, highlighting how even in some crowdsourcing scenarios, there could be substantial advantages to using full label information.

5.1 Master results

For our specializations, we first provide master results that allow semi-parametric estimation of the link functions. We consider Lipschitz symmetric link functions, where for $\mathsf{L}>0$ we define

\displaystyle\mathcal{F}_{\mathsf{link}}^{\mathsf{L}}:=\{\sigma\mid\sigma\not\equiv 1/2\text{ is non-decreasing, symmetric, and}~{}\mathsf{L}\text{-Lipschitz continuous}\}\subset\mathcal{F}_{\mathsf{link}}^{0}.

We consider the general case where there are $m$ distinct labeler link functions $\sigma_{1}^{\star},\ldots,\sigma_{m}^{\star}$ . To eliminate ambiguity in the links, we assume the model is normalized, $\left\|{\theta^{\star}}\right\|_{2}=1$ so $\theta^{\star}=u^{\star}$ . To distinguish from the typical case, we write $\vec{\sigma}=(\sigma_{1},\ldots,\sigma_{m})$ , and for $(x,y)\in\mathbb{R}^{d}\times\{\pm 1\}^{m}$ define

\ell_{\vec{\sigma},\theta}(y\mid x)\coloneqq\frac{1}{m}\sum_{j=1}^{m}\ell_{\sigma_{j},\theta}(y_{j}\mid x),

which allows us to consider both the standard margin-based loss and the case in which we learn separate measures of quality per labeler. With this notation, we can then naturally define the population loss

L(\theta,\vec{\sigma})\coloneqq\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[\ell_{\sigma_{j},\theta}(Y_{j}\mid X)].

For any sequence $\{\vec{\sigma}_{n}\}\subset(\mathcal{F}_{\mathsf{link}}^{\mathsf{L}})^{m}$ of (estimated) links and data $(X_{i},Y_{i})$ for $Y_{i}=(Y_{i1},\ldots,Y_{im})$ , we define the semi-parametric estimator

\displaystyle\widehat{\theta}^{\textup{sp}}_{n,m}\coloneqq\operatorname*{argmin}_{\theta\in\mathbb{R}^{d}}\frac{1}{n}\sum_{i=1}^{n}\ell_{\vec{\sigma}_{n},\theta}(Y_{i}\mid X_{i}).

We will demonstrate both consistency and asymptotic normality under appropriate convergence and regularity assumptions for the link functions. We assume there is a (semiparametric) collection $\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}\subset(\mathcal{F}_{\mathsf{link}}^{\mathsf{L}})^{m}$ of link functions of interest, which may coincide with $(\mathcal{F}_{\mathsf{link}}^{\mathsf{L}})^{m}$ but may be smaller, making estimation easier. Define the distance $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}$ on $\mathbb{R}^{d}\times\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}$ by

\displaystyle d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}\left((\theta_{1},\vec{\sigma}_{1}),(\theta_{2},\vec{\sigma}_{2})\right)\coloneqq\left\|{\theta_{1}-\theta_{2}}\right\|_{2}+\left\|{\vec{\sigma}_{1}(-Y\langle X,u^{\star}\rangle)-\vec{\sigma}_{2}(-Y\langle X,u^{\star}\rangle)}\right\|_{L^{2}(\mathbb{P})}.

We make the following assumption.

Assumption A5.

The links $\vec{\sigma}^{\star}\in\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}$ are normalized so that $\mathbb{P}(Y_{j}=y\mid X=x)=\sigma^{\star}_{j}(y\langle x,u^{\star}\rangle)$ , and the sequence $\{\vec{\sigma}_{n}\}\subset\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}$ is consistent:

\left\|{\vec{\sigma}_{n}(-Y\langle X,u^{\star}\rangle)-\vec{\sigma}^{\star}(-Y\langle X,u^{\star}\rangle)}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0.

Additionally, the mapping $(\theta,\vec{\sigma})\mapsto\nabla_{\theta}^{2}L(\theta,\vec{\sigma})$ is continuous for $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}$ at $(u^{\star},\vec{\sigma}^{\star})$ .

The continuity of $\nabla_{\theta}^{2}L(\theta,\vec{\sigma})$ at $(u^{\star},\vec{\sigma}^{\star})$ allows us to develop local asymptotic normality. To see that we may expect the assumption to hold, we give reasonably simple conditions sufficient for it, including that the collection of links $\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}$ is sufficiently smooth or the data distribution is continuous enough. (See Appendix H.1 for a proof.)

Lemma 5.1.

Let $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}$ be the distance in Assumption A5. Let Assumption A1 hold, where $|Z|>0$ with probability one, has nonzero and continuously differentiable density $p(z)$ on $(0,\infty)$ satisfying $\lim_{z\to s}z^{2}p(z)=0$ for $s\in\{0,\infty\}$ . The mapping $(\theta,\vec{\sigma})\mapsto\nabla_{\theta}^{2}L(\theta,\vec{\sigma})$ is continuous for $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}$ at $(u^{\star},\vec{\sigma}^{\star})$ whenever $\mathbb{E}[\left\|{X}\right\|_{2}^{4}]<\infty$ and either of the following conditions holds:

(1)

For any $\vec{\sigma}=(\sigma_{1},\ldots,\sigma_{m})\in\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}$ , $\sigma_{j}^{\prime}$ are Lipschitz continuous.
(2)

$X$ has continuous density on $\mathbb{R}^{d}$ .

We can now present the master result for semi-parametric approaches, which characterizes the asymptotic behavior of the semi-parametric estimator with the variance function

\displaystyle C_{m,\vec{\sigma}^{\star}}:=\frac{1}{m}\cdot\frac{\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[\sigma_{j}^{\star}(Z)(1-\sigma_{j}^{\star}(Z))]}{(\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[{\sigma_{j}^{\star}}^{\prime}(Z)])^{2}}.

Theorem 4.

Let Assumption A1 hold and assume $|Z|>0$ with probability one and has nonzero and continuous density $p(z)$ on $(0,\infty)$ . Let Assumption A5 hold and assume that $\mathbb{E}[\left\|{X}\right\|_{2}^{4}]<\infty$ . Then $\sqrt{n}(\widehat{\theta}^{\textup{sp}}_{n,m}-u^{\star})$ is asymptotically normal, and the normalized estimator $\widehat{u}^{\textup{sp}}_{n,m}=\widehat{\theta}^{\textup{sp}}_{n,m}/\|{\widehat{\theta}^{\textup{sp}}_{n,m}}\|_{2}$ satisfies

\sqrt{n}\left({\widehat{u}^{\textup{sp}}_{n,m}-u^{\star}}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left({0,C_{m,\vec{\sigma}^{\star}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}}\right).

See Appendix G for proof details. Notably, Theorem 4 exhibits optimal $1/m$ scaling in the covariance whenever $\mathbb{E}[{\sigma^{\star}_{j}}^{\prime}(Z)]\gtrsim 1$ .

5.2 A single index model

Our first example application of Theorem 4 is to a single index model. We present a multi-phase estimator that first estimates the direction $u^{\star}=\theta^{\star}/\|{\theta^{\star}}\|_{2}$ , then uses this estimate to find a (consistent) estimate of the link $\sigma^{\star}$ , which we can then substitute directly into Theorem 4. We defer all proofs of this section to Appendix I, which also includes a few auxiliary results that we use to prove the results proper.

We present the the abstract convergence result for link functions first, considering a scenario where we have an initial guess $u_{n}^{\textup{init}}$ of the direction $u^{\star}$ , independent of $(X_{i},Y_{ij})_{i\leq n,j\leq m}$ , for example constructed via a small held-out subset of the data. We set

\vec{\sigma}_{n}=\operatorname*{argmin}_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}\sum_{i=1}^{n}\sum_{j=1}^{m}\left(\sigma_{j}(\langle u_{n}^{\textup{init}},X_{i}\rangle)-Y_{ij}\right)^{2},

where $\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}\subset(\mathcal{F}_{\mathsf{link}}^{\mathsf{L}})^{m}$ and so it consists of nondecreasing $\mathsf{L}$ -Lipschitz link functions with $\sigma(0)=\frac{1}{2}$ . We assume that for all $n$ , there exists a (potentially random) $\epsilon_{n}$ such that

\|{u_{n}^{\textup{init}}-u^{\star}}\|_{2}\leq\epsilon_{n}.

Proposition 5.

Let $X_{i}$ be vectors with $\mathbb{E}[\left\|{X}\right\|_{2}^{k}]<\infty$ , where $k\geq 2$ . Then with probability $1$ , there is a finite (random) $C<\infty$ such that for all large enough $n$ ,

\left\|{\vec{\sigma}_{n}(Y\langle u^{\star},X\rangle)-\vec{\sigma}^{\star}(Y\langle u^{\star},X\rangle)}\right\|_{L^{2}(\mathbb{P})}^{2}\leq C\left[n^{\frac{2}{3k}-\frac{2}{3}}+\epsilon_{n}^{2}+\mathsf{L}n^{-\frac{k}{2(k+1)}}\right].

The proof is more or less a consequence of standard convergence results for nonparametric function estimation, though we include it for completeness in Appendix I.2 as it includes a few additional technicalities because of the initial estimate of $u^{\star}$ .

Summarizing, we see that a natural procedure is available: if we have models powerful enough to accurately estimate the conditional label probabilities $Y\mid X$ , then Proposition 5 coupled with Theorem 4 shows that we can achieve estimation with near-optimal asymptotic covariance. In particular, if $u_{n}^{\textup{init}}$ is consistent (so $\epsilon_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ ), then $\widehat{\theta}^{\textup{sp}}_{n,m}$ induces a normalized estimator $\widehat{u}^{\textup{sp}}_{n}=\widehat{\theta}^{\textup{sp}}_{n,m}/\|{\widehat{\theta}^{\textup{sp}}_{n,m}}\|_{2}$ satisfying $\sqrt{n}(\widehat{u}^{\textup{sp}}_{n}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}(0,C_{m,\vec{\sigma}}({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}})^{\dagger})$ .

5.3 Crowdsourcing model

Crowdsourcing typically targets estimating rater reliability, then using these reliability estimates to recover ground truth labels as accurately as possible, with versions of this approach central since at least Dawid and Skene’s Expectation-Maximization-based approaches [10, 48, 35]. We focus here on a simple model of rater reliability, highlighting how—at least in our stylized model of classifier learning—by combining a crowdsourcing reliability model and still using all labels in estimating a classifier, we can achieve asymptotically efficient estimates of $\theta^{\star}$ , rather than the robust but slower estimates $\widehat{\theta}_{n,m}^{\textup{mv}}$ arising from “cleaned” labels.

We adopt Whitehill et al.’s roughly “low-rank” model for label generation [48]: for binary classification with $m$ labelers and distinct link functions $\sigma_{i}^{\star}$ , model the difficulty of $X_{i}$ by $\beta_{i}\in(-\infty,\infty)$ , where $\mathsf{sign}(\beta_{i})$ denotes the true class $X_{i}$ belongs to. A parameter $\alpha_{j}$ models the expertise of annotator $j$ , and the probability labeler $j$ correctly classifies $X_{i}$ is

\displaystyle\mathbb{P}(Y_{ij}=1)=\frac{1}{1+\exp(-\alpha_{i}\beta_{j})}.

(See also Raykar et al. [35].) The focus in these papers was to construct gold-standard labels and datasets $(X_{i},Y_{i})$ ; here, we take the alternative perspective we have so far advocated to show how using all labels can yield strong performance.

We thus adopt a semiparametric approach: we model the labelers, assuming a black-box crowdsourcing model that can infer each labeler’s ability, then fit the classifier. We represent labeler $j$ ’s expertise by a scalar $\alpha_{j}^{\star}\in(0,\infty)$ . Given data $X_{i}=X$ and the normalized $\theta^{\star}=u^{\star}$ , we assume a modified logistic link

\displaystyle\mathbb{P}(Y_{ij}=1\mid X_{i}=x)=\frac{1}{1+\exp(-\alpha^{\star}_{j}\langle\theta^{\star},x\rangle)}=\sigma^{\textup{lr}}(\alpha_{j}^{\star}\langle u^{\star},x\rangle),

so $\alpha_{j}^{\star}=\infty$ represents an omniscient labeler while $\alpha_{j}^{\star}=0$ means the labeler chooses random labels regardless of the data. Let $\alpha^{\star}=(\alpha_{1}^{\star},\ldots,\alpha_{m}^{\star})\in\mathbb{R}^{m}_{+}$ . Then, in keeping with the plug-in approach of the master Theorem 4, we assume the blackbox crowdsourcing model generates an estimate $\alpha_{n}=(\alpha_{n,1},\ldots,\alpha_{n,m})\in\mathbb{R}_{+}^{m}$ of $\alpha^{\star}$ from the data $\{(X_{i},(Y_{i1},\ldots,Y_{im}))\}_{i=1}^{n}$ .

We consider the algorithm using the blackbox crowdsourcing model and empirical risk minimization with the margin-based loss $\ell_{\vec{\sigma}_{n},\theta}$ as in Section 5.1, with $\vec{\sigma}_{n}(t)\coloneqq(\sigma^{\textup{lr}}(\alpha_{n,1}t),\dots,\sigma^{\textup{lr}}(\alpha_{n,m}t))$ , or equivalently using the rescaled logistic loss

\displaystyle\ell_{\vec{\sigma}_{n},\theta}(y\mid x)=\frac{1}{m}\sum_{j=1}^{m}\ell^{\textup{lr}}_{\theta}(y_{j}\mid\alpha_{n,j}x).

This allows us to apply our general semiparametric Theorem 4 as long as the crowdsourcing model produces a consistent estimate $\alpha_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}\alpha^{\star}$ (see Appendix H.2 for a proof):

Proposition 6.

Let Assumption A1 hold, $|Z|>0$ have nonzero and continuous density $p(z)$ on $(0,\infty)$ , and $\mathbb{E}[\left\|{X}\right\|_{2}^{4}]<\infty$ . If $\alpha_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}\alpha^{\star}\in\mathbb{R}^{m}$ , then $\sqrt{n}(\widehat{\theta}^{\textup{sp}}_{n,m}-u^{\star})$ is asymptotically normal, and the normalized estimator $\widehat{u}^{\textup{sp}}_{n,m}=\widehat{\theta}^{\textup{sp}}_{n,m}/\|{\widehat{\theta}^{\textup{sp}}_{n,m}}\|_{2}$ satisfies

\sqrt{n}\left({\widehat{u}^{\textup{sp}}_{n,m}-u^{\star}}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left({0,\frac{1}{\sum_{j=1}^{m}\mathbb{E}[\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z)(1-\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z))]}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}}\right).

By Proposition 6, the semiparametric estimator $\widehat{u}^{\textup{sp}}_{n,m}$ is efficient when the rater reliability estimates $\alpha_{n}\in\mathbb{R}^{m}$ are consistent. It is also immediate that if $\alpha_{j}^{\star}\leq\alpha_{\max}<\infty$ are bounded, then the asymptotic covariance multiplier $C_{m,\alpha^{\star}}=(\sum_{j=1}^{m}\mathbb{E}[\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z)(1-\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z))])^{-1}=O(1/m)$ , so we recover the $1/m$ scaling of the MLE, as opposed to the slower rates of the majority vote estimators in Section 3.2. At this point, the refrain is perhaps unsurprising: using all the label information can yield much stronger convergence guarantees.

6 Experiments

We conclude the paper with several experiments to test whether the (sometimes implicit) methodological suggestions hold merit. Before delving into our experimental results, we detail a few of the expected behaviors our theory suggests; if we fail to see them, then the model we have proposed is too unrealistic to inform practice. First, based on the results in Section 2, we expect the classification error to be better for the non-aggregated algorithm $\widehat{\theta}^{\textup{lr}}_{n,m}$ , and the gap between the two algorithms to become larger for less noisy problems. Moreover, we only expect $\widehat{\theta}^{\textup{lr}}_{n,m}$ to be calibrated, and our theory predicts that the majority-vote estimator’s calibration worsens as the number of labels $m$ increases. Generally, so long as we model uncertainty with enough fidelity, Corollary 4 suggests that multilabel estimators should exhibit better performance than those using majority vote labels $Y^{+}$ .

To that end, we provide experiments on two real datasets and one semi-synthetic dataset: the BlueBirds dataset (Section 6.1), CIFAR-10H (Section 6.2), and a modified version of the CIFAR-10 dataset (Section 6.3). We consider our two main algorithmic models:

(i)

The maximum likelihood estimator $\widehat{\theta}^{\textup{lr}}_{n,m}$ based on non-aggregated data in Eq. (2).
(ii)

The majority-vote based estimator $\widehat{\theta}^{\textup{mv}}_{n,m}$ of Eq. (3), our proxy for modern data pipelines.

Unfortunately, a paucity of large-scale multi-label datasets that we know of precludes experiments on fully real datasets; the ImageNet creators [11, 37] no longer have any intermediate label information from their construction. To simulate the larger scale data collection process, we create a large scale semisynthetic dataset (Section 6.3) using the raw CIFAR-10 data and using trained neural networks as labelers, giving us a semi-synthetic dataset with multiple labels. As the ground truth model is well-specified and we know the link function, beyond the aforementioned estimators with full information and majority vote, it is also of practical interest to see how the aggregated labels from other crowdsourcing methods compare to majority vote, and to knowing all individual labels.

6.1 BlueBirds

We begin with the BlueBirds dataset [47], which is a relatively small dataset consisting of 108 images with ResNet features. The classification problem is challenging, and the task is to classify each image as one of Indigo Bunting or Blue Grosbeak (two similar-looking blue bird species). For each image, we have $39$ labels, obtained through Amazon Mechanical Turk workers. We use a pretrained (on ImageNet) ResNet50 model to generate image features, then apply PCA to reduce the dimensionality from $d_{\textup{init}}=2048$ to $d=25$ .

We repeat the following experiment $T=100$ times. For each number $m=1,\ldots,35$ of labelers, we fit the multilabel logistic model (2) and the majority vote estimator (3), finding calibration and classification errors using 10-fold cross validation. We measure calibration error on a held-out example $x$ by $|\mathop{\rm logit}(\widetilde{p}(x))-\mathop{\rm logit}(\widehat{p}(x))|$ , where $\widetilde{p}(x)$ is the predicted probability and $\widehat{p}(x)$ is the empirical probability (over the labelers), where $\mathop{\rm logit}(p)=\log\frac{p}{1-p}$ ; we measure classification error on example $x$ with labels $(y_{1},\ldots,y_{m})$ by $\frac{1}{m}\sum_{j=1}^{m}\mathds{1}\{y_{j}\neq\mathsf{sign}(\widetilde{p}(x)-\frac{1}{2})\}$ , giving an inherent noise floor because of labeler uncertainty. We report the results in Figure 1. These plots corroborate our theoretical predictions: as the number of labelers $m$ increases, both the majority vote method and the full label method exhibit improved classification error, but considering all labels gives a (significant) improvement in accuracy and in calibration error.

\begin{overpic}[width=208.13574pt]{figure/resnet-class-bluebirds.pdf} \put(30.0,-1.0){{\small Number of labelers $m$}} \put(-3.0,20.0){ \rotatebox{90.0}{{\small Classification error}}} \end{overpic}	\begin{overpic}[width=208.13574pt]{figure/resnet-calib-bluebirds.pdf} \put(30.0,-1.0){{\small Number of labelers $m$}} \put(-3.0,20.0){ \rotatebox{90.0}{{\small Calibration error}}} \end{overpic}
(a)	(b)

Figure 1: Experiments on BlueBirds dataset. (a) Classification error. (b) Calibration error

|\mathop{\rm logit}(\widetilde{p})-\mathop{\rm logit}(p)|

with ResNet features reduced via PCA to dimension

d=25

. Error bars show 2 standard error confidence bands over

T=100

trials.

6.2 CIFAR-10H

\begin{overpic}[width=208.13574pt]{figure/resnet-class-new.pdf} \put(30.0,-1.0){{\small Number of labelers $m$}} \put(-3.0,20.0){ \rotatebox{90.0}{{\small Classification error}}} \end{overpic}	\begin{overpic}[width=208.13574pt]{figure/resnet-calib-new.pdf} \put(30.0,-1.0){{\small Number of labelers $m$}} \put(-3.0,20.0){ \rotatebox{90.0}{{\small Calibration error}}} \end{overpic}
(a)	(b)

Figure 2: Experiments on CIFAR-10H dataset. (a) Classification error. (b) Calibration error

|\mathop{\rm logit}(\widetilde{p})-\mathop{\rm logit}(p)|

with ResNet features reduced via PCA to dimension

d=40

. Error bars show 2 standard error confidence bands over

T=100

trials.

For our second experiment, we consider Peterson et al.’s CIFAR-10H dataset [29], which consists of $10,\!000$ images from CIFAR-10 test set with soft labeling in that for each image, we have approximately $50$ labels from different annotators. Each $32\times 32$ image in the dataset belongs to one of the ten classes airplane, automobile, bird, cat, dog, frog, horse, ship, or truck; labelers assign each image to one of the classes. To maintain some fidelity to the binary classification setting we analyze throughout the paper, we transform the problem into a set of 10 binary classification problems. For each class $c$ , we take each initial image/label pair $(x,y)\in\mathbb{R}^{32\times 32}\times\{1,\ldots,10\}$ , assigning binary label $1$ if $y=c$ and $0$ otherwise (so the annotator labels it as an alternative class $y\neq c$ ). Most of the images in the dataset are very easy to classify: more than 80% have a unanimous label from each of the $m=50$ labelers, meaning that the MLE and majority vote estimators (2) and (3) coincide for these. (In experiments with this full dataset, we saw little difference between the two estimators.)

As our theoretical results highlight the importance of classifier difficulty, we therefore balance the dataset by considering subsets of harder images as follows. For each fixed target $c$ (e.g., cat) and for image $i$ , let $\widehat{p}_{i}$ be the empirical probability of the target among the 50 annotator labels. Then for $p\in[\frac{1}{2},1]$ , define the subsets

\displaystyle\mathcal{S}_{p}=\left\{i\in[n]:\max\left\{\widehat{p}_{i},1-\widehat{p}_{i}\right\}\leq p\right\},

so that $p=\frac{1}{2}$ corresponds to images with substantial confusion, and $p=1$ to all images (most of which are easy). We test on $\mathcal{S}_{0.9}$ (labelers have at most 90% agreement), which consists of with $441$ images. For image $i$ , we again generate features $x_{i}\in\mathbb{R}^{d}$ by by taking the last layer of a pretrained ResNet50 neural network $\tilde{x}_{i}\in\mathbb{R}^{d_{\textup{init}}}$ , using PCA to reduce to a $d=40$ -dimensional feature. We follow the same procedure as in Sec. 6.1, subsampling $m=1,2,\ldots,45$ labelers and using 10-fold cross validation to evaluate classification and calibration error. We report the results in Figure 2. Again we see that—as the number of labelers increases—both aggregated and non-aggregated methods evidence improved classification error, but the majority vote procedure (cleaned data) yields less improvement than one with access to all (uncertain) labels. These results are again consistent with our theoretical predictions.

6.3 Crowdsourcing methods on semisynthetic CIFAR-10 labels

In our final set of experiments, we adapt the original CIFAR-10 [22] dataset, which consists of 6000 $32\times 32$ images from each of $k=10$ classes (60,000 total images). To mimic collecting and cleaning data with noisy labelers—rather than the single-label “gold standard” in the base CIFAR-10 data—we construct pseudo-labelers using a ResNet18 network whose accuracy we can directly adjust. This allows us to compare the maximum-likelihood estimator $\widehat{\theta}^{\textup{mle}}_{n,m}$ , the majority vote estimator $\widehat{\theta}^{\textup{mv}}_{n,m}$ , and, to test if the predictions of our stylized models still hold for more advanced aggregation strategies, we include Dawid-Skene [10] and GLAD [48] crowdsouring estimators for the labels.

To generate the semisynthetic labelers and labels, we use a pretrained ResNet18 model [18], an eighteen layer residual network, $f:\mathcal{X}\to\mathbb{R}^{k}$ . For $s\in\mathbb{R}^{k}$ , define the softmax mapping $\textup{softmax}(s)=[e^{s_{y}}/\sum_{l=1}^{k}e^{s_{l}}]_{y=1}^{k}$ . Then for an input $x\in\mathcal{X}$ , the model $f$ outputs a score $f(x)\in\mathbb{R}^{k}$ , where $\textup{softmax}(f(x))$ indicates the probabilities $\mathbb{P}(Y=y\mid X=x)$ . The model $f$ has the form

f(x)={\Theta^{\star}}^{\top}\phi(x),

where $\Theta^{\star}=[\theta^{\star}_{1}~{}\cdots~{}\theta^{\star}_{k}]\in\mathbb{R}^{d\times k}$ is a matrix of per-class weights, and $\phi:\mathcal{X}\to\mathbb{R}^{d}$ represents the second-to-last layer outputs of the neural network. As we fit linear models to the data, we therefore take these outputs as our data, so that for the $i$ th image $x_{i}$ , we have $X_{i}=\phi(x_{i})$ , and $\Theta^{\star}$ is the ground-truth parameter. We construct semi-synthetic labels $Y_{ij}\in\{1,\ldots,k\}$ , $j=1,\ldots,m$ of varying accuracy/labeler expertise as follows. Letting $\alpha>0$ be a chosen constant we vary to model labeler expertise, we draw

\mathbb{P}(Y_{ij}=y\mid X_{i}=x)=\textup{softmax}(\alpha f(x))_{y}=\frac{\exp(\alpha\langle\theta^{\star}_{y},\phi(x)\rangle)}{\sum_{l=1}^{k}\exp(\alpha\langle\theta^{\star}_{l},\phi(x)\rangle)}.

We vary $\alpha$ in experiments to adjust that the median value of $p_{i}\coloneqq\max_{y\in[m]}\textup{softmax}(\alpha f(x_{i}))_{y}$ , where $p_{i}\in(1/k,1)$ . In the “expert labelers” case, we take $\alpha\to\infty$ so that $\mbox{median}(\{p_{i}\})\to 1$ , while the “crowd labelers” case corresponds to $\alpha\downarrow 0$ and $\mbox{median}(\{p_{i}\})\to 1/k$ .

Given a semisynthetic dataset $\{(X_{i},(Y_{i1},\dots,Y_{im}))\}_{i=1}^{n}$ , $\widehat{\theta}\in\mathbb{R}^{d\times k}$ estimates $\Theta^{\star}$ . We fit $\widehat{\theta}^{\textup{mv}}_{n,m}$ and $\widehat{\theta}^{\textup{mle}}_{n,m}$ using the multiclass logistic loss. We also investigate two standard crowdsourcing approaches for aggregating labels—the Dawid-Skene and GLAD methods [10, 48]—which also produce soft labels. Both methods parameterize the ability of labelers, and GLAD additionally parameterizes task hardness, using the Expectation-Maximization algorithm to estimate the latent parameters. Both methods output estimated (hard) labels $\widehat{Y}_{i}$ as well as soft-labels $\widehat{p}_{i}\in\mathbb{R}_{+}^{k}$ for each example $i$ , where $\widehat{p}_{iy}$ indicates the crowdsourcing model’s estimated probability that example $i$ is of class $y$ . Given these imputed labels, we can fit estimators

\widehat{\theta}=\operatorname*{argmin}_{\theta_{1},\ldots,\theta_{k}}\sum_{i=1}^{n}\log\left(\sum_{l=1}^{k}\exp(\langle X_{i},\theta_{l}-\theta_{\widehat{Y}_{i}})\right)~{}~{}\mbox{or}~{}~{}\widehat{\theta}=\operatorname*{argmin}_{\theta_{1},\ldots,\theta_{k}}\sum_{i=1}^{n}\sum_{y=1}^{k}\widehat{p}_{iy}\log\left(\sum_{l=1}^{k}\exp(\langle X_{i},\theta_{l}-\theta_{y})\right),

the former the hard label estimator and the latter the soft. We let $\widehat{\theta}^{\textup{DS}}_{n,m}$ and $\widehat{\theta}^{\textup{GLAD}}_{n,m}$ denote the corresponding estimators using one-hot hard labels, and $\widehat{\theta}^{\textup{DS-prob}}_{n,m}$ and $\widehat{\theta}^{\textup{GLAD-prob}}_{n,m}$ denote those trained using the estimated per-example label probabilities.

\begin{overpic}[width=281.85034pt]{figure/hard-test-error.pdf} \end{overpic}	\begin{overpic}[width=281.85034pt]{figure/soft-test-error.pdf} \end{overpic}
(a)	(b)
\begin{overpic}[width=216.81pt]{figure/105.pdf} \end{overpic}	\begin{overpic}[width=216.81pt]{figure/42.pdf} \end{overpic}
(c)	(d)

Figure 3: Experiments on the semisynthetic CIFAR-10 dataset. The median labeler accuracy represents the median

p

p_{i}=\max_{y}\textup{softmax}(\alpha f(x_{i}))_{y}

the training data. Results are averaged over

20

trials, and vertical axes give mis-classification rate from (synthetic) ground-truth labels on held-out test set. Legend keys correspond to maximum likelihood

\widehat{\theta}^{\textup{mle}}_{n,m}

(MLE), majority vote

\widehat{\theta}^{\textup{mv}}_{n,m}

(MV), hard-labeled Dawid-Skene (DS) and GLAD crowdsourced estimators

\widehat{\theta}^{\textup{DS}}_{n,m}

and

\widehat{\theta}^{\textup{GLAD}}_{n,m}

, and soft-labaled Dawid-Skene (DS prob) and GLAD (GLAD prob) estimators

\widehat{\theta}^{\textup{DS-prob}}_{n,m}

and

\widehat{\theta}^{\textup{GLAD-prob}}_{n,m}

. (a) Comparison of methods using hard labels. (b) Comparison with crowdsourced estimated soft-labels. (c) Number of labelers

m

versus test error for fixed median accuracy

p=.105

(noisy labelers). (d) Number of labelers

m

versus test error for fixed median accuracy

p=.4

. Both (c) and (d) report 95% error bars over the trials.

We report the results in Fig. 3. As we see, having the ground truth model well-specified, using the full information of soft labels, the MLE $\widehat{\theta}^{\textup{mle}}_{n,m}$ outperforms not only all aggregation methods, but also black-box crowdsourcing models that generate soft labels. The more advanced aggregation methods $\widehat{\theta}^{\textup{DS}}_{n,m}$ and $\widehat{\theta}^{\textup{GLAD}}_{n,m}$ yield smaller test error than $\widehat{\theta}^{\textup{mv}}_{n,m}$ , and using the soft labels, $\widehat{\theta}^{\textup{DS-prob}}_{n,m}$ and $\widehat{\theta}^{\textup{GLAD-prob}}_{n,m}$ further reduce the error, suggesting the benefits of using soft labels from crowdsourcing methods for training even if the underlying model is unknown.

7 Discussion

In spite of the technical detail we require to prove our results, we view this work as almost preliminary and hope that it inspires further work on the full pipeline of statistical machine learning, from dataset creation to model release. Many questions remain both on the theoretical and applied sides of the work.

On the theoretical side, our main focus has been on a stylized model of label aggregation, with majority vote mostly—with the exception of the crowdsourcing model in Sec. 5–functioning as the stand-in for more sophisticated aggregation strategies. It seems challenging to show that no aggregation strategy can work as well as multi-label strategies; it would be interesting to more precisely delineate the benefits and drawbacks of more sophisticated denoising and whether it is useful. We focus throughout on low-dimensional asymptotics, using asymptotic normality to compare estimators. While these insights are valuable, and make predictions consistent with the experimental work we provide, investigating how things may change with high-dimensional scaling or via non-asymptotic results might yield new insights both for theory and methodology. As one example, with high-dimensional scaling, classification datasets often become separable [6, 7], which is consistent with modern applied machine learning [49] but makes the asymptotics we derive impossible. One option is to investigate the asymptotics of maximum-margin estimators, which coincide with the limits of ridge-regularized logistic regression [36, 41].

On the methodological and applied side, given the extent to which the test/challenge dataset methodology drives progress in machine learning [12], it seems that developing newer datasets to incorporate labeler uncertainty could yield substantial benefits. A particular refrain is that modern deep learning methods are overconfident in their predictions [15, 49]; perhaps by calibrating them to labeler uncertainty we could substantially improve their robustness and performance. Additionally, recent large-scale models now frequently use only distantly supervised labels [33, 38, 13], making the study appropriate data cleaning and collection more timely. We look forward to deeper investigations of the intricacies and intellectual foundations of the full practice of statistical machine learning.

Support

This research was partially supported by the Office of Naval Research under awards N00014-19-2288 and N00014-22-1-2669, the Stanford DAWN Consortium, and the National Science Foundation grants IIS-2006777 and HDR-1934578.

References

Asuncion and Newman [2007] A. Asuncion and D. J. Newman. UCI machine learning repository, 2007. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.
Bartlett et al. [2006] P. L. Bartlett, M. I. Jordan, and J. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101:138–156, 2006.
Beygelzimer et al. [2021] A. Beygelzimer, P. Liang, J. Wortman Vaughan, and Y. Dauphin. NeurIPS 2021 datasets and benchmarks track. https://neurips.cc/Conferences/2021/CallForDatasetsBenchmarks, 2021.
Bickel et al. [1998] P. Bickel, C. A. J. Klaassen, Y. Ritov, and J. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Springer Verlag, 1998.
Boucheron et al. [2005] S. Boucheron, O. Bousquet, and G. Lugosi. Theory of classification: a survey of some recent advances. ESAIM: Probability and Statistics, 9:323–375, 2005.
Candès and Sur [2019] E. Candès and P. Sur. A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences, 116(29):14516–14525, 2019.
Candès and Sur [2020] E. Candès and P. Sur. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. Annals of Statistics, 48(1):27–42, 2020.
Charikar et al. [2017] M. Charikar, J. Steinhardt, and G. Valiant. Learning from untrusted data. In Proceedings of the Forty-Ninth Annual ACM Symposium on the Theory of Computing, 2017.
Chen et al. [2010] L. H. Chen, L. Goldstein, and Q.-M. Shao. Normal Approximation by Stein’s method. Springer, 2010.
Dawid and Skene [1979] A. Dawid and A. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, 28:20–28, 1979.
Deng et al. [2009] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. ImageNet: a large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
Donoho [2017] D. L. Donoho. 50 years of data science. Journal of Computational and Graphical Statistics, 26(4):745–766, 2017.
Gadre et al. [2023] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, S. Song, H. Hajishirzi, A. Farhadi, R. Beaumont, S. Oh, A. Dimakis, J. Jitsev, Y. Carmon, V. Shankar, and L. Schmidt. DataComp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems 36, 2023.
Giné and Nickl [2021] E. Giné and R. Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models. Cambridge University Press, 2021.
Goodfellow et al. [2015] I. Goodfellow, O. Vinyals, and A. Saxe. Qualitatively characterizing neural network optimization problems. In Proceedings of the Third International Conference on Learning Representations, 2015.
Hardt and Recht [2022] M. Hardt and B. Recht. Patterns, Predictions, and Actions: A story about machine learning. Princeton University Press, 2022. Available at https://mlstory.org/.
Hastie et al. [2009] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, second edition, 2009.
He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
Horn and Johnson [1985] R. A. Horn and C. R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
Howe [2006] J. Howe. The rise of crowdsourcing. Wired Magazine, 14(6):1–4, 2006.
Karger et al. [2014] D. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1–24, 2014.
Krizhevsky and Hinton [2009] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Le Cam and Yang [2000] L. Le Cam and G. L. Yang. Asymptotics in Statistics: Some Basic Concepts. Springer, 2000.
Ledoux and Talagrand [1991] M. Ledoux and M. Talagrand. Probability in Banach Spaces. Springer, 1991.
Lewis et al. [2004] D. Lewis, Y. Yang, T. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361–397, 2004.
Mammen and Tsybakov [1999] E. Mammen and A. B. Tsybakov. Smooth discrimination analysis. Annals of Statistics, 27:1808–1829, 1999.
Marcus et al. [1994] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19:313–330, 1994.
Owen [1990] A. Owen. Empirical likelihood ratio confidence regions. The Annals of Statistics, 18(1):90–120, 1990.
Peterson et al. [2019] J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626, 2019.
Plan and Vershynin [2013] Y. Plan and R. Vershynin. Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach. IEEE Transactions on Information Theory, 59(1):482–494, 2013.
Plan and Vershynin [2016] Y. Plan and R. Vershynin. The generalized lasso with non-linear observations. IEEE Transactions on Information Theory, 62(3):1528–1537, 2016.
Platanios et al. [2020] E. A. Platanios, M. Al-Shedivat, E. Xing, and T. Mitchell. Learning from imperfect annotations. arXiv:2004.03473 [cs.LG], 2020.
Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Ratner et al. [2017] A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré. Snorkel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269–282, 2017.
Raykar et al. [2010] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11(4), 2010.
Rosset et al. [2004] S. Rosset, J. Zhu, and T. Hastie. Boosting as a regularized path to a maximum margin classifier. Journal of Machine Learning Research, 5:941–973, 2004.
Russakovsky et al. [2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems 35, 2022. journal = arXiv:2210.08402 [cs.CV].
Shapiro et al. [2009] A. Shapiro, D. Dentcheva, and A. Ruszczyński. Lectures on Stochastic Programming: Modeling and Theory. SIAM and Mathematical Programming Society, 2009.
Shevtsova [2014] I. Shevtsova. On the absolute constants in the Berry-Esseen-type inequalities. Doklady Mathematics, 89(3):378–381, 2014.
Soudry et al. [2018] D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(18):1–57, 2018.
Tian and Zhu [2015] T. Tian and J. Zhu. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems 28, pages 1621–1629, 2015.
van der Vaart [1998] A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
van der Vaart and Wellner [1996] A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York, 1996.
Vapnik [1995] V. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
Wainwright [2019] M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019.
Welinder et al. [2010] P. Welinder, S. Branson, P. Perona, and S. Belongie. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pages 2424–2432, 2010.
Whitehill et al. [2009] J. Whitehill, T. Wu, J. Bergsma, J. Movellan, and P. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems 22, 22:2035–2043, 2009.
Zhang et al. [2021] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.

Appendix A Technical lemmas

We collect several technical lemmas and their proofs in this section, which will be helpful in the main proofs. See Appendices A.1, A.2, A.3 and A.4 for their proofs.

Lemma A.1.

Suppose $A,B\in\mathbb{R}^{d\times d}$ are symmetric, $AB=BA=0$ and the matrix $A+B$ is invertible. Then

\displaystyle\left(A+B\right)^{-1}=A^{\dagger}+B^{\dagger}.

The next two lemmas characterize asymptotic behaviors of expectations involving a fixed function $f$ and some random variable $Z$ that satisfies Assumption A2 for given $\beta>0$ and $c_{Z}<\infty$ . To facilitate stating the theorems, we recall that such $Z$ are $(\beta,c_{Z})$ -regular.

Lemma A.2.

Let $\beta>0$ and $f$ be a function on $\mathbb{R}_{+}$ such that $z^{\beta-1}f(z)$ is integrable. If $Z$ is $(\beta,c_{Z})$ -regular (Assumption A2), then

\displaystyle\lim_{t\to\infty}t^{\beta}\mathbb{E}[f(t|Z|)]=c_{Z}\int_{0}^{\infty}z^{\beta-1}f(z)dz.

Lemma A.3.

Let $\beta>0$ and $c_{Z}<\infty$ , and $Z$ be $(\beta,c_{Z})$ -regular, let $f:\mathbb{R}_{+}\to\mathbb{R}$ satisfy $|f(z)|\leq a_{0}+a_{p}z^{p}$ for some $a_{0},a_{p}<\infty$ and all $z\in\mathbb{R}$ , and assume $|Z|$ has finite $p$ th moment. Additionally let $\rho_{m}(t)$ be the majority vote prediction function (15) and Assumption A4 hold for each $\sigma_{j}^{\star}$ with limiting average derivative ${\overline{\sigma}^{\star}}^{\prime}(0)$ at zero. Then for any $c>0$

\displaystyle\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}|Z|)(1-\rho_{m}(cZ))\right]=c_{Z}\int_{0}^{\infty}z^{\beta-1}f(z)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)cz\right)dz,

where $\Phi(z)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-\frac{t^{2}}{2}}dt$ is the standard normal cumulative distribution function.

The fourth lemma is a uniform convergence result for the empirical risk in the case that we have potentially distinct link functions (cf. Sec. 5.1).

Lemma A.4.

Assume $\mathbb{E}[\left\|{X}\right\|_{2}^{\gamma}]\leq M^{\gamma}$ for some $M\geq 1$ and $\gamma\geq 2$ and let the radius $1\leq r<\infty$ . Let $\mathcal{F}_{\mathsf{link}}\subset\{\sigma:\mathbb{R}\to[0,1],\left\|{\sigma}\right\|_{\textup{Lip}}\leq\mathsf{L}\}$ . Then for a constant $C\lesssim\sqrt{d\mathsf{L}}$ , we have

\mathbb{E}\left[\sup_{\left\|{\theta}\right\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})|\right]\leq C\cdot(Mr)^{\frac{4\gamma}{3\gamma+1}}\left(\frac{m}{n}\right)^{\frac{\gamma}{3\gamma+1}}\sqrt{\log(rn)}.

A.1 Proof of Lemma A.1

As $AB=BA=0$ , the symmetric matrices $A$ and $B$ commute and so are simultaneously orthogonally diagonalizable [19, Thm. 4.5.15]. As $AB=BA=0$ , we can thus write

\displaystyle A=U\begin{bmatrix}\Lambda_{1}&0\\ 0&0\end{bmatrix}U^{\top},\qquad B=U\begin{bmatrix}0&0\\ 0&\Lambda_{2}\end{bmatrix}U^{\top},

for some orthogonal $U\in\mathbb{R}^{d\times d}$ , and as $A+B$ is invertible, $\Lambda_{1},\Lambda_{2}$ are invertible diagonal matrices. We conclude the proof by writing

\displaystyle(A+B)^{-1}=U\begin{bmatrix}\Lambda_{1}^{-1}&0\\ 0&\Lambda_{2}^{-1}\end{bmatrix}U^{\top}=U\begin{bmatrix}\Lambda_{1}^{-1}&0\\ 0&0\end{bmatrix}U^{\top}+U\begin{bmatrix}0&0\\ 0&\Lambda_{2}^{-1}\end{bmatrix}U^{\top}=A^{\dagger}+B^{\dagger}.

A.2 Proof of Lemma A.2

By the change of variables $w=tz$ , we have

	$\displaystyle t^{\beta}\mathbb{E}[f(t\|Z\|)]$	$\displaystyle=\int_{0}^{\infty}f(tz)\cdot t^{\beta-1}p(z)\cdot tdz=\int_{0}^{\infty}f(w)\cdot t^{\beta-1}p(w/t)dw$
		$\displaystyle=\int_{0}^{\infty}w^{\beta-1}f(w)\cdot(w/t)^{1-\beta}p(w/t)dw.$

As $|w^{\beta-1}f(w)\cdot(w/t)^{1-\beta}p(w/t)|\leq\sup_{z\in(0,\infty)}z^{1-\beta}p(z)\cdot|w^{\beta-1}f(w)|$ , where $w^{\beta-1}f(w)$ is integrable by assumption, we can invoke dominated convergence to see that

\displaystyle\lim_{t\to\infty}t^{\beta}\mathbb{E}[f(t|Z|)]=\int_{0}^{\infty}w^{\beta-1}f(w)\cdot\lim_{t\to\infty}(w/t)^{1-\beta}p(w/t)dw=c_{Z}\int_{0}^{\infty}w^{\beta-1}f(w)dw.

A.3 Proof of Lemma A.3

By rescaling arguments, it suffices to prove the theorem for any function $f$ satisfying $|f(z)|\leq 1+z^{p}$ on $\mathbb{R}_{+}$ for any $p\in\mathbb{N}$ such that $|Z|$ has finite $p$ th moment. For such $f$ , we wish to show

\displaystyle\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}|Z|)(1-\rho_{m}(cZ))\right]=c_{Z}\int_{0}^{\infty}z^{\beta-1}f(z)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)cz\right)dz,

where $\Phi$ is the standard normal cdf. The key insight is that we can approximate $1-\rho_{m}(t)$ by a suitable Gaussian cumulative distribution function (recognizing that $\rho_{m}(t)>\frac{1}{2}$ for $t\neq 0$ by definition (15) as the probability the majority vote is correct given margin $\langle\theta^{\star},X\rangle=t$ ).

We first assume $Z\geq 0$ with probability $1$ , as the general result follows by writing $Z=(Z)_{+}-(-Z)_{+}$ . We decompose into two expectations, depending on $Z$ being large or small:

	$\displaystyle m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}\|Z\|)(1-\rho_{m}(cZ))\right]$
		$\displaystyle=\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}Z)(1-\rho_{m}(cZ))\mathds{1}\left\{0\leq Z\leq\frac{M}{\sqrt{m}}\right\}\right]}_{\mathrm{(I)}}+\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}Z)(1-\rho_{m}(cZ))\mathds{1}\left\{Z>\frac{M}{\sqrt{m}}\right\}\right]}_{\mathrm{(II)}}.$		(16)

The proof consists of three main parts.

1.

We approximate $1-\rho_{m}(t)$ by a Gaussian cdf.

We can approximate term (I) by replacing $1-\rho_{m}(t)$ with the Gaussian cdf, showing that

\displaystyle\left|\lim_{m\to\infty}\mathrm{(I)}-c_{Z}\int_{0}^{\infty}z^{\beta-1}f(z)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)cz\right)dz\right|=o_{M}(1).

(17)

3.

For term (II), we show $1-\rho_{m}(cz)$ is small when $Z>M/\sqrt{m}$ , which allows us to show that

$\limsup_{m\to\infty}\left|\mathrm{(II)}\right|=o_{M}(1).$

Thus by adding the two preceding displays and taking $M\to\infty$ , we obtain the lemma.

Before we dive into further details, we use the shorthand functions

\sigma_{m}^{\textup{std}}(z)\coloneqq,\frac{\sum_{j=1}^{m}({\sigma_{j}^{\star}(cz)-\frac{1}{2}})}{\sqrt{\sum_{j=1}^{m}\sigma_{j}^{\star}(cz)({1-\sigma_{j}^{\star}(cz)})}},~{}~{}~{}\Delta_{m}(z)\coloneqq 1-\rho_{m}(cz)-\Phi(-\sigma_{m}^{\textup{std}}(z)),

(18)

and we also write $p_{\infty}(\beta):=\sup_{z\in(0,\infty)}z^{1-\beta}p(z)<\infty$ .

Part 1. Normal approximation for $1-\rho_{m}(t)$ when $t=O(1/\sqrt{m})$ .

Let $p_{j}=\sigma_{j}^{\star}(t)$ for shorthand and $Y_{j}\sim\mathrm{Bernoulli}(p_{j})$ be independent random variables. For $t>0$ , then

\displaystyle 1-\rho_{m}(t)

\displaystyle=\mathbb{P}\left({Y_{1}+\cdots+Y_{m}<\frac{1}{2}}\right)=\mathbb{P}\left({\frac{\sum_{j=1}^{m}(Y_{j}-p_{j})}{\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}}<-\frac{\sum_{j=1}^{m}(p_{j}-\frac{1}{2})}{\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}}}\right).

Consider the centered and standardized random variables $\xi_{j}=(Y_{j}-p_{j})/\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}$ so that $\xi_{1},\dots,\xi_{m}$ are zero mean, mutually independent, and satisfy

	$\displaystyle\sum_{j=1}^{m}\mathsf{Var}\left({\xi_{j}}\right)$	$\displaystyle=1,$
	$\displaystyle\sum_{j=1}^{m}\mathbb{E}\left[{\left\|\xi_{j}\right\|^{3}}\right]$	$\displaystyle=\frac{\sum_{j=1}^{m}p_{j}(1-p_{j})(p_{j}^{2}+(1-p_{j})^{2})}{({\sum_{j=1}^{m}p_{j}(1-p_{j})})^{3/2}}\leq\frac{\max_{1\leq j\leq m}(p_{j}^{2}+(1-p_{j})^{2})}{\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}}.$

By the Berry-Esseen theorem (cf. Chen et al. [9], Shevtsova [40]), for all $t>0$

\displaystyle\left|1-\rho_{m}(t)-\Phi\left(-\frac{\sum_{j=1}^{m}\left(p_{j}-\frac{1}{2}\right)}{\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}}\right)\right|\leq\frac{3}{4}\cdot\frac{\max_{1\leq j\leq m}(p_{j}^{2}+(1-p_{j})^{2})}{\sqrt{\sum_{j=1}^{m}p_{j}(1-p_{j})}}.

Fix any $M<\infty$ . Then for $0\leq t\leq cM/\sqrt{m}$ and large enough $m$ , the right hand side of the preceding display has the upper bound $2/\sqrt{m}$ , as in numerator we have $p_{j}^{2}+(1-p_{j})^{2}\leq 1$ , and in denominator $\min_{1\leq j\leq m}p_{j}(1-p_{j})\to 1/4$ for all $j$ as $m\to\infty$ , which follows from Assumption A4 that

\frac{1}{2}\leq\limsup_{m\to\infty}\max_{1\leq j\leq m}p_{j}\leq\limsup_{m\to\infty}\sup_{1\leq j<\infty}{\sigma_{j}^{\star}}\left({\frac{cM}{\sqrt{m}}}\right)=\frac{1}{2}.

By repeating the same argument for $-cM/\sqrt{m}\leq t<0$ , we obtain that for large $m$ and $|t|\leq cM/\sqrt{m}$ ,

\displaystyle\left|1-\rho_{m}(t)-\Phi\left(-\left|\frac{\sum_{j=1}^{m}\left(\sigma_{j}^{\star}(t)-\frac{1}{2}\right)}{\sqrt{\sum_{j=1}^{m}\sigma_{j}^{\star}(t)\left({1-\sigma_{j}^{\star}(t)}\right)}}\right|\right)\right|\leq\frac{2}{\sqrt{m}}.

(19)

Part 2. Approximating (I) by Gaussian cdf.

For the first term (I) in (16), we further decompose into a normal approximation term and an error term,

\displaystyle\mathrm{(I)}=\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}Z)\Phi(-\sigma_{m}^{\textup{std}}(z))\mathds{1}\left\{0\leq Z\leq\frac{M}{\sqrt{m}}\right\}\right]}_{\mathrm{(III)}}+\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[f(\sqrt{m}Z)\Delta_{m}(z)\mathds{1}\left\{0\leq Z\leq\frac{M}{\sqrt{m}}\right\}\right]}_{\mathrm{(IV)}},

where $\Delta_{m}(z)=1-\rho_{m}(cz)-\Phi(-\sigma_{m}^{\textup{std}}(z))$ as in def. (18). We will show $\mathrm{(IV)}\to 0$ and so $\mathrm{(III)}$ dominates. By the change of variables $w=\sqrt{m}z$ , we can further write (III) as

	$\displaystyle\mathrm{(III)}$	$\displaystyle=m^{\frac{\beta}{2}}\int_{0}^{\frac{M}{\sqrt{m}}}f(\sqrt{m}z)\Phi(-\sigma_{m}^{\textup{std}}(z))\cdot p(z)dz$
		$\displaystyle=\int_{0}^{M}w^{\beta-1}f(w)\Phi\left(-\sigma_{m}^{\textup{std}}\left({\frac{w}{\sqrt{m}}}\right)\right)\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})dw.$

We want to take the limit $m\to\infty$ and apply dominated convergence theorem. Because $(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})\leq p_{\infty}(\beta)<\infty$ and $\sigma_{m}^{\textup{std}}(w/\sqrt{m})\geq 0$ , we have

\displaystyle w^{\beta-1}f(w)\Phi\left(-\sigma_{m}^{\textup{std}}\left({\frac{w}{\sqrt{m}}}\right)\right)\cdot\left(-\sigma_{m}^{\textup{std}}\left({\frac{w}{\sqrt{m}}}\right)\right)\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})\leq w^{\beta-1}f(w)\cdot\Phi(0)p_{\infty}(\beta).

As $\beta>0$ and $|f(w)|\leq 1+w^{p}$ , $w^{\beta-1}f(w)$ is integrable on $[0,M]$ , and by (14a) and (14c) in Assumption A4,

\displaystyle\lim_{m\to\infty}\sigma_{m}^{\textup{std}}\left({\frac{w}{\sqrt{m}}}\right)

\displaystyle=\lim_{m\to\infty}\frac{\sqrt{m}\left(\overline{\sigma}_{m}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)-\frac{1}{2}\right)}{\sqrt{\frac{1}{m}\sum_{j=1}^{m}\sigma_{j}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)\left({1-\sigma_{j}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)}\right)}}=2{\overline{\sigma}^{\star}}^{\prime}(0)cw.

Using the above display and that $\lim_{m\to\infty}(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})=c_{Z}$ , we can thus apply dominated convergence theorem to conclude that

\displaystyle\lim_{m\to\infty}\mathrm{(III)}

\displaystyle=c_{Z}\int_{0}^{M}w^{\beta-1}f(w)\cdot\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)cw\right)dw=c_{Z}\int_{0}^{\infty}w^{\beta-1}f(w)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)cw\right)dw+o_{M}(1).

Next we turn to the error term (IV). By the bound (19), $|\Delta_{m}(z)|\leq 2/\sqrt{m}$ when $|z|\leq M/\sqrt{m}$ for large enough $m$ , and substituting $w=\sqrt{m}z$ ,

	$\displaystyle\|\mathrm{(IV)}\|$	$\displaystyle\leq m^{\frac{\beta}{2}}\mathbb{E}\left[\|f(\sqrt{m}Z)\|\cdot\frac{2}{\sqrt{m}}\cdot\mathds{1}\left\{0\leq Z\leq\frac{M}{\sqrt{m}}\right\}\right]$
		$\displaystyle=\frac{2}{\sqrt{m}}\int_{0}^{\frac{M}{\sqrt{m}}}\sqrt{m}\cdot\|f(\sqrt{m}z)\|\cdot m^{\frac{\beta-1}{2}}p(z)dz=\frac{2}{\sqrt{m}}\int_{0}^{M}w^{\beta-1}\|f(w)\|\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})dw.$

By using Assumption A2 again that $(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})\leq p_{\infty}(\beta)<\infty$ , we further have

|\mathrm{(IV)}|\leq\frac{2p_{\infty}(\beta)}{\sqrt{m}}\cdot\int_{0}^{M}w^{\beta-1}|f(w)|dw\leq\frac{2p_{\infty}(\beta)}{\sqrt{m}}\cdot\left({\frac{M^{\beta}}{\beta}+\frac{M^{p+\beta}}{p+\beta}}\right)\to 0,

where we use $|f(w)|\leq 1+w^{p}$ . We have thus shown the limit (17).

Part 3. Upper bounding (II).

In term (II), when $Z$ is large, the key is that the quantity $1-\rho_{m}(t)$ is small when $|t|\geq cM/\sqrt{m}$ : Hoeffding’s inequality implies the tail bound

\displaystyle 0\leq 1-\rho_{m}(t)\leq e^{-2\left(\overline{\sigma}_{m}^{\star}(t)-\frac{1}{2}\right)^{2}m}.

Thus

\displaystyle|\mathrm{(II)}|

\displaystyle\leq\int_{\frac{M}{\sqrt{m}}}^{1}\sqrt{m}|f(\sqrt{m}z)|e^{-2\left(\overline{\sigma}_{m}^{\star}(cz)-\frac{1}{2}\right)^{2}m}\cdot m^{\frac{\beta-1}{2}}p(z)dz+\int_{1}^{\infty}m^{\frac{\beta}{2}}|f(\sqrt{m}z)|e^{-2\left(\overline{\sigma}_{m}^{\star}(cz)-\frac{1}{2}\right)^{2}m}\cdot p(z)dz.

Using the assumption that $\gamma\coloneqq\liminf_{m}\inf_{t\geq c}\left({\overline{\sigma}_{m}^{\star}(t)-\frac{1}{2}}\right)>0$ from (14b), that $|f(z)|\leq 1+z^{p}$ and $|Z|$ has finite $p$ th moment, we observe that

\displaystyle\int_{1}^{\infty}m^{\frac{\beta}{2}}|f(\sqrt{m}z)|e^{-2\left(\overline{\sigma}_{m}^{\star}(cz)-\frac{1}{2}\right)^{2}m}\cdot p(z)dz\leq m^{\frac{1}{2}(\beta+p)}e^{-2\gamma^{2}m}\int_{1}^{\infty}(1+z^{p})p(z)dz\to 0,

and consequently

	$\displaystyle\limsup_{m\to\infty}\|\mathrm{(II)}\|$	$\displaystyle\leq\limsup_{m\to\infty}\int_{\frac{M}{\sqrt{m}}}^{1}\sqrt{m}\|f(\sqrt{m}z)\|e^{-2\left(\overline{\sigma}_{m}^{\star}(cz)-\frac{1}{2}\right)^{2}m}\cdot m^{\frac{\beta-1}{2}}p(z)dz$
		$\displaystyle=\limsup_{m\to\infty}\int_{M}^{\sqrt{m}}w^{\beta-1}\|f(w)\|e^{-2\left(\overline{\sigma}_{m}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)-\frac{1}{2}\right)^{2}m}\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})dw.$

For $w\in[M,\sqrt{m}]$ , we have

\displaystyle\left(\overline{\sigma}_{m}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)-\frac{1}{2}\right)^{2}m

\displaystyle=\left(\frac{\overline{\sigma}_{m}^{\star}\left({\frac{cw}{\sqrt{m}}}\right)-\frac{1}{2}}{\frac{cw}{\sqrt{m}}}\right)^{2}c^{2}w^{2}\geq\left(\inf_{0<t\leq c}\frac{\overline{\sigma}_{m}^{\star}(t)-\frac{1}{2}}{t}\right)^{2}c^{2}w^{2},

while Assumption (14b) gives $\delta:=\inf_{0<t\leq c}\frac{\overline{\sigma}_{m}^{\star}(t)-\frac{1}{2}}{t}>0$ , so

\displaystyle\limsup_{m\to\infty}|\mathrm{(II)}|

\displaystyle\leq\int_{M}^{\sqrt{m}}w^{\beta-1}|f(w)|e^{-\delta w^{2}}\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})dw.

Using the inequality $(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})\leq p_{\infty}(\beta)<\infty$ , we apply dominated convergence:

\displaystyle\lim_{m\to\infty}\int_{M}^{\sqrt{m}}w^{\beta-1}|f(w)|e^{-\delta w^{2}}\cdot(w/\sqrt{m})^{1-\beta}p(w/\sqrt{m})dw=c_{Z}\int_{M}^{\infty}w^{\beta-1}|f(w)|e^{-\delta w^{2}}dw=o_{M}(1).

A.4 Proof of Lemma A.4

We follow a typical symmetrization approach, then construct a covering that we use to prove the lemma. Let $P_{n}^{0}=n^{-1}\sum_{i=1}^{n}\varepsilon_{i}1_{X_{i},Y_{i}}$ be the (random) symmetrized measure with point masses at $(X_{i},Y_{i})$ for $Y_{i}=(Y_{i1},\ldots,Y_{im})$ . Then by a standard symmetrization argument, we have

\mathbb{E}\left[\sup_{\left\|{\theta}\right\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})|\right]\leq 2\mathbb{E}\left[\sup_{\left\|{\theta}\right\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}\left|P_{n}^{0}\ell_{\vec{\sigma},\theta}\right|\right].

(20)

We use a covering argument to bound the symmetrized expectation (20). Let $R<\infty$ to be chosen, and for an (again, to be determined) $\epsilon>0$ let $\mathcal{G}\subset\mathcal{F}_{\mathsf{link}}$ denote an $\epsilon$ -cover of $\mathcal{F}_{\mathsf{link}}$ in the supremum norm on $[-R,R]$ , that is, $\left\|{g-\sigma}\right\|=\sup_{t\in[-R,R]}|g(t)-\sigma(t)|$ , and so for each $\sigma\in\mathcal{F}_{\mathsf{link}}$ there exists $g\in\mathcal{G}$ such that $\left\|{g-\sigma}\right\|\leq\epsilon$ . Then [44, Ch. 2.7] we have $\log\textup{card}(\mathcal{G})\leq O(1)\frac{R\mathsf{L}}{\epsilon}$ . Let $\Theta_{\epsilon}$ be a minimal $\epsilon$ -cover of $\{\theta\mid\left\|{\theta}\right\|_{2}\leq r\}$ in $\left\|{\cdot}\right\|_{2}$ , so that $\log\textup{card}(\Theta_{\epsilon})\leq d\log(1+\frac{2r}{\epsilon})$ and $\max_{\theta\in\Theta_{\epsilon}}\left\|{\theta}\right\|_{2}\leq r$ . We claim that for each $\left\|{\theta}\right\|_{2}\leq r$ and $\sigma\in\mathcal{F}_{\mathsf{link}}$ , there exists $v\in\Theta_{\epsilon}$ and $g\in\mathcal{G}$ such that

\left|\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}\left(\ell_{\sigma,\theta}(Y_{ij}\mid X_{i})-\ell_{g,v}(Y_{ij}\mid X_{i})\right)\right|\leq\epsilon+\frac{1}{n}\sum_{i=1}^{n}\left\|{X_{i}}\right\|_{2}\epsilon+\frac{1}{n}\sum_{i=1}^{n}1\!\left\{{\left\|{X_{i}}\right\|_{2}\geq R/r}\right\}.

(21)

Indeed, for any $g\in\mathcal{F}_{\mathsf{link}}$ and $\theta,v\in\mathbb{R}^{d}$ , we have

	$\displaystyle\left\|\ell_{\sigma,\theta}(y\mid x)-\ell_{g,v}(y\mid x)\right\|$	$\displaystyle=\left\|\int_{0}^{y\langle x,\theta\rangle}\sigma(-t)dt-\int_{0}^{y\langle x,v\rangle}g(-t)dt\right\|$
		$\displaystyle\leq\sup_{\|t\|\leq r\left\\|{x}\right\\|_{2}}\|\sigma(t)-g(t)\|+\left\|\int_{y\langle x,v\rangle}^{y\langle x,\theta\rangle}\|\sigma(-t)-g(-t)\|dt\right\|$
		$\displaystyle\leq\left\\|{\sigma-g}\right\\|+1\!\left\{{\left\\|{x}\right\\|_{2}\geq R/r}\right\}+\|\langle x,\theta-v\rangle\|$
		$\displaystyle\leq\left\\|{\sigma-g}\right\\|+\left\\|{x}\right\\|_{2}\left\\|{\theta-v}\right\\|_{2}+1\!\left\{{\left\\|{x}\right\\|_{2}\geq R/r}\right\},$

where we have used that $\sigma,g\in[0,1]$ . Taking the elements $g,v$ in the respective coverings to minimize the above bound gives the guarantee (21).

We now leverage inequality (21) in the symmetrization step (20). We have

	$\displaystyle\mathbb{E}\left[\sup_{\left\\|{\theta}\right\\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}\|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})\|\right]$
		$\displaystyle\lesssim\mathbb{E}\left[\max_{\theta\in\Theta_{\epsilon}}\max_{\vec{g}\in\mathcal{G}^{m}}\left\|P_{n}^{0}\ell_{\vec{g},\theta}\right\|\right]+\epsilon+\mathbb{E}[\left\\|{X_{1}}\right\\|_{2}]\epsilon+\frac{1}{n}\sum_{i=1}^{n}\mathbb{P}(\left\\|{X_{i}}\right\\|_{2}\geq R/r)$
		$\displaystyle\stackrel{{\scriptstyle(i)}}{{\lesssim}}\sqrt{\frac{dmR\mathsf{L}}{\epsilon}\log\left(1+\frac{2r}{\epsilon}\right)}\mathbb{E}\left[\frac{r^{2}}{n}\sum_{i=1}^{n}\left\\|{X_{i}}\right\\|_{2}^{2}\right]^{1/2}+\epsilon+\mathbb{E}[\left\\|{X_{1}}\right\\|_{2}]\epsilon+\frac{\mathbb{E}[\left\\|{X_{i}}\right\\|_{2}^{\gamma}]}{(R/r)^{\gamma}}$
		$\displaystyle\leq Mr\sqrt{\frac{dmR\mathsf{L}}{n\epsilon}\log\left(1+\frac{2r}{\epsilon}\right)}+(M+1)\epsilon+\frac{M^{\gamma}}{(R/r)^{\gamma}},$

where inequality $(i)$ uses that if $Z_{i}$ are $\tau^{2}$ -sub-Gaussian, then $\mathbb{E}[\max_{i\leq N}|Z_{i}|]\leq\sqrt{2\tau^{2}\log N}$ , and that conditional on $\{X_{i},Y_{i}\}_{i=1}^{n}$ , the symmetrized sum $\sum_{i=1}^{n}\varepsilon_{i}\frac{1}{m}\sum_{j=1}^{m}\ell_{g_{j},\theta}(Y_{ij}\mid X_{i})$ is $r^{2}\sum_{i=1}^{n}\left\|{X_{i}}\right\|_{2}^{2}$ -sub-Gaussian, as $|\ell_{g_{j},\theta}(Y_{ij}\mid X_{i})|\leq|\langle X_{i},\theta\rangle|\leq r\left\|{X_{i}}\right\|_{2}$ . We optimize this bound to get the final guarantee of the lemma: set $\epsilon=(Rm/n)^{1/3}$ (and note that we will choose $R\geq 1$ ) to obtain

\mathbb{E}\left[\sup_{\left\|{\theta}\right\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})|\right]\lesssim\sqrt{d\mathsf{L}\log(rn)}Mr\left(\frac{Rm}{n}\right)^{1/3}+\frac{(Mr)^{\gamma}}{R^{\gamma}}.

Choose $R=((Mr)^{3(\gamma-1)}n/m)^{\frac{1}{3\gamma+1}}$ .

Appendix B Proof of Lemma 3.1

Recall that $h=h_{t^{\star},m}$ is the calibration gap function (10). We see that because $\mathbb{E}[Z]=0$ , we have $h(0)=-2\mathbb{E}[Z\varphi_{m}(t^{\star}Z)]<0$ , while using Assumption A3 and that $\mathbb{E}[|Z|]<\infty$ , we apply dominated convergence and that $\lim_{t\to\infty}(\sigma(tZ)-\frac{1}{2})Z=c|Z|$ with probability 1 to obtain

\displaystyle\lim_{t\to\infty}h(t)=\mathbb{E}[c|Z|(1-\varphi_{m}(t^{\star}Z))]+\mathbb{E}[c|Z|\varphi_{m}(t^{\star}Z)]=c\mathbb{E}[|Z|]>0.

Because $h^{\prime}(t)=\mathbb{E}[\sigma^{\prime}(tZ)Z^{2}(1-\varphi_{m}(t^{\star}Z))]+\mathbb{E}[\sigma^{\prime}(-tZ)Z^{2}\varphi_{m}(t^{\star}Z)]>0$ , we see that there is a unique $t_{m}$ solving $h(t_{m})=0$ , and evidently $\theta^{\star}_{L}=t_{m}u^{\star}$ is a minimizer of $L$ .

We compute the Hessian of $L$ . For this, we again let $\theta=tu^{\star}$ to write

	$\displaystyle\nabla^{2}L(\theta,\sigma)$	$\displaystyle=\mathbb{E}[\sigma^{\prime}(-\langle\theta,X\rangle)XX^{\top}\varphi_{m}(\langle X,\theta^{\star}\rangle)]+\mathbb{E}[\sigma^{\prime}(\langle\theta,X\rangle)XX^{\top}(1-\varphi_{m}(\langle X,\theta^{\star}\rangle))]$
		$\displaystyle=\mathbb{E}\left[\sigma^{\prime}(-tZ)\varphi_{m}(t^{\star}Z)\left(Z^{2}u^{\star}{u^{\star}}^{\top}+WW^{\top}\right)\right]+\mathbb{E}\left[\sigma^{\prime}(tZ)(1-\varphi_{m}(t^{\star}Z))\left(Z^{2}u^{\star}{u^{\star}}^{\top}+WW^{\top}\right)\right]$
		$\displaystyle=\mathbb{E}\left[\left(\sigma^{\prime}(-tZ)\varphi_{m}(t^{\star}Z)+\sigma^{\prime}(tZ)(1-\varphi_{m}(t^{\star}Z))\right)Z^{2}\right]u^{\star}{u^{\star}}^{\top}$
		$\displaystyle\quad+\mathbb{E}\left[\sigma^{\prime}(-tZ)\varphi_{m}(t^{\star}Z)+\sigma^{\prime}(tZ)(1-\varphi_{m}(t^{\star}Z))\right]\mathbb{E}[WW^{\top}],$

which gives $\nabla^{2}L(\theta,\sigma)\succ 0$ and so $\theta^{\star}_{L}$ is unique. Finally, the desired form of the Hessian follows as $\mathbb{E}[WW^{\top}]=\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}$ .

Appendix C Proof of Theorem 1

Let $t_{m}$ be the solution to $h_{t^{\star},m}(t)=0$ as in Lemma 3.1. The consistency argument is immediate: the losses $\ell$ are convex, continuous, and locally Lipschitz, so Shapiro et al. [39, Thm. 5.4] gives $\widehat{\theta}_{n}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\theta^{\star}_{L}$ . By an appeal to standard M-estimator theory [e.g. 43, Thm. 5.23], we thus obtain

\sqrt{n}\big{(}\widehat{\theta}_{n}-\theta^{\star}_{L}\big{)}\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\nabla^{2}L(\theta^{\star}_{L},\sigma)^{-1}\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y_{j}\mid X)\bigg{)}\nabla^{2}L(\theta^{\star}_{L},\sigma)^{-1}\right).

(22)

We expand the covariance term to obtain the first main result of the theorem. For shorthand, let $G_{j}=\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y_{j}\mid X)$ , so that $\sum_{j=1}^{m}\mathbb{E}[G_{j}]=0$ and the $G_{j}$ are conditionally independent given $X$ . Applying the law of total covariance, we have

	$\displaystyle\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}G_{j}\bigg{)}$	$\displaystyle=\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[G_{j}\mid X]\bigg{)}+\mathbb{E}\left[\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}G_{j}\mid X\bigg{)}\right]$
		$\displaystyle=\underbrace{\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[G_{j}\mid X]\bigg{)}}_{\mathrm{(I)}}+\frac{1}{m^{2}}\sum_{j=1}^{m}\underbrace{\mathbb{E}\left[\mathsf{Cov}(G_{j}\mid X)\right]}_{\mathrm{(II)}},$

where we have used the conditional independence of the $Y_{j}$ conditional on $X$ . We control each of terms (I) and (II) in turn.

For the first, we have by the independent decomposition $X=Zu^{\star}+\mathsf{P}_{u^{\star}}^{\perp}W$ that

\mathbb{E}[G_{j}\mid X]=\left(\sigma(t_{m}Z)(1-\sigma_{j}^{\star}(t^{\star}Z))-\sigma(-t_{m}Z)\sigma_{j}^{\star}(t^{\star}Z)\right)X,

and so

	$\displaystyle\mathrm{(I)}$	$\displaystyle=\mathbb{E}\left[\left(\sigma(t_{m}Z)(1-\overline{\sigma}^{\star}(t^{\star}Z))-\sigma(-t_{m}Z)\overline{\sigma}^{\star}(t^{\star}Z)\right)^{2}XX^{\top}\right]$
		$\displaystyle=\mathbb{E}\left[\left(\sigma(t_{m}Z)(1-\overline{\sigma}^{\star}(t^{\star}Z))-\sigma(-t_{m}Z)\overline{\sigma}^{\star}(t^{\star}Z)\right)^{2}Z^{2}\right]u^{\star}{u^{\star}}^{\top}$
		$\displaystyle\qquad~{}+\mathbb{E}\left[\left(\sigma(t_{m}Z)(1-\overline{\sigma}^{\star}(t^{\star}Z))-\sigma(-t_{m}Z)\overline{\sigma}^{\star}(t^{\star}Z)\right)^{2}\right]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}$
		$\displaystyle=\mathbb{E}[\mathsf{le}(Z)^{2}Z^{2}]u^{\star}{u^{\star}}^{\top}+\mathbb{E}[\mathsf{le}(Z)^{2}]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp},$

where we used the independence of $W$ and $Z$ . For term (II) above, we see that conditional on $X$ , $\nabla\ell_{\sigma,\theta}(Y_{j}\mid X)$ is a binary random variable taking values in $\{-\sigma(-t_{m}Z)X,\sigma(t_{m}Z)X\}$ with probabilities $\sigma_{j}^{\star}(t^{\star}Z)$ and $1-\sigma_{j}^{\star}(t^{\star}Z)$ , respectively, so that a calculation leveraging the variance of Bernoulli random variables yields

\displaystyle\mathsf{Cov}(G_{j}\mid X)

\displaystyle=\sigma_{j}^{\star}(t^{\star}Z)(1-\sigma_{j}^{\star}(t^{\star}Z))\left(\sigma(t_{m}Z)+\sigma(-t_{m}Z)\right)^{2}XX^{\top}=v_{j}(Z)XX^{\top},

where we used the definition (12) of the variance terms. A similar calculation to that we used for term (I) then gives that

\displaystyle\mathrm{(II)}

\displaystyle=\frac{1}{m^{2}}\sum_{j=1}^{m}\mathbb{E}[v_{j}(Z)Z^{2}]u^{\star}{u^{\star}}^{\top}+\frac{1}{m^{2}}\sum_{j=1}^{m}\mathbb{E}[v_{j}(Z)]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.

Applying Lemma A.1 then allows us to decompose the the covariance in expression (22) into terms in the span of $u^{\star}{u^{\star}}^{\top}$ and those perpendicular to it, so that the asymptotic covariance is

\displaystyle\frac{\mathbb{E}[\mathsf{le}(Z)^{2}Z^{2}]+\frac{1}{m^{2}}\sum_{j=1}^{m}\mathbb{E}[v_{j}(Z)Z^{2}]}{\mathbb{E}[\mathsf{he}(Z)Z^{2}]^{2}}u^{\star}{u^{\star}}^{\top}+\frac{\mathbb{E}[\mathsf{le}(Z)^{2}]+\frac{1}{m^{2}}\sum_{j=1}^{m}\mathbb{E}[v_{j}(Z)]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger},

by applying Lemma 3.1 for the form of the Hessian $\nabla^{2}L(\theta_{L}^{\star},\sigma)$ and Lemma A.1 for the inverse, giving the first result of the theorem.

To obtain the second result, we apply the delta method with the mapping $\phi(x)=x/\left\|{x}\right\|_{2}$ , which satisfies $\nabla\phi(x)=(I-\phi(x)\phi(x)^{\top})/\left\|{x}\right\|_{2}$ , so that for $\widehat{u}_{n}=\widehat{\theta}_{n}/\|{\widehat{\theta}_{n}}\|_{2}$ we have

\displaystyle\sqrt{n}(\widehat{u}_{n}-u^{\star})

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{1}{t_{m}^{2}}\frac{\mathbb{E}[\mathsf{le}(Z)^{2}]+\frac{1}{m^{2}}\sum_{j=1}^{m}\mathbb{E}[v_{j}(Z)]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}\right)

as desired.

Appendix D Proof of Theorem 2

As in the proof of Theorem 1, we begin with a consistency result. Let $t_{m}$ be the solution to $h_{t^{\star},m}(t)=0$ as in Lemma 3.1. The once again, Shapiro et al. [39, Thm. 5.4] shows that $\widehat{\theta}_{n}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\theta^{\star}_{L}$ . As previously, appealing to standard M-estimator theory [e.g. 43, Thm. 5.23], we obtain

\sqrt{n}\big{(}\widehat{\theta}_{n}-\theta^{\star}_{L}\big{)}\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\nabla^{2}L(\theta^{\star}_{L},\sigma)^{-1}\mathsf{Cov}\left(\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y^{+}\mid X)\right)\nabla^{2}L(\theta^{\star}_{L},\sigma)^{-1}\right),

(23)

the difference from the asymptotic (22) appearing in the covariance term. Here, we recognize that conditional on $X=Zu^{\star}+W$ , the vector $\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y^{+}\mid X)$ takes on the values $\{-\sigma(-t_{m}Z)X,\sigma(t_{m}Z)X\}$ each with probabilities $\varphi_{m}(t^{\star}Z)$ and $1-\varphi_{m}(t^{\star}Z)$ , respectively, while $\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y^{+}\mid X)$ is (unconditionally) mean zero. Thus we have

	$\displaystyle\mathsf{Cov}\left(\nabla\ell_{\sigma,\theta^{\star}_{L}}(Y^{+}\mid X)\right)$	$\displaystyle=\mathbb{E}\left[\sigma(-t_{m}Z)^{2}\varphi_{m}(t^{\star}Z)XX^{\top}+\sigma(t_{m}Z)^{2}(1-\varphi_{m}(t^{\star}Z))XX^{\top}\right]$
		$\displaystyle=\mathbb{E}\left[\left(\sigma(-t_{m}Z)^{2}\varphi_{m}(t^{\star}Z)+\sigma(t_{m}Z)^{2}(1-\varphi_{m}(t^{\star}Z))\right)Z^{2}\right]u^{\star}{u^{\star}}^{\top}$
		$\displaystyle\qquad~{}+\mathbb{E}\left[\sigma(-t_{m}Z)^{2}\varphi_{m}(t^{\star}Z)+\sigma(t_{m}Z)^{2}(1-\varphi_{m}(t^{\star}Z))\right]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.$

Applying Lemma A.1 as in the proof of Theorem 1 to decompose the covariance terms in the asymptotic (23), and substituting in $\rho_{m}(t)=\sigma(t)\mathds{1}\{t\geq 0\}+(1-\sigma(t))\mathds{1}\{t<0\}$ , the limiting covariance in expression (23) becomes

	$\displaystyle\frac{\mathbb{E}[(\sigma(-t_{m}\|Z\|)^{2}\rho_{m}(t^{\star}Z)+\sigma(t_{m}\|Z\|)^{2}(1-\rho_{m}(t^{\star}Z)))Z^{2}]}{\mathbb{E}[\mathsf{he}(Z)Z^{2}]^{2}}u^{\star}{u^{\star}}^{\top}$
	$\displaystyle\qquad~{}+\frac{\mathbb{E}[\sigma(-t_{m}\|Z\|)^{2}\rho_{m}(t^{\star}Z)+\sigma(t_{m}\|Z\|)^{2}(1-\rho_{m}(t^{\star}Z))]}{\mathbb{E}[\mathsf{he}(Z)]^{2}}\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}.$

Lastly, we apply the delta method to $\phi(x)=x/\left\|{x}\right\|_{2}$ , exactly as in the proof of Theorem 1, which gives the theorem.

Appendix E Proofs of asymptotic normality

In this appendix, we include proofs of the convergence results in Propositions 1, 2, and 3. In each, we divide the proof into three steps: we characterize the loss minimizer, apply one of the master Theorems 1 or 2 to obtain asymptotic normality, and then characterize the behavior of the asymptotic covariance as $m\to\infty$ .

E.1 Proof of Proposition 1

Asymptotic normality of the MLE.

The asymptotic normality result is an immediate consequence of the classical asymptotics for maximum likelihood estimators [43, Thm. 5.29].

Normalized estimator.

For the normalized estimator, we appeal to the master results developed in Section 3.1. In particular, since we are in the well-specified logistic model, we can invoke Corollary 3 and write directly that

\sqrt{n}(\widehat{u}_{n,m}^{\textup{lr}}-u^{\star})\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\frac{1}{m}\cdot\frac{1}{{t^{\star}}^{2}}\frac{\mathbb{E}[\sigma^{\textup{lr}}(t^{\star}Z)(1-\sigma^{\textup{lr}}(t^{\star}Z))]}{\mathbb{E}[{\sigma^{\textup{lr}}}^{\prime}(t^{\star}Z)]^{2}}\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}\right),

which immediately implies

\displaystyle C(t)

\displaystyle=\frac{1}{t^{2}}\frac{\mathbb{E}[\sigma^{\textup{lr}}(tZ)(1-\sigma^{\textup{lr}}(tZ))]}{\mathbb{E}[{\sigma^{\textup{lr}}}^{\prime}(tZ)]^{2}}=\frac{1}{t^{2}\mathbb{E}\left[\frac{e^{tZ}}{(1+e^{tZ})^{2}}\right]},

and that further $C(t){t}^{2-\beta}={t}^{-\beta}\mathbb{E}[\frac{e^{t|Z|}}{(1+e^{t|Z|})^{2}}]^{-1}$ . To compute the limit when $t\to\infty$ , we invoke Lemma A.2 and we conclude that

\displaystyle\lim_{t\to\infty}C(t)t^{2-\beta}=\lim_{t\to\infty}\frac{1}{t^{\beta}\mathbb{E}[\frac{e^{t|Z|}}{(1+e^{t|Z|})^{2}}]}=\frac{1}{c_{Z}\int_{0}^{\infty}\frac{z^{\beta-1}e^{z}}{(1+e^{z})^{2}}dz}.

E.2 Proof of Proposition 2

Minimizer of the population loss.

We can see identity (6) still holds with the calibration gap $h(t)=h_{m}(t)=\mathbb{E}[|Z|(1-\rho_{m}(t^{\star}|Z|))]-\mathbb{E}[\frac{|Z|}{1+e^{t|Z|}}]$ in Eq. (5) as $X-u^{\star}Z$ and $u^{\star}Z$ are independent. The function $h(t)$ is monotonically increasing in $t$ with $h(\infty)=\mathbb{E}\left[{|Z|(1-\rho_{m}(t^{\star}|Z|))}\right]>0$ , while $1-\rho_{m}(t|Z|)\leq 1-\rho_{1}(t|Z|)=\frac{1}{1+e^{t|Z|}}$ , we must have $h(t^{\star})\leq 0$ . Therefore there must be a unique zero point $t_{m}\geq t^{\star}$ of $h(t)$ , and so $t_{m}u^{\star}$ is the unique minimizer of the population loss $L^{\textup{mv}}_{m}(\theta)$ .

Asymptotic variance.

As $t_{m}$ solves $h_{m}(t_{m})=0$ , Eq. (6) guarantees that $t_{m}u^{\star}$ is the global minimizer of the population loss $L^{\textup{mv}}_{m}$ . Appealing to Theorem 2, it follows that $\widehat{\theta}^{\textup{mv}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}t_{m}u^{\star}$ , and

\displaystyle\sqrt{n}(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star})

\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,C_{m}(t^{\star})\left({\mathsf{P}_{u^{\star}}\Sigma\mathsf{P}_{u^{\star}}}\right)^{\dagger}\right)

for the variance function (13), which in this case simplifies to

\displaystyle C_{m}(t^{\star})

\displaystyle=\frac{\mathbb{E}[\frac{1}{(1+e^{t_{m}|Z|})^{2}}\rho_{m}(t^{\star}|Z|)+\frac{1}{(1+e^{-t_{m}|Z|})^{2}}(1-\rho_{m}(t^{\star}|Z|))]}{t_{m}^{2}\mathbb{E}[\frac{e^{t_{m}Z}}{(1+e^{t_{m}Z})^{2}}]^{2}}

via the symmetry $\rho_{m}(t)=\rho_{m}(-t)$ .

Large $m$ behavior.

The remainder of the proof is to characterize the behavior of $C_{m}(t^{\star})$ as $m\to\infty$ . We first derive asymptotics for $t_{m}$ . To simplify notation, we let $\left\|{\theta}\right\|_{2}=t=t^{\star}$ . Because $t_{m}$ solves $h_{m}(t_{m})=0$ we have

\displaystyle\mathbb{E}\left[{\frac{|Z|}{1+e^{t_{m}|Z|}}}\right]=\mathbb{E}\left[{|Z|(1-\rho_{m}(t|Z|))}\right],

(24)

we must have $t_{m}\to\infty$ as $m\to\infty$ because $\rho_{m}(t)\to 1$ for any $t>0$ as $m\to\infty$ , so the right side of equality (24) converges to 0 by the dominated convergence theorem, and hence so must the left hand side. Invoking Lemma A.2 for the left hand side, it follows that

\displaystyle\lim_{m\to\infty}t_{m}^{\beta+1}\mathbb{E}\left[{\frac{|Z|}{1+e^{t_{m}|Z|}}}\right]

\displaystyle=\lim_{m\to\infty}t_{m}^{\beta}\mathbb{E}\left[{\frac{t_{m}|Z|}{1+e^{t_{m}|Z|}}}\right]=c_{Z}\int_{0}^{\infty}\frac{z^{\beta}}{1+e^{z}}dz,

while invoking Lemma A.3 for the right hand side of (24), it follows that

	$\displaystyle\lim_{m\to\infty}m^{\frac{\beta+1}{2}}\mathbb{E}\left[{\|Z\|(1-\rho_{m}(t\|Z\|))}\right]$	$\displaystyle=\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[{\sqrt{m}\|Z\|(1-\rho_{m}(t\|Z\|))}\right]$
		$\displaystyle=c_{Z}\int_{0}^{\infty}z^{\beta}\Phi\left(-\frac{tz}{2}\right)dz=c_{Z}{t}^{-\beta-1}\int_{0}^{\infty}z^{\beta}\Phi\left(-\frac{z}{2}\right)dz,$

where the last line follows from change of variables $tz\mapsto z$ . The identity (24) implies that the ratio $\mathbb{E}[|Z|/(1+e^{t_{m}|Z|})]/\mathbb{E}[|Z|(1-\rho_{m}(t|Z|))]=1$ and so we have that as $m\to\infty$ ,

\displaystyle\frac{t_{m}}{\sqrt{m}}

\displaystyle=\left(\frac{t_{m}^{\beta+1}}{m^{\frac{\beta+1}{2}}}\right)^{\frac{1}{\beta+1}}=\left(\frac{t_{m}^{\beta+1}\mathbb{E}\left[{\frac{|Z|}{1+e^{t_{m}|Z|}}}\right]}{m^{\frac{\beta+1}{2}}\mathbb{E}\left[{|Z|(1-\rho_{m}(t|Z|))}\right]}\right)^{\frac{1}{\beta+1}}\to\left(\frac{\int_{0}^{\infty}\frac{z^{\beta}}{1+e^{z}}dz}{\int_{0}^{\infty}z^{\beta}\Phi\left(-\frac{z}{2}\right)dz}\right)^{\frac{1}{\beta+1}}\cdot t=:at.

In particular, $t_{m}/t^{\star}\sqrt{m}=a(1+o_{m}(1))$ .

We finally proceed to compute asymptotic behavior of $C_{m}(t^{\star})$ , the variance (13). By Lemma A.2 the limit of its denominator as $t_{m}\to\infty$ satisfies

\displaystyle\lim_{m\to\infty}\underbrace{t_{m}^{2\beta}\mathbb{E}\left[\frac{e^{t_{m}Z}}{(1+e^{t_{m}Z})^{2}}\right]^{2}}_{:=\mathsf{den}(C_{m}(t))}

\displaystyle=\lim_{t\to\infty}\left(t^{\beta}\mathbb{E}\left[\frac{e^{tZ}}{(1+e^{tZ})^{2}}\right]\right)^{2}=\left(c_{Z}\int_{0}^{\infty}\frac{z^{\beta-1}e^{z}}{(1+e^{z})^{2}}dz\right)^{2}.

We decompose the numerator into the two parts

	$\displaystyle m^{\frac{\beta}{2}}\mathbb{E}\left[\frac{1}{\left(1+e^{t_{m}\|Z\|}\right)^{2}}\rho_{m}(t\|Z\|)+\frac{1}{\left(1+e^{-t_{m}\|Z\|}\right)^{2}}(1-\rho_{m}(t\|Z\|))\right]$
		$\displaystyle=\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[\left(\frac{1}{\left(1+e^{-t_{m}\|Z\|}\right)^{2}}-\frac{1}{\left(1+e^{t_{m}\|Z\|}\right)^{2}}\right)(1-\rho_{m}(t\|Z\|))\right]}_{\mathrm{(I)}}+\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[\frac{1}{\left(1+e^{t_{m}\|Z\|}\right)^{2}}\right]}_{\mathrm{(II)}}.$

As we have already shown that $m^{-\frac{1}{2}}t_{m}\to at$ , we know for any $\epsilon>0$ that for large enough $m$ , $(1-\epsilon)at\sqrt{m}\leq t_{m}\leq(1+\epsilon)at\sqrt{m}$ . We can thus invoke Lemma A.2 to get

\displaystyle\lim_{m\to\infty}\mathrm{(II)}=\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[\frac{1}{\left(1+e^{\sqrt{m}at|Z|}\right)^{2}}\right]

\displaystyle=c_{Z}\int_{0}^{\infty}\frac{z^{\beta-1}}{\left(1+e^{atz}\right)^{2}}dz=c_{Z}{t}^{-\beta}\int_{0}^{\infty}\frac{z^{\beta-1}}{\left(1+e^{az}\right)^{2}}dz.

With the same argument, we apply Lemma A.3 to establish the convergence

	$\displaystyle\lim_{m\to\infty}\mathrm{(I)}$	$\displaystyle=c_{Z}\int_{0}^{\infty}z^{\beta-1}\left(\frac{1}{\left(1+e^{-atz}\right)^{2}}-\frac{1}{\left(1+e^{atz}\right)^{2}}\right)\Phi\left(-\frac{tz}{2}\right)dz$
		$\displaystyle=c_{Z}\int_{0}^{\infty}z^{\beta-1}\frac{e^{atz}-1}{e^{atz}+1}\Phi\left(-\frac{tz}{2}\right)dz=c_{Z}{t}^{-\beta}\int_{0}^{\infty}z^{\beta-1}\frac{e^{az}-1}{e^{az}+1}\Phi\left(-\frac{z}{2}\right)dz,$

where we use the change of variables $tz\mapsto z$ . Taking limits, we have

	$\displaystyle\lim_{m\to\infty}m^{1-\frac{1}{2}\beta}C_{m}(t)$	$\displaystyle=\lim_{m\to\infty}\frac{m^{\frac{\beta}{2}}\mathbb{E}[\frac{1}{(1+e^{t_{m}\|Z\|})^{2}}\rho_{m}(t\|Z\|)+\frac{1}{(1+e^{-t_{m}\|Z\|})^{2}}(1-\rho_{m}(t\|Z\|))]}{m^{\beta-1}t_{m}^{2-2\beta}\cdot t_{m}^{2\beta}\mathbb{E}[\frac{e^{t_{m}Z}}{(1+e^{t_{m}Z})^{2}}]^{2}}$
		$\displaystyle=\lim_{m\to\infty}\left(\frac{t_{m}}{\sqrt{m}}\right)^{2\beta-2}\cdot\frac{\lim_{m\to\infty}\mathrm{(I)}+\lim_{m\to\infty}\mathrm{(II)}}{\mathsf{den}(C_{m}(t))}$
		$\displaystyle=(at)^{2\beta-2}\cdot\frac{c_{Z}{t}^{-\beta}\int_{0}^{\infty}z^{\beta-1}\left(\frac{1}{\left(1+e^{az}\right)^{2}}+\frac{e^{az}-1}{e^{az}+1}\Phi\left(-\frac{z}{2}\right)\right)dz}{(c_{Z}\int_{0}^{\infty}\frac{z^{\beta-1}e^{z}}{(1+e^{z})^{2}}dz)^{2}},$

where we used that $t_{m}/\sqrt{m}\to at$ as above.

E.3 Proof of Proposition 3

Minimizer of the population loss.

By Lemma 3.1, we know the gap $h(t)=0$ has a unique solution $t_{m}$ , with $t_{m}u^{\star}$ minimizing the population loss $L(\theta,\sigma)$ .

Asymptotic variance.

Directly invoking Theorem 2 yields asymptotic normality:

\sqrt{n}\left(\widehat{u}^{\textup{mv}}_{n,m}-u^{\star}\right)\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left({0,C_{m}(t^{\star})\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}}\right),

where the covariance function (13) has the form

C_{m}(t^{\star})=\frac{1}{t_{m}^{2}}\frac{\mathbb{E}[\sigma(-t_{m}|Z|)^{2}\rho_{m}(t^{\star}Z)+\sigma(t_{m}|Z|)^{2}(1-\rho_{m}(t^{\star}Z))]}{\mathbb{E}[\sigma^{\prime}(t_{m}Z)]^{2}}

(25)

and again $t_{m}$ is the implicitly defined zero of $h(t)=0$ .

Large $m$ behavior.

We derive the large $m$ asymptotics of $t_{m}$ and $C_{m}$ under Assumption A4 and using the shorthand $\left\|{\theta^{\star}}\right\|_{2}=t$ . The proof is essentially identical to that of Proposition 2 in Appendix E.2. First, recalling the probability (15), $\rho_{m}(t)=\mathbb{P}(Y^{+}=\mathsf{sign}(\langle X,\theta^{\star}\rangle)\mid\langle X,\theta^{\star}\rangle=t)$ , we see that $1-\rho_{m}(tz)\to 0$ for any $z\neq 0$ as $m\to\infty$ , and thus by dominated convergence, $\mathbb{E}\left[{|Z|\left({1-\rho_{m}(tZ)}\right)}\right]\to 0$ . The analogue of the identity (24) in the proof of Proposition 2, that $t_{m}$ is the zero of $h(t)=\mathbb{E}[\sigma(t|Z|)|Z|(1-\rho_{m}(t^{\star}Z))]-\mathbb{E}[\sigma(-t|Z|)|Z|\rho_{m}(t^{\star}Z)]$ , implies

\mathbb{E}[\sigma(t_{m}|Z|)|Z|(1-\rho_{m}(t^{\star}Z))]=\mathbb{E}\left[{\sigma(-t_{m}|Z|)|Z|\rho_{m}(t^{\star}Z)}\right].

(26)

As $\sigma$ is bounded and $\mathbb{E}[\sigma(-t|Z|)|Z|\rho_{m}(t^{\star}|Z|)]\to\mathbb{E}[\sigma(-t|Z|)|Z|]$ for any $t$ , the convergence of the left hand side of equality (26) to 0 as $m\to\infty$ means we must have $t_{m}\to\infty$ . Invoking Lemma A.2 yields

\displaystyle\lim_{m\to\infty}t_{m}^{\beta+1}\mathbb{E}\left[{|Z|\sigma(-t_{m}|Z|)}\right]=\lim_{m\to\infty}t_{m}^{\beta}\mathbb{E}\left[{t_{m}|Z|\sigma(-t_{m}|Z|)}\right]=c_{Z}\int_{0}^{\infty}z^{\beta}\sigma(-z)dz.

Applying Lemma A.3 gives

\displaystyle m^{\frac{\beta+1}{2}}\mathbb{E}\left[{|Z|(1-\rho_{m}(tZ))}\right]=m^{\frac{\beta}{2}}\mathbb{E}\left[{\sqrt{m}|Z|(1-\rho_{m}(tZ))}\right]\to c_{Z}{t}^{-\beta-1}\int_{0}^{\infty}z^{\beta}\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)z\right)dz.

Rewriting the identity (26) using the symmetry of $\sigma$ , so that $\sigma(t)+\sigma(-t)=1$ , we have the equivalent statement that $\mathbb{E}[|Z|(1-\rho_{m}(t^{\star}Z))]=\mathbb{E}[\sigma(-t_{m}|Z|)|Z|]$ , or $\mathbb{E}[\sigma(-t_{m}|Z|)|Z|]/\mathbb{E}[|Z|(1-\rho_{m}(t^{\star}Z))]=1$ . Using this identity ratio, we find that

\frac{t_{m}}{\sqrt{m}}=\left(\frac{t_{m}^{\beta+1}\mathbb{E}\left[{|Z|\sigma(-t_{m}|Z|)}\right]}{m^{\frac{\beta+1}{2}}\mathbb{E}\left[{|Z|(1-\rho_{m}(tZ))}\right]}\right)^{\frac{1}{\beta+1}}\to\left(\frac{\int_{0}^{\infty}z^{\beta}\sigma(-z)dz}{\int_{0}^{\infty}z^{\beta}\Phi(-2{\overline{\sigma}^{\star}}^{\prime}(0)z)dz}\right)^{\frac{1}{\beta+1}}\cdot t^{\star}=:at^{\star}.

This concludes the asymptotic characterization that $t_{m}=\sqrt{m}at^{\star}\cdot(1+o_{m}(1))$ .

Finally, we turn to the asymptotics for $C_{m}(t^{\star})$ in (25). By Lemma A.2 its denominator has limit

\displaystyle\lim_{m\to\infty}\underbrace{t_{m}^{2\beta}\mathbb{E}\left[\sigma^{\prime}(t_{m}Z)\right]^{2}}_{:=\mathsf{den}{C_{m}(t^{\star})}}=\lim_{t\to\infty}\left(t^{\beta}\mathbb{E}\left[\sigma^{\prime}(tZ)\right]\right)^{2}=\left(c_{Z}\int_{0}^{\infty}z^{\beta-1}\sigma^{\prime}(z)dz\right)^{2}.

We decompose the (rescaled) numerator of the variance (25) into the two parts

\displaystyle\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[\left(\sigma(t_{m}|Z|)^{2}-\sigma(-t_{m}|Z|)^{2}\right)(1-\rho_{m}(t|Z|))\right]}_{\mathrm{(I)}}+\underbrace{m^{\frac{\beta}{2}}\mathbb{E}\left[\sigma(-t_{m}|Z|)^{2}\right]}_{\mathrm{(II)}}.

Lemmas A.2 and that $t_{m}=a\sqrt{m}t^{\star}(1+o_{m}(1))\to\infty$ , coupled with the dominated convergence theorem, establishes the convergence

\displaystyle\lim_{m\to\infty}\mathrm{(II)}=\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[\sigma(-\sqrt{m}at^{\star}|Z|)^{2}\right]=c_{Z}{t}^{-\beta}\int_{0}^{\infty}\frac{z^{\beta-1}}{\left(1+e^{az}\right)^{2}}dz.

Similarly, Lemma A.3 and that $t_{m}=a\sqrt{m}t^{\star}(1+o_{m}(1))$ gives that

	$\displaystyle\lim_{m\to\infty}\mathrm{(I)}$	$\displaystyle=\lim_{m\to\infty}m^{\frac{\beta}{2}}\mathbb{E}\left[\left(\sigma(at^{\star}\sqrt{m}\|Z\|)^{2}-\sigma(-at^{\star}\sqrt{m}\|Z\|)^{2}\right)(1-\rho_{m}(tZ))\right]$
		$\displaystyle=c_{Z}\int_{0}^{\infty}z^{\beta-1}\left(\sigma(at^{\star}z)^{2}-\sigma(-at^{\star}z)^{2}\right)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)tz\right)dz$
		$\displaystyle=c_{Z}{t^{\star}}^{-\beta}\int_{0}^{\infty}z^{\beta-1}\left(\sigma(az)^{2}-\sigma(-az)^{2}\right)\Phi\left(-2{\overline{\sigma}^{\star}}^{\prime}(0)z\right)dz,$

where in the last line we use change of variables $tz\mapsto z$ . Hence we have

	$\displaystyle\lim_{m\to\infty}m^{1-\frac{1}{2}\beta}C_{m}(t^{\star})$	$\displaystyle=\lim_{m\to\infty}\frac{1}{m^{\beta-1}t_{m}^{2-2\beta}}\cdot\frac{\lim_{m\to\infty}\mathrm{(I)}+\lim_{m\to\infty}\mathrm{(II)}}{\mathsf{den}(C_{m}(t^{\star}))}$
		$\displaystyle=\lim_{m\to\infty}\left({\frac{t_{m}}{\sqrt{m}}}\right)^{2\beta-2}\cdot\frac{c_{Z}{t^{\star}}^{-\beta}\int_{0}^{\infty}z^{\beta-1}({\sigma(az)^{2}+(\sigma(az)^{2}-\sigma(-az)^{2})\Phi(-2{\overline{\sigma}^{\star}}^{\prime}(0)z)})dz}{(c_{Z}\int_{0}^{\infty}\frac{z^{\beta-1}e^{z}}{(1+e^{z})^{2}}dz)^{2}}.$

Finally, as $t_{m}/\sqrt{m}=at^{\star}(1+o_{m}(1))$ we obtain that $m^{1-\beta/2}C_{m}(t^{\star})={t^{\star}}^{\beta-2}b$ for some constant $b$ depending only on $\beta,c_{Z},{\overline{\sigma}^{\star}}^{\prime}(0)$ , and $\sigma$ .

Appendix F Proofs of fundamental limits of estimation with aggregate labels

F.1 Proof of Proposition 4

We assume without loss of generality $m$ is odd. When $m$ is even the proof is identical, except that we randomize $Y^{+}$ when we have equal votes for both classes. As the marginal distribution of $X$ is $\mathsf{N}(0,I_{d})$ for both $\mathbb{P}_{(X,Y^{+})}^{\sigma^{\star},\theta^{\star},m}$ and $\mathbb{P}_{(X,Y^{+})}^{\overline{\sigma},\overline{\theta},\overline{m}}$ , we only have to show the existence of a link $\overline{\sigma}\in\mathcal{F}_{\mathsf{link}}^{0}$ such that the conditional distribution on any $X=x$ is the same, i.e.,

\displaystyle\mathbb{P}_{\sigma^{\star},\theta^{\star},m}(Y^{+}=1\mid X=x)=\mathbb{P}_{\overline{\sigma},\overline{\theta},\overline{m}}(Y^{+}=1\mid X=x).

For $m\in\mathbb{N}$ , define the probability that a $\mathsf{Binomial}(m,t)$ is at least $\lceil m/2\rceil$ through the one-to-one transformation

\displaystyle P_{m}(t):=\sum_{i=\lceil m/2\rceil}^{m}\binom{m}{i}t^{m}(1-t)^{m-i},

For any $m$ , $P_{m}(t)$ monotonically increases in $t$ , and $P_{m}(0)=0,P_{m}(\frac{1}{2})=\frac{1}{2},P_{m}(1)=1$ , and by symmetry $P_{m}(t)+P_{m}(1-t)=1$ . By the definition of majority vote

\displaystyle\mathbb{P}_{\sigma^{\star},\theta^{\star},m}(Y^{+}=1\mid X=x)

\displaystyle=\sum_{i=\lceil m/2\rceil}^{m}\binom{m}{i}\sigma^{\star}(\langle\theta^{\star},x\rangle)^{m}(1-\sigma^{\star}(\langle\theta^{\star},x\rangle))^{m-i}=P_{m}\circ\sigma^{\star}(\langle\theta^{\star},x\rangle).

Therefore, the link

\displaystyle\overline{\sigma}(t):=P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\|{\theta}\right\|_{2}}{\left\|{\overline{\theta}}\right\|_{2}}t\right)

satisfies $\overline{\sigma}(0)=\frac{1}{2}$ and $\overline{\sigma}(t)-\frac{1}{2}>0$ for all $t>0$ , since $P_{m}$ maps $(\frac{1}{2},1]$ to $(\frac{1}{2},1]$ . We also have

	$\displaystyle\overline{\sigma}(t)+\overline{\sigma}(-t)$	$\displaystyle=P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(-\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(i)}}}{{=}}P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ P_{m}\circ\left(1-\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)\right)$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(ii)}}}{{=}}P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ\left(1-P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)\right)$
		$\displaystyle=1,$

where (i) and (ii) follow from symmetry of $\sigma^{\star}$ and $P_{m}$ , respectively. Thus $\overline{\sigma}$ belongs to $\mathcal{F}_{\mathsf{link}}^{0}$ and is a valid link function. This $\bar{\sigma}$ yields the desired equality:

\displaystyle\mathbb{P}_{\overline{\sigma},\overline{\theta},\overline{m}}(Y^{+}=1\mid X=x)

\displaystyle=P_{\overline{m}}\circ P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\|{\theta}\right\|_{2}}{\left\|{\overline{\theta}}\right\|_{2}}\cdot\langle\overline{\theta},x\rangle\right)=\mathbb{P}_{\sigma^{\star},\theta^{\star},m}(Y^{+}=1\mid X=x).

F.2 Proof of Theorem 3

Throughout this proof, we use $a_{m}\sim b_{m}$ to mean that $a_{m}/b_{m}\to 1$ as $m\to\infty$ , or equivalantly that $a_{m}=b_{m}\cdot(1+o_{m}(1))$ . By Lemma 4.1, we have

\displaystyle\mathsf{I}_{m}(\theta^{\star})=A_{m}(t^{\star})\cdot u^{\star}{u^{\star}}^{\top}+B_{m}(t^{\star})\cdot\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}

with


$\displaystyle A_{m}(t)$	$\displaystyle:=\mathbb{E}\left[{\frac{\rho_{m}^{\prime}(t\|Z\|)^{2}Z^{2}}{\rho_{m}(t\|Z\|)(1-\rho_{m}(t\|Z\|))}}\right],$	(27a)
$\displaystyle B_{m}(t)$	$\displaystyle:=\mathbb{E}\left[{\frac{\rho_{m}^{\prime}(t\|Z\|)^{2}}{\rho_{m}(t\|Z\|)(1-\rho_{m}(t\|Z\|))}}\right].$	(27b)

It is crucial to understand the behavior of $\rho_{m}$ and $\rho_{m}^{\prime}$ , which have the exact forms

	$\displaystyle\rho_{m}(t)$	$\displaystyle=\sum_{k>m/2}\binom{m}{k}\frac{e^{kt}}{(1+e^{t})^{m}},$
	$\displaystyle\rho_{m}^{\prime}(t)$	$\displaystyle=\sum_{k>m/2}\binom{m}{k}\frac{ke^{kt}(1+e^{t})-me^{(k+1)t}}{(1+e^{t})^{m+1}}.$

Throughout this proof, we assume without loss of generality that $m$ is odd. When $m$ is even, we replace $\binom{m}{\frac{m+1}{2}}$ with $\binom{m}{\frac{m}{2}+1}$ , but otherwise all arguments are identical.

Lemma F.1.

The function $\rho_{m}^{\prime}$ is even, and for any $t>0$ ,

\displaystyle\rho_{m}^{\prime}(t)

\displaystyle=\binom{m}{\frac{m+1}{2}}\cdot\frac{m+1}{2}\frac{e^{\frac{m+1}{2}t}}{(1+e^{t})^{m+1}}.

Proof.

By direct calculations

	$\displaystyle\rho_{m}^{\prime}(t)$	$\displaystyle=\sum_{k>m/2}\binom{m}{k}\frac{ke^{kt}(1+e^{t})-me^{(k+1)t}}{(1+e^{t})^{m+1}}$
		$\displaystyle\stackrel{{\scriptstyle(i)}}{{=}}\sum_{k>m/2}\left\{\binom{m}{k}\frac{ke^{kt}(1+e^{t})}{(1+e^{t})^{m+1}}-\binom{m}{k}\frac{ke^{(k+1)t}}{(1+e^{t})^{m+1}}-\binom{m}{k+1}\frac{(k+1)e^{(k+1)t}}{(1+e^{t})^{m+1}}\right\}$
		$\displaystyle=\frac{1}{(1+e^{t})^{m+1}}\sum_{k>m/2}\left\{\binom{m}{k}ke^{kt}-\binom{m}{k+1}(k+1)e^{(k+1)t}\right\},$

where (i) uses the combinatorial identity

\displaystyle m\binom{m}{k}=k\binom{m}{k}+(k+1)\binom{m}{k+1}.

This yields a telescoping sum and establishes the result. ∎

Next, we define the function

\displaystyle\eta_{m}(t)=\frac{\rho_{m}^{\prime}(t)^{2}}{\rho_{m}(t)(1-\rho_{m}(t))}.

We can then write for the Fisher information $\mathsf{I}_{m}(\theta^{\star})=A_{m}(t^{\star})\cdot u^{\star}{u^{\star}}^{\top}+B_{m}(t^{\star})\cdot\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}$ via

\displaystyle A_{m}(t)=\mathbb{E}\left[{\eta_{m}(tZ)Z^{2}}\right],\qquad B_{m}(t)=\mathbb{E}\left[{\eta_{m}(tZ)}\right],

as $\eta_{m}$ is a even function. The next lemma provides the key asymptotic guarantees for $\eta_{m}(t)$ in the above displays. See Appendix F.3 for a proof.

Lemma F.2.

There exists a function $c_{m}(t)$ satisfying the following:

(i)

For any $t>0$ , $c_{m}(t)\geq 1-e^{-t}$ and

$\displaystyle\eta_{m}(t)=\frac{(m+1)^{2}}{4}\binom{m}{\frac{m+1}{2}}\cdot\frac{e^{\frac{m+3}{2}t}}{(1+e^{t})^{m+2}}\cdot c_{m}(t),$
(ii)

For any $\delta\in(0,1)$ , there exist $C=C(\delta)>0$ and $M=M(\delta)$ such that for all $m\geq M$ and all $t>0$ ,

$\displaystyle c_{m}(t)\leq\frac{C\cdot(1-e^{-t})}{1-e^{-(\lfloor\delta\sqrt{m}\rfloor+1)t}}.$

(iii)

For any fixed $w>0$ ,

\displaystyle\lim_{m\to\infty}\sqrt{m}c_{m}\left({\frac{w}{\sqrt{m}}}\right)=e^{-\frac{w^{2}}{8}}\sqrt{\frac{2}{\pi}}\cdot\frac{1}{\Phi(-w/2)\Phi(w/2)}.

We use the asymptotics in Lemma F.2 to compute $A_{m}(t)$ and $B_{m}(t)$ , respectively.

Part I: Asymptotics for $B_{m}(t)$ .

We begin by showing the claimed asymptotic formula for $B_{m}(t)$ in the theorem’s statement. By Stirling’s formula, the multiplicative factor in $\eta_{m}$ satisfies

\displaystyle\frac{(m+1)^{2}}{4}\binom{m}{\frac{m+1}{2}}\sim\frac{m^{2}}{4}\cdot 2^{m}\sqrt{\frac{2}{\pi m}},

and thus

	$\displaystyle B_{m}(t)$	$\displaystyle\sim\frac{m^{2}}{2\sqrt{2\pi m}}\mathbb{E}\left[{2^{m}\frac{e^{\frac{m+3}{2}tZ}}{(1+e^{tZ})^{m+2}}\cdot c_{m}(tZ)}\right]=\frac{m^{2}}{8\sqrt{2\pi m}}\mathbb{E}\left[{2^{m+2}\frac{e^{\frac{m+3}{2}tZ}}{(1+e^{tZ})^{m+2}}\cdot c_{m}(tZ)}\right]$
		$\displaystyle=\frac{m^{2}}{8\sqrt{2\pi m}}\mathbb{E}\left[{\left({\frac{2e^{\frac{tZ}{2}}}{1+e^{tZ}}}\right)^{m+2}\cdot e^{\frac{tZ}{2}}c_{m}(tZ)}\right]\sim\frac{\sqrt{m}}{8\sqrt{2\pi}}\cdot\underbrace{(m+2)\mathbb{E}\left[{\left({\frac{2e^{\frac{tZ}{2}}}{1+e^{tZ}}}\right)^{m+2}\cdot e^{\frac{tZ}{2}}c_{m}(tZ)}\right]}_{=:J_{m+2}(t)}.$

Our goal now is to show that

\lim_{m\to\infty}m^{\frac{\beta-1}{2}}J_{m}(t)=\frac{c_{Z}}{t^{\beta}}\sqrt{\frac{8}{\pi}}\cdot\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta-1}}{\Phi(-z)\Phi(z)}dz,

(28)

which then immediately implies the asymptotic $B_{m}(t)\sim bm(t\sqrt{m})^{-\beta}$ for $b=\frac{c_{Z}}{4\pi}\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta-1}}{\Phi(-z)\Phi(z)}dz$ , as claimed in the theorem.

The proof of the limit (28) involves an argument via dominated convergence. By the change of variables $w:=\sqrt{m}u$ ,

	$\displaystyle J_{m}(t)$	$\displaystyle=\int_{0}^{\infty}m\left({\frac{2e^{\frac{tu}{2}}}{1+e^{tu}}}\right)^{m}e^{\frac{tu}{2}}c_{m-2}(tu)p(u)du$
		$\displaystyle=\int_{0}^{\infty}\sqrt{m}\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)p\left({\frac{w}{\sqrt{m}}}\right)dw$
		$\displaystyle=\left({\frac{1}{\sqrt{m}}}\right)^{\beta-1}\int_{0}^{\infty}\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta-1}\cdot\sqrt{m}\left({\frac{w}{\sqrt{m}}}\right)^{1-\beta}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)p\left({\frac{w}{\sqrt{m}}}\right)dw.$

We now demonstrate dominating functions for the various terms in the above integrand. To bound the lsat several terms, we use Assumption A2 that $\kappa:=\sup_{z\in(0,\infty)}z^{1-\beta}p(z)<\infty$ and Lemma F.2, which combine to give the upper bound

	$\displaystyle\sqrt{m}\left({\frac{w}{\sqrt{m}}}\right)^{1-\beta}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)p\left({\frac{w}{\sqrt{m}}}\right)$	$\displaystyle\leq\kappa\sqrt{m}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)$
		$\displaystyle\leq\frac{\kappa c\sqrt{m}\left({1-e^{-\frac{tw}{\sqrt{m}}}}\right)}{1-e^{-(\lfloor\delta\sqrt{m-2}\rfloor+1)\frac{tw}{\sqrt{m}}}}\leq c^{\prime}\sup_{w\in(0,\infty)}\frac{tw}{1-e^{-\delta tw}}<\infty$

for some $c^{\prime}=c(\delta,\kappa)$ and all sufficiently large $m$ . The next lemma controls the first terms in the integral above. (We defer its proof to Appendix F.4.)

Lemma F.3.

There exists a universal constant $0<C$ such that for all $m$ ,

\displaystyle\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}

\displaystyle\leq\frac{1}{C}w^{\beta}\exp\left\{{-C\min\{t^{2}w^{2},\sqrt{tw}\}}\right\}.

Certainly the right hand side term in Lemma F.3 is integrable on $(0,\infty)$ . Rewriting $J_{m}(t)$ , we can therefore invoke dominated convergence to exchange the limit in $m$ and integral to obtain

	$\displaystyle\lim_{m\to\infty}m^{\frac{\beta-1}{2}}J_{m}(t)$	$\displaystyle=\int_{0}^{\infty}\lim_{m\to\infty}\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta-1}\cdot\sqrt{m}\left({\frac{w}{\sqrt{m}}}\right)^{1-\beta}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)p\left({\frac{w}{\sqrt{m}}}\right)dw$
		$\displaystyle\stackrel{{\scriptstyle(i)}}{{=}}c_{Z}\int_{0}^{\infty}\lim_{m\to\infty}w^{\beta-1}\exp\left\{{-\frac{t^{2}w^{2}}{2}\cdot\left({\frac{\sqrt{m}}{tw}\cdot\left({1-e^{-\frac{tw}{2\sqrt{m}}}}\right)}\right)^{2}}\right\}\cdot\lim_{m\to\infty}\sqrt{m}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)dw$
		$\displaystyle\stackrel{{\scriptstyle(ii)}}{{=}}c_{Z}\int_{0}^{\infty}e^{-\frac{1}{8}t^{2}w^{2}}\cdot e^{-\frac{t^{2}w^{2}}{8}}\sqrt{\frac{2}{\pi}}\cdot\frac{w^{\beta-1}}{\Phi(-tw/2)\Phi(tw/2)}dw$
		$\displaystyle=\frac{c_{Z}}{t^{\beta}}\sqrt{\frac{8}{\pi}}\cdot\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta-1}}{\Phi(-z)\Phi(z)}dz,$

where in equality $(i)$ we invoke Assumption A2 and in $(ii)$ we use Lemma F.2. This gives the desired limit (28).

Part II: Asymptotics for $A_{m}(t)$ .

The calculations are similar to those we have done to approximate $B_{m}(t)$ . In this case

\displaystyle A_{m}(t)\sim\frac{1}{8\sqrt{2\pi m}}\cdot\underbrace{(m+2)^{2}\mathbb{E}\left[{\left({\frac{2e^{\frac{tZ}{2}}}{1+e^{tZ}}}\right)^{m+2}\cdot e^{\frac{tZ}{2}}c_{m}(tZ)Z^{2}}\right]}_{=:I_{m+2}(t)},

and again invoking dominated convergence,

	$\displaystyle\lim_{m\to\infty}m^{\frac{\beta-1}{2}}I_{m}(t)$	$\displaystyle=\int_{0}^{\infty}\lim_{m\to\infty}\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta-1}\cdot\sqrt{m}\left({\frac{w}{\sqrt{m}}}\right)^{1-\beta}c_{m-2}\left({\frac{tw}{\sqrt{m}}}\right)p\left({\frac{w}{\sqrt{m}}}\right)w^{2}dw$
		$\displaystyle=c_{Z}\int_{0}^{\infty}e^{-\frac{1}{8}t^{2}w^{2}}\cdot e^{-\frac{t^{2}w^{2}}{8}}\sqrt{\frac{2}{\pi}}\cdot\frac{w^{\beta+1}}{\Phi(-tw/2)\Phi(tw/2)}dw$
		$\displaystyle=\frac{8c_{Z}}{t^{\beta+2}}\sqrt{\frac{2}{\pi}}\cdot\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta+1}}{\Phi(-z)\Phi(z)}dz.$

Hence

\displaystyle A_{m}(t)=\frac{a}{t^{2}}\left({\frac{1}{t\sqrt{m}}}\right)^{\beta}(1+o_{m}(1)),\qquad a=\frac{c_{Z}}{\pi}\int_{0}^{\infty}\frac{e^{-z^{2}}z^{\beta+1}}{\Phi(-z)\Phi(z)}dz.

F.3 Proof of Lemma F.2

We begin by proving part (i). Define

s_{m}(t)=\sum_{k<m/2}\binom{m}{k}e^{kt}\leq\frac{1}{2}(1+e^{t})^{m},

where the inequality is valid for $t\geq 0$ . Invoking Lemma F.1 yields

	$\displaystyle\eta_{m}(t)$	$\displaystyle=\frac{\rho_{m}^{\prime}(t)^{2}}{\rho_{m}(t)(1-\rho_{m}(t))}=\frac{\frac{(m+1)^{2}}{4}\binom{m}{\frac{m+1}{2}}^{2}\cdot\frac{e^{(m+1)t}}{(1+e^{t})^{2m+2}}}{\left({\sum_{k>m/2}\binom{m}{k}e^{kt}}\right)\left({\sum_{k<m/2}\binom{m}{k}e^{kt}}\right)\cdot\frac{1}{(1+e^{t})^{2m}}}$
		$\displaystyle=\frac{(m+1)^{2}}{4}\binom{m}{\frac{m+1}{2}}^{2}\cdot\frac{e^{(m+1)t}}{(1+e^{t})^{m+2}}\cdot\frac{1}{(1-s_{m}(t)/(1+e^{t})^{m})s_{m}(t)}$
		$\displaystyle=\frac{(m+1)^{2}}{4}\binom{m}{\frac{m+1}{2}}\cdot\frac{e^{\frac{m+3}{2}t}}{(1+e^{t})^{m+2}}\cdot\underbrace{\frac{\binom{m}{\frac{m+1}{2}}e^{\frac{m-1}{2}t}}{(1-s_{m}(t)/(1+e^{t})^{m})s_{m}(t)}}_{:=c_{m}(t)},$

which gives the equality in part (i). It then remains to show $c_{m}(t)\geq 1-e^{-t}$ . For this, we upper bound

\displaystyle e^{-\frac{m-1}{2}t}s_{m}(t)\leq\sum_{k<m/2}\binom{m}{\frac{m+1}{2}}e^{-\left({\frac{m-1}{2}-k}\right)t}\leq\binom{m}{\frac{m+1}{2}}\frac{1}{1-e^{-t}},

and noting the inequality $\frac{1}{2}\leq 1-\frac{s_{m}(t)}{(1+e^{t})^{m}}\leq 1$ , it then follows that

\displaystyle c_{m}(t)\geq\frac{\binom{m}{\frac{m+1}{2}}e^{\frac{m-1}{2}t}}{s_{m}(t)}\geq 1-e^{-t}.

Proceeding to the proof of part (ii), for any $\delta\in(0,1)$ ,

	$\displaystyle\binom{m}{\frac{m+1}{2}}^{-1}e^{-\frac{m-1}{2}t}s_{m}(t)$	$\displaystyle=\sum_{l=0}^{\infty}\binom{m}{\frac{m+1}{2}}^{-1}\binom{m}{\frac{m+1}{2}-l}e^{-lt}\geq\underbrace{\binom{m}{\frac{m+1}{2}}^{-1}\binom{m}{\frac{m+1}{2}-\lfloor\delta\sqrt{m}\rfloor}}_{:=q_{m}(\delta)}\sum_{l=0}^{\lfloor\delta\sqrt{m}\rfloor}e^{-lt}$
		$\displaystyle=q_{m}(\delta)\cdot\frac{1-e^{-(\lfloor\delta\sqrt{m}\rfloor+1)t}}{1-e^{-t}},$

where the inequality uses that $\binom{m}{\frac{m+1}{2}-l}$ is decreasing in $l$ . We next establish that $q_{m}(\delta)\sim e^{-2\delta^{2}}$ . Indeed, applying Stirling’s formula yields

	$\displaystyle q_{m}(\delta)$	$\displaystyle=\frac{\left({\frac{m}{2}}\right)^{\frac{m}{2}}\cdot\left({\frac{m}{2}}\right)^{\frac{m}{2}}}{m^{m}}\cdot\frac{m^{m}}{\left({\frac{m}{2}-\delta\sqrt{m}}\right)^{\frac{m}{2}-\delta\sqrt{m}}\cdot\left({\frac{m}{2}+\delta\sqrt{m}}\right)^{\frac{m}{2}+\delta\sqrt{m}}}\cdot(1+o_{m}(1))$
		$\displaystyle=\left({1-\frac{2\delta}{\sqrt{m}}}\right)^{-\frac{m}{2}+\delta\sqrt{m}}\cdot\left({1+\frac{2\delta}{\sqrt{m}}}\right)^{-\frac{m}{2}-\delta\sqrt{m}}\cdot(1+o_{m}(1))$
		$\displaystyle=\left({1-\frac{4\delta^{2}}{m}}\right)^{-\frac{m}{2}}\cdot\left({1-\frac{2\delta}{\sqrt{m}}}\right)^{\delta\sqrt{m}}\cdot\left({1+\frac{2\delta}{\sqrt{m}}}\right)^{-\delta\sqrt{m}}\cdot(1+o_{m}(1))$
		$\displaystyle=e^{2\delta^{2}}\cdot e^{-2\delta^{2}}\cdot e^{-2\delta^{2}}\cdot(1+o_{m}(1))=e^{-2\delta^{2}}\cdot(1+o_{m}(1)).$

Substituting the above displays into the definition $c_{m}(t)=\binom{m}{\frac{m+1}{2}}e^{\frac{m-1}{2}t}s_{m}(t)^{-1}(1-\frac{s_{m}(t)}{(1+e^{t})^{m}})^{-1}$ , we can then establish the upper bound using $1-s_{m}(t)/(1+e^{t})^{m}\geq\frac{1}{2}$ , so

\displaystyle c_{m}(t)\leq\frac{2\binom{m}{\frac{m+1}{2}}e^{\frac{m-1}{2}t}}{s_{m}(t)}\leq\frac{2(1-e^{-t})}{q_{m}(\delta)\cdot(1-e^{-(\lfloor\delta\sqrt{m}\rfloor+1)t})}.

Since $q_{m}(\delta)\sim e^{-2\delta^{2}}$ , this inequality establishes part (ii).

For part (iii), we compute $\lim_{m\to\infty}\sqrt{m}c_{m}(w/\sqrt{m})$ . For any fixed $\delta>0$ , we explicitly expand $s_{m}(w/\sqrt{m})$ to obtain

	$\displaystyle\binom{m}{\frac{m+1}{2}}^{-1}e^{-\frac{m-1}{2}\cdot\frac{w}{\sqrt{m}}}s_{m}(w/\sqrt{m})$	$\displaystyle=\sum_{l=0}^{\infty}\binom{m}{\frac{m+1}{2}}^{-1}\binom{m}{\frac{m+1}{2}-l}e^{-lw/\sqrt{m}}$
		$\displaystyle=\sqrt{m}\sum_{k=0}^{\infty}\underbrace{\frac{1}{\sqrt{m}}\sum_{\lfloor k\delta\sqrt{m}\rfloor\leq l<\lfloor(k+1)\delta\sqrt{m}\rfloor}\binom{m}{\frac{m+1}{2}}^{-1}\binom{m}{\frac{m+1}{2}-l}e^{-lw/\sqrt{m}}}_{:=a_{m,k}(\delta)}.$

Here, we partition the set of all nonnegative integers $\{l\in\mathbb{N}\mid 0\leq l<\infty\}$ into sets $L_{k}=\{\lfloor k\delta\sqrt{m}\rfloor\leq l<\lfloor(k+1)\delta\sqrt{m}\rfloor\}$ for $k=0,1,\ldots$ . Using the same asymptotics derived above for $q_{m}(\delta)$ and that $e^{-(k+1)\delta w}\leq e^{-lw/\sqrt{m}}\leq e^{-k\delta w}$ for all $l\in L_{k}$ , it holds for some remainder term $r_{m,k}$ satisfying $|r_{m,k}(\delta)|\leq 1-e^{-\delta w}$ that

	$\displaystyle a_{m,k}(\delta)$	$\displaystyle=\delta\cdot\frac{1}{\delta\sqrt{m}}\sum_{\lfloor k\delta\sqrt{m}\rfloor\leq l<\lfloor(k+1)\delta\sqrt{m}\rfloor}e^{-2k^{2}\delta^{2}}\cdot(1+o_{m}(1))\cdot e^{-lw/\sqrt{m}}$
		$\displaystyle=\delta\cdot\frac{1+r_{m,k}(\delta)}{\delta\sqrt{m}}\sum_{\lfloor k\delta\sqrt{m}\rfloor\leq l<\lfloor(k+1)\delta\sqrt{m}\rfloor}e^{-2k^{2}\delta^{2}}\cdot(1+o_{m}(1))\cdot e^{-k\delta w}$
		$\displaystyle=\left({1+r_{m,k}(\delta)}\right)\cdot\delta e^{-2k^{2}\delta^{2}-k\delta w}\cdot(1+o_{m}(1)).$

We will invoke dominated convergence for series to allow us to exchange summation and limits in $m$ . We observe the following termwise domination:

	$\displaystyle a_{m,k}(\delta)$	$\displaystyle\leq\frac{1}{\sqrt{m}}\cdot\left({\lfloor(k+1)\delta\sqrt{m}\rfloor-\lfloor k\delta\sqrt{m}\rfloor}\right)e^{-\lfloor k\delta\sqrt{m}\rfloor w/\sqrt{m}}$
		$\displaystyle\leq\frac{\delta\sqrt{m}+1}{\sqrt{m}}e^{-k\delta w+w/\sqrt{m}}\leq(\delta+1)e^{-k\delta w+w},$

which is summable in $k$ . We can then invoke dominated convergence theorem for infinite series to obtain that for some $|r(\delta)|\leq 1-e^{-\delta w}$ ,

\displaystyle\lim_{m\to\infty}\sum_{k=0}^{\infty}a_{m,k}(\delta)=(1+r(\delta))\cdot\sum_{k=0}^{\infty}\delta e^{-2k^{2}\delta^{2}-k\delta w},

implying that for any $\delta>0$ ,

\displaystyle\binom{m}{\frac{m+1}{2}}^{-1}e^{-\frac{m-1}{2}\cdot\frac{w}{\sqrt{m}}}s_{m}(w/\sqrt{m})\sim\sqrt{m}\cdot(1+r(\delta))\cdot\sum_{k=0}^{\infty}\delta e^{-2k^{2}\delta^{2}-k\delta w}.

Since the left hand side does not depend on $\delta$ , we can then send $\delta$ to zero on the right hand side and use the definition of Riemann integral to obtain

	$\displaystyle\binom{m}{\frac{m+1}{2}}^{-1}e^{-\frac{m-1}{2}\cdot\frac{w}{\sqrt{m}}}s_{m}(w/\sqrt{m})$	$\displaystyle\sim\sqrt{m}\lim_{\delta\to 0}\sum_{k=0}^{\infty}\delta e^{-2k^{2}\delta^{2}-k\delta w}$
		$\displaystyle=\sqrt{m}\int_{0}^{\infty}e^{-2x^{2}-wx}dx=\sqrt{\frac{\pi m}{2}}e^{\frac{w^{2}}{8}}\Phi(-w/2).$

Now we compute the limit of the remaining term in $c_{m}(t)$ ,

	$\displaystyle s_{m}(w/\sqrt{m})/(1+e^{w/\sqrt{m}})^{m}$	$\displaystyle\sim\left({\frac{1}{1+e^{w/\sqrt{m}}}}\right)^{m}\cdot\binom{m}{\frac{m+1}{2}}e^{-w\sqrt{m}/2}\cdot\sqrt{\frac{\pi m}{2}}e^{\frac{w^{2}}{8}}\Phi(-w/2)$
		$\displaystyle\sim\left({\frac{2}{1+e^{w/\sqrt{m}}}}\right)^{m}\cdot e^{-w\sqrt{m}/2}e^{\frac{w^{2}}{8}}\Phi(-w/2)$
		$\displaystyle=\left({\frac{2e^{-\frac{w}{2\sqrt{m}}}}{1+e^{\frac{w}{\sqrt{m}}}}}\right)^{m}\cdot e^{\frac{w^{2}}{8}}\Phi(-w/2)=\left({1-\frac{\left({1-e^{-\frac{w}{2\sqrt{m}}}}\right)^{2}}{1+e^{\frac{w}{\sqrt{m}}}}}\right)^{m}\cdot e^{\frac{w^{2}}{8}}\Phi(-w/2)$
		$\displaystyle\sim\left({1-\frac{1}{8}\frac{w^{2}}{m}}\right)^{m}\cdot e^{\frac{w^{2}}{8}}\Phi(-w/2)\sim\Phi(-w/2).$

Taking the above displays collectively into

\displaystyle c_{m}(w/\sqrt{m})=\frac{\binom{m}{\frac{m+1}{2}}e^{\frac{m-1}{2}\cdot\frac{w}{\sqrt{m}}}}{s_{m}(w/\sqrt{m})}\cdot\frac{1}{1-s_{m}(w/\sqrt{m})/(1+e^{w/\sqrt{m}})^{m}}

we establish part (iii) of the lemma, that is,

\displaystyle\lim_{m\to\infty}\sqrt{m}c_{m}\left({\frac{w}{\sqrt{m}}}\right)=\lim_{m\to\infty}\frac{\sqrt{m}}{\sqrt{\frac{\pi m}{2}}e^{\frac{w^{2}}{8}}\Phi(-w/2)}\cdot\frac{1}{1-\Phi(-w/2)}=e^{-\frac{w^{2}}{8}}\sqrt{\frac{2}{\pi}}\cdot\frac{1}{\Phi(-w/2)\Phi(w/2)}.

F.4 Proof of Lemma F.3

We pick some constant $C>0$ . When $w\leq C\sqrt{m}/t$ ,

	$\displaystyle\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}$
	$\displaystyle=\left({1-\frac{\left({1-e^{-\frac{tw}{2\sqrt{m}}}}\right)^{2}}{1+e^{-\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}\leq e^{C}w^{\beta}\exp\left\{{-\frac{m\left({1-e^{-\frac{tw}{2\sqrt{m}}}}\right)^{2}}{1+e^{-\frac{tw}{\sqrt{m}}}}}\right\}$
	$\displaystyle\leq e^{C}w^{\beta}\exp\left\{{-\frac{1}{2}\cdot m\left({1-e^{-\frac{tw}{2\sqrt{m}}}}\right)^{2}}\right\}=e^{C}w^{\beta}\exp\left\{{-\frac{t^{2}w^{2}}{2}\cdot\left({\frac{\sqrt{m}}{tw}\cdot\left({1-e^{-\frac{tw}{2\sqrt{m}}}}\right)}\right)^{2}}\right\}$
	$\displaystyle\leq e^{C}w^{\beta}\cdot\exp\left\{-\frac{t^{2}w^{2}}{2}\cdot\left({\inf_{z\in(0,C]}\frac{1-e^{-z/2}}{z}}\right)^{2}\right\},$

and when $w>C\sqrt{m}/t$ ,

	$\displaystyle\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}$	$\displaystyle=\left({\frac{2e^{-\frac{tw}{2\sqrt{m}}}}{1+e^{-\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}$
		$\displaystyle\leq\left({2e^{-\frac{tw}{4\sqrt{m}}}}\right)^{m}\cdot\exp\left\{{-\frac{\sqrt{tw}}{4\sqrt{m}}\cdot\sqrt{tw}}\right\}e^{\frac{tw}{\sqrt{m}}}w^{\beta}$
		$\displaystyle\leq\left({2e^{-\frac{tw}{4\sqrt{m}}+\frac{tw}{m\sqrt{m}}}}\right)^{m}\cdot w^{\beta}\exp\left\{{-\frac{C}{4}\sqrt{tw}}\right\}$
		$\displaystyle\leq\left({2e^{-C\left({\frac{1}{4}-\frac{1}{m}}\right)}}\right)^{m}w^{\beta}\exp\left\{{-\frac{C}{4}\sqrt{tw}}\right\}.$

Then for $m\geq 8$ and $C>8>\log 2$ , it holds $2e^{-C\left({\frac{1}{4}-\frac{1}{m}}\right)}<1$ and consequently

\displaystyle\left({\frac{2e^{\frac{tw}{2\sqrt{m}}}}{1+e^{\frac{tw}{\sqrt{m}}}}}\right)^{m}e^{\frac{tw}{\sqrt{m}}}w^{\beta}\leq w^{\beta}\exp\left\{{-\frac{C}{4}\sqrt{tw}}\right\}.

F.5 Proof of Corollary 6

As we have the explicit formula for the density $p_{\theta}(X,Y^{+})=\mathbb{P}(Y^{+}\mid X)q(X)$ , with $\mathbb{P}(Y^{+}=\mathsf{sign}(\langle x,\theta^{\star}\rangle)\mid X=x)=\rho_{m}(|\langle x,\theta^{\star}\rangle|)$ and $q$ being the $d$ -dimensional centered density with covariance $\Sigma$ . It is differentiable in quadratic mean and allows us to invoke van der Vaart [43, Thm. 8.9]. It only remains to show the asymptotic rate in $m$ for $\Sigma_{\theta^{\star}}$ and $\Sigma_{u^{\star}}$ . Indeed, by Lemma A.1,

\displaystyle\Sigma_{\theta^{\star}}=\mathsf{I}_{m}(\theta^{\star})^{-1}=A_{m}(t^{\star})^{-1}\cdot u^{\star}{u^{\star}}^{\top}+B_{m}(t^{\star})^{-1}\cdot\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger},

and Theorem 3 implies

\displaystyle\Sigma_{\theta^{\star}}\asymp{t^{\star}}^{\beta+2}m^{\frac{\beta}{2}}\cdot u^{\star}{u^{\star}}^{\top}+{t^{\star}}^{\beta}m^{\frac{\beta-2}{2}}\cdot\left({\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}}\right)^{\dagger}.

Analogously we establish the proof for $\Sigma_{u^{\star}}$ .

Appendix G Proof of Theorem 4

We divide the theorem into two main parts, consistent with the typical division of asymptotic normality results into a consistency result and a distributional result. Recall the notation $L(\theta,\vec{\sigma})=\mathbb{E}[\ell_{\vec{\sigma},\theta}(Y\mid X)]$ and $\left\|{\vec{\sigma}-\vec{g}}\right\|_{L^{2}(\mathbb{P})}^{2}=\mathbb{E}[\left\|{\vec{\sigma}(Z)-\vec{g}(Z)}\right\|_{2}^{2}]$ , where $Z$ has the distribution that Assumption A1 specifies.

G.1 Proof of consistency

We demonstrate the consistency $\widehat{\theta}^{\textup{sp}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}u^{\star}$ in three parts, which we present as Lemmas G.1, G.2, and G.4. The first presents an analogue of Lemma 3.1 generalized to the case in which there are $m$ distinct link functions, allowing us to characterize the link-dependent minimizers

\theta^{\star}_{\vec{\sigma}}\coloneqq\operatorname*{argmin}_{\theta}L(\theta,\vec{\sigma})

via a one-dimensional scalar on the line $\{tu^{\star}\mid t\in\mathbb{R}_{+}\}$ . The second, Lemma G.2, then shows that $\theta^{\star}_{\vec{\sigma}}\to u^{\star}$ as $\vec{\sigma}$ approaches $\vec{\sigma}^{\star}$ , where the scaling that $\|{\theta^{\star}_{\vec{\sigma}}}\|_{2}\to 1$ follows by the normalization Assumption A5. Finally, the third lemma demonstrates the probabilistic convergence $\widehat{\theta}^{\textup{sp}}_{n,m}-\theta^{\star}_{\vec{\sigma}_{n}}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ whenever $\vec{\sigma}_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}\vec{\sigma}^{\star}$ in $L^{2}(P)$ .

We begin with the promised analogue of Lemma 3.1.

Lemma G.1.

Define the calibration gap function

h_{\vec{\sigma}}(t)\coloneqq\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}\left[\left(\sigma_{j}(tZ)-\sigma_{j}^{\star}(Z)\right)Z\right].

(29)

Then the loss $L(\theta,\vec{\sigma})$ has unique minimizer $\theta^{\star}_{\vec{\sigma}}=t_{\vec{\sigma}}u^{\star}$ for the unique $t_{\vec{\sigma}}\in(0,\infty)$ solving $h_{\vec{\sigma}}(t)=0$ . Additionally, taking $\mathsf{he}_{j}(t,z)\coloneqq\sigma_{j}^{\prime}(-tz)\sigma_{j}^{\star}(z)+\sigma_{j}^{\prime}(tz)\sigma_{j}^{\star}(-z)$ , we have

\nabla^{2}L(tu^{\star},\vec{\sigma})=\frac{1}{m}\sum_{j=1}^{m}\left(\mathbb{E}[\mathsf{he}_{j}(t,Z)Z^{2}]u^{\star}{u^{\star}}^{\top}+\mathbb{E}[\mathsf{he}_{j}(t,Z)]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}\right)\succ 0.

Proof.

We perform a derivation similar to that we used to derive the gap function (10), with a few modifications to allow collections of $m$ link functions. Note that

\displaystyle L(\theta,\vec{\sigma})

\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[\ell_{\sigma_{j},\theta}(1\mid X)\sigma^{\star}_{j}(\langle X,u^{\star}\rangle)+\ell_{\sigma_{j},\theta}(-1\mid X)\sigma^{\star}_{j}(-\langle X,u^{\star}\rangle)],

so that leveraging the usual ansatz that $\theta=tu^{\star}$ , we have

	$\displaystyle\nabla L(\theta,\vec{\sigma})$	$\displaystyle=\mathbb{E}\left[\frac{1}{m}\sum_{j=1}^{m}\left(\sigma^{\star}_{j}(-\langle X,u^{\star}\rangle)\sigma_{j}(\langle\theta,X\rangle)-\sigma^{\star}_{j}(\langle X,u^{\star}\rangle)\sigma_{j}(-\langle\theta,X\rangle)\right)X\right]$
		$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}\left[\left(\sigma_{j}^{\star}(-Z)\sigma_{j}(tZ)-\sigma_{j}^{\star}(Z)\sigma_{j}(-tZ)Z\right)\right]u^{\star}$
		$\displaystyle=h_{\vec{\sigma}}(t)u^{\star},$

where $h_{\vec{\sigma}}(t)=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[(\sigma_{j}^{\star}(-Z)\sigma_{j}(tZ)-\sigma_{j}^{\star}(Z)\sigma_{j}(-tZ))Z]$ is the immediate generalization of the gap (10). We now simplify it to the form (29). Using the symmetries $\sigma^{\star}(z)=1-\sigma^{\star}(-z)$ and $\sigma(z)=1-\sigma(-z)$ for any $\sigma,\sigma^{\star}\in\mathcal{F}_{\mathsf{link}}$ , we observe that

\sigma(tz)(1-\sigma^{\star}(z))-\sigma(-tz)\sigma^{\star}(z)=\sigma(tz)-(\sigma(tz)+\sigma(-tz))\sigma^{\star}(z)=\sigma(tz)-\sigma^{\star}(z),

and so $h_{\vec{\sigma}}(t)=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[(\sigma_{j}(tZ)-\sigma_{j}^{\star}(Z))Z]$ . Of course, $h_{\vec{\sigma}}(0)=-\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[\sigma_{j}^{\star}(Z)Z]<0$ , as $\mathbb{E}[Z]=0$ , while $\lim_{t\to\infty}h_{\vec{\sigma}}(t)=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[|Z|-\sigma_{j}^{\star}(Z)Z]>0$ . Then as $h_{\vec{\sigma}}^{\prime}(t)=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[\sigma_{j}^{\prime}(tZ)Z^{2}]>0$ , there exists a unique $t_{\vec{\sigma}}\in(0,\infty)$ satisfying $h_{\vec{\sigma}}(t_{\vec{\sigma}})=0$ .

We turn to the Hessian derivation, where as in the proof of Lemma 3.1, we write for $\theta=tu^{\star}$ that

	$\displaystyle\nabla^{2}L(\theta,\vec{\sigma})$	$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}\left[(\sigma_{j}^{\prime}(-tZ)\sigma^{\star}_{j}(Z)+\sigma_{j}^{\prime}(tZ)\sigma^{\star}_{j}(-Z))Z^{2}\right]u^{\star}{u^{\star}}^{\top}$
		$\displaystyle\qquad~{}+\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}\left[(\sigma_{j}^{\prime}(-tZ)\sigma^{\star}_{j}(Z)+\sigma_{j}^{\prime}(tZ)\sigma^{\star}_{j}(-Z))\right]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp},$

and so $\nabla^{2}L(\theta,\vec{\sigma})\succ 0$ and $L(\theta,\vec{\sigma})$ has unique minimizer $\theta_{\vec{\sigma}}^{\star}=t_{\vec{\sigma}}u^{\star}$ . ∎

With Lemma G.1 serving as the analogue of Lemma 3.1, we can now show the continuity of the optimizing parameter $\theta^{\star}_{\vec{\sigma}}$ in $\vec{\sigma}$ :

Lemma G.2.

As $\left\|{\vec{\sigma}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}\to 0$ , we have $\theta^{\star}_{\vec{\sigma}}-u^{\star}\to 0$ .

Proof.

Via Lemma G.1, it is evidently sufficient to show that the solution $t_{\vec{\sigma}}$ to $h_{\vec{\sigma}}(t)=0$ converges to $1$ . To show this, note the expansion

mh_{\vec{\sigma}}(t)=\sum_{j=1}^{m}\left(\mathbb{E}[(\sigma_{j}(tZ)-\sigma_{j}^{\star}(tZ))Z]+\mathbb{E}[(\sigma_{j}^{\star}(tZ)-\sigma_{j}^{\star}(Z))Z]\right).

(30)

We use the following claim, which shows that the first term tends to 0 uniformly in $t$ near $1$ :

Claim G.3.

For any $\sigma,\sigma^{\star}\in\mathcal{F}_{\mathsf{link}}$ , we have $\sup_{t\in[\frac{1}{2},2]}\left\|{\sigma(tZ)-\sigma^{\star}(tZ)}\right\|_{L^{2}(\mathbb{P})}\to 0$ whenever $\left\|{\sigma^{\star}-\sigma}\right\|_{L^{2}(\mathbb{P})}\to 0$ .

Proof.

We use that the density $p(z)$ of $|Z|$ is continuous and nonzero on $(0,\infty)$ . Take any $0<M_{0}<M_{1}<\infty$ . Then

	$\displaystyle\sup_{t\in[\frac{1}{2},2]}\left\\|{\sigma^{\star}(tZ)-\sigma(tZ)}\right\\|_{L^{2}(\mathbb{P})}=\sup_{t\in[\frac{1}{2},2]}\sqrt{\int_{0}^{\infty}\left({\sigma^{\star}(tz)-\sigma(tz)}\right)^{2}p(z)dz}$
		$\displaystyle\stackrel{{\scriptstyle(i)}}{{\leq}}\mathbb{P}(\|Z\|\leq M_{0})+\mathbb{P}(\|Z\|\geq M_{1})+\sup_{t\in[\frac{1}{2},2]}\sqrt{\int_{M_{0}}^{M_{1}}\left({\sigma^{\star}(tz)-\sigma(tz)}\right)^{2}p(tz)\cdot\frac{p(z)}{p(tz)}dz}$
		$\displaystyle\leq\mathbb{P}(\|Z\|\leq M_{0})+\mathbb{P}(\|Z\|\geq M_{1})+\sup_{\frac{M_{0}}{2}\leq z,z^{\prime}\leq 2M_{1}}\sqrt{\frac{p(z)}{p(z^{\prime})}}\sup_{t\in[\frac{1}{2},2]}\sqrt{\int_{M_{0}}^{M_{1}}\left({\sigma^{\star}(tz)-\sigma(tz)}\right)^{2}p(tz)dz},$

where in $(i)$ we use that the link functions $\sigma^{\star}$ and $\sigma$ are bounded within $[0,1]$ . For any fixed $0<M_{0}\leq M_{1}<\infty$ , the ratio $\frac{p(z)}{p(z^{\prime})}$ is bounded for $z,z^{\prime}\in[\frac{1}{2}M_{0},2M_{1}]$ , and using the substitution $tz\mapsto z$ we have

\displaystyle\sup_{t\in[\frac{1}{2},2]}\sqrt{\int_{M_{0}}^{M_{1}}\left({\sigma^{\star}(tz)-\sigma(tz)}\right)^{2}p(tz)dz}\leq\sqrt{2}\cdot\left\|{\sigma^{\star}-\sigma}\right\|_{L^{2}(\mathbb{P})}.

We thus have that $\left\|{\sigma(tZ)-\sigma^{\star}(tZ)}\right\|_{L^{2}(\mathbb{P})}\leq K\left\|{\sigma-\sigma^{\star}}\right\|_{L^{2}(\mathbb{P})}+\mathbb{P}(|Z|\not\in[M_{0},M_{1}])$ , where $K$ depends only on $M_{0},M_{1}$ and the distribution of $Z$ . Take $M_{0}\downarrow 0$ and $M_{1}\uparrow\infty$ . ∎

Leveraging the expansion (30) preceding Claim G.3 and the claim itself, we see that

h_{\vec{\sigma}}(t)=\mathbb{E}[Z^{2}]\cdot o(1)+\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}[(\sigma_{j}^{\star}(tZ)-\sigma_{j}^{\star}(Z))Z]

uniformly in $t\in[\frac{1}{2},2]$ as $\left\|{\vec{\sigma}^{\star}-\vec{\sigma}}\right\|_{L^{2}(\mathbb{P})}\to 0$ . The monotonicity of each $\sigma_{j}^{\star}$ guarantees that if $f_{j}(t)=\mathbb{E}[(\sigma_{j}^{\star}(tZ)-\sigma_{j}^{\star}(Z))Z]$ , then $f^{\prime}_{j}(t)=\mathbb{E}[{\sigma_{j}^{\star}}^{\prime}(tZ)Z^{2}]>0$ , and so $t=1$ uniquely solves $f_{j}(t)=0$ and we must have $t_{\vec{\sigma}}\to 1$ as $\left\|{\vec{\sigma}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}\to 0$ . ∎

Finally, we proceed to the third part of the consistency argument: the convergence in probability.

Lemma G.4.

If $\left\|{\vec{\sigma}_{n}-\vec{\sigma}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ , then $\widehat{\theta}^{\textup{sp}}_{n,m}-\theta_{\vec{\sigma}_{n}}^{\star}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ .

Proof.

By Lemma G.1 and the assumed continuity of the population Hessian, there exists $\delta>0$ such that

L(\theta,\vec{\sigma})\geq L(\theta^{\star}_{\vec{\sigma}},\vec{\sigma})+\frac{\lambda}{2}\left\|{\theta-\theta^{\star}_{\vec{\sigma}}}\right\|_{2}^{2}

(31)

whenever both $\left\|{\vec{\sigma}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}\leq\delta$ and $\left\|{\theta^{\star}_{\vec{\sigma}}-\theta}\right\|_{2}\leq\delta$ . Applying the uniform convergence Lemma A.4, we see that for any $r<\infty$ , we have

\sup_{\left\|{\theta}\right\|_{2}\leq r,\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})|\stackrel{{\scriptstyle p}}{{\rightarrow}}0.

For $\delta>0$ , define the events

\mathcal{E}_{n}(\delta)\coloneqq\left\{\left\|{\theta^{\star}_{\vec{\sigma}}-u^{\star}}\right\|_{2}\leq\delta,\left\|{\vec{\sigma}_{n}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}\leq\delta\right\},

where Lemma G.2 and the assumption that $\left\|{\vec{\sigma}_{n}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ imply that $\mathbb{P}(\mathcal{E}_{n}(\delta))\to 1$ for all $\delta>0$ . By the growth condition (31) and uniform convergence $P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ over $\left\|{\theta}\right\|_{2}\leq r$ , we therefore have that with probability tending to 1,

\inf_{\|{\theta-\theta^{\star}_{\vec{\sigma}_{n}}}\|_{2}=\delta}\left\{P_{n}\ell_{\theta,\vec{\sigma}_{n}}-P_{n}\ell_{\theta^{\star}_{\vec{\sigma}_{n}},\vec{\sigma}_{n}}\right\}\geq\frac{\lambda}{4}\delta^{2}.

The convexity of the losses $\ell_{\theta,\vec{\sigma}}$ in $\theta$ and that $\widehat{\theta}^{\textup{sp}}_{n,m}$ minimizes $P_{n}\ell_{\theta,\vec{\sigma}_{n}}$ then guarantee the desired convergence $\widehat{\theta}^{\textup{sp}}_{n,m}-\theta^{\star}_{\vec{\sigma}_{n}}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ . ∎

G.2 Asymptotic normality via Donsker classes

While we do not have $\left\|{\vec{\sigma}_{n}-\vec{\sigma}^{\star}}\right\|_{L^{2}(\mathbb{P})}=o_{P}(n^{-1/2})$ , which allows the cleanest and simplest asymptotic normality results with nuisance parameters (e.g. [43, Thm. 25.54]), we still expect $\sqrt{n}(\widehat{\theta}^{\textup{sp}}_{n,m}-\theta^{\star}_{\vec{\sigma}_{n}})$ to be asymptotically normal, and therefore, as $\theta^{\star}_{\vec{\sigma}}=tu^{\star}$ for some $t>0$ , the normalized estimators $\widehat{\theta}^{\textup{sp}}_{n,m}/\|{\widehat{\theta}^{\textup{sp}}_{n,m}}\|_{2}$ should be asymptotically normal to $u^{\star}$ . To develop the asymptotic normality results, we perform an analysis of the empirical process centered at the estimators $\widehat{\theta}^{\textup{sp}}_{n,m}$ , rather than the “true” parameter $u^{\star}$ as would be typical.

We begin with an expansion. We let $\theta^{\star}_{n}=\theta^{\star}_{\vec{\sigma}_{n}}$ for shorthand. Then as $\mathbb{P}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}=0$ and $\mathbb{P}\nabla_{\theta}\ell_{\theta^{\star}_{n},\vec{\sigma}_{n}}=0$ , we can derive

	$\displaystyle\mathbb{G}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}$	$\displaystyle=\sqrt{n}\left({\mathbb{P}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}-\mathbb{P}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}}\right)$
		$\displaystyle=\sqrt{n}\left({\mathbb{P}\nabla_{\theta}\ell_{\theta^{\star}_{n},\vec{\sigma}_{n}}-\mathbb{P}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}}\right)$
		$\displaystyle=\sqrt{n}\left({\nabla_{\theta}L(\theta^{\star}_{n},\vec{\sigma}_{n})-\nabla_{\theta}L(\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n})}\right)$
		$\displaystyle=\left({\int_{0}^{1}\nabla_{\theta}^{2}L((1-t)\widehat{\theta}^{\textup{sp}}_{n,m}+t\theta^{\star}_{n},\vec{\sigma}_{n})dt}\right)\cdot\sqrt{n}(\theta^{\star}_{n}-\widehat{\theta}^{\textup{sp}}_{n,m}).$

The assumed continuity of $\nabla_{\theta}^{2}L(\theta,\vec{\sigma})$ at $(u^{\star},\vec{\sigma}^{\star})$ (recall Assumption A5) then implies that

	$\displaystyle\sqrt{n}(\theta^{\star}_{n}-\widehat{\theta}^{\textup{sp}}_{n,m})$	$\displaystyle=\left({\int_{0}^{1}\nabla_{\theta}^{2}L((1-t)\theta^{\star}_{n}+t\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n})dt}\right)^{-1}\cdot\,\mathbb{G}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}$
		$\displaystyle=\left({\nabla_{\theta}^{2}L(u^{\star},\vec{\sigma}^{\star})+o_{P}(1)}\right)^{-1}\cdot\mathbb{G}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}},$		(32)

where we have used the consistency guarantees $\theta^{\star}_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}u^{\star},\widehat{\theta}^{\textup{sp}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}u^{\star}$ by Lemmas G.2 and G.4 and that $\left\|{\vec{\sigma}_{n}-\vec{\sigma}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ by assumption.

The expansion (32) forms the basis of our asymptotic normality result; while $\widehat{\theta}^{\textup{sp}}_{n,m}$ and $\vec{\sigma}_{n}$ may be data dependent, by leveraging uniform central limit theorems and the theory of Donsker function classes, we can show that $\mathbb{G}_{n,m}\nabla\ell$ has an appropriate normal limit. To that end, define the function classes

\displaystyle\mathcal{F}_{\delta}\coloneqq\left\{\nabla_{\theta}\ell_{\theta,\sigma}\mid\left\|{\theta-u^{\star}}\right\|_{2}\leq\delta,\sigma\in\mathcal{F}_{\mathsf{link}}\right\},

where we leave the Lipschitz constant $\mathsf{L}$ in $\mathcal{F}_{\mathsf{link}}$ tacit. By Assumption A5 and that $\widehat{\theta}^{\textup{sp}}_{n,m}\stackrel{{\scriptstyle p}}{{\rightarrow}}u^{\star}$ , we know with probability tending to $1$ we have the membership $\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}\in\mathcal{F}_{\delta}$ . The key result is then that $\mathcal{F}_{\delta}$ is a Donsker class:

Lemma G.5.

Assume that $\mathbb{E}[\left\|{X}\right\|_{2}^{4}]<\infty$ . Then $\mathcal{F}_{\delta}$ is a Donsker class, and moreover, if $d_{\mathcal{F}_{\mathsf{link}}}((\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}),(u^{\star},\vec{\sigma}^{\star}))\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ , then

\mathbb{G}_{n,m}\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}-\mathbb{G}_{n,m}\nabla_{\theta}\ell_{u^{\star},\vec{\sigma}^{\star}}\stackrel{{\scriptstyle p}}{{\rightarrow}}0.

Temporarily deferring the proof of Lemma G.5, let us see how it leads to the proof of Theorem 4. Using Lemma G.5 and Slutsky’s lemmas in the equality (32), we obtain

	$\displaystyle\sqrt{n}(\widehat{\theta}^{\textup{sp}}_{n,m}-\theta_{n}^{\star})$	$\displaystyle=-\left({\nabla_{\theta}^{2}L(u^{\star},\vec{\sigma}^{\star})+o_{P}(1)}\right)^{-1}\cdot\mathbb{G}_{n,m}\nabla_{\theta}\ell_{u^{\star},\vec{\sigma}^{\star}}+o_{P}(1)$
		$\displaystyle\stackrel{{\scriptstyle d}}{{\rightarrow}}\mathsf{N}\left(0,\nabla^{2}L(u^{\star},\vec{\sigma}^{\star})\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}\nabla\ell_{u^{\star},\sigma_{j}^{\star}}(Y_{j}\mid X)\bigg{)}\nabla^{2}L(u^{\star},\vec{\sigma}^{\star})\right).$

Calculations completely similar to those we use in the proof of Theorem 1 then give Theorem 4: because $\mathbb{P}(Y_{j}=y\mid X=x)=\sigma_{j}^{\star}(y\langle u^{\star},x\rangle)$ by assumption,

\mathsf{Cov}\bigg{(}\frac{1}{m}\sum_{j=1}^{m}\nabla\ell_{u^{\star},\sigma_{j}^{\star}}(Y_{j}\mid X)\bigg{)}=\frac{1}{m^{2}}\sum_{j=1}^{m}\mathsf{Cov}(\nabla\ell_{u^{\star},\sigma_{j}^{\star}}(Y_{j}\mid X))

because $Y_{j}\mid X$ are conditionally independent, while

	$\displaystyle\mathsf{Cov}(\nabla\ell_{u^{\star},\sigma_{j}^{\star}}(Y_{j}\mid X))$	$\displaystyle=\mathbb{E}[\sigma_{j}^{\star}(Z)(1-\sigma_{j}^{\star}(Z))XX^{\top}]$
		$\displaystyle=\mathbb{E}[\sigma_{j}^{\star}(Z)(1-\sigma_{j}^{\star}(Z))Z^{2}]u^{\star}{u^{\star}}^{\top}+\mathbb{E}[\sigma_{j}^{\star}(Z)(1-\sigma_{j}^{\star}(Z))]\mathsf{P}_{u^{\star}}^{\perp}\Sigma\mathsf{P}_{u^{\star}}^{\perp}.$

When $\sigma_{j}=\sigma_{j}^{\star}$ , the Hessian function $\mathsf{he}_{j}$ in Lemma G.1 simplifies to $\mathsf{he}_{j}(1,z)={\sigma_{j}^{\star}}^{\prime}(z)$ as $\sigma_{j}^{\star}$ is symmetric about 0. We then apply the delta method as in the proof of Theorem 1.

Finally, we return to the proof of Lemma G.5.

Proof of Lemma G.5.

To prove $\mathcal{F}_{\delta}$ is Donsker, we show that each coordinate of $\nabla_{\theta}\ell_{\theta,\sigma}\in\mathcal{F}_{\delta}$ is, and as $\nabla_{\theta}\ell_{\theta,\sigma}(y\mid x)=-y\sigma(-y\langle x,\theta\rangle)x$ , this amounts to showing the coordinate functions $f_{\theta,\sigma}^{(i)}(v)=v_{i}\sigma(-\langle v,\theta\rangle)$ form a Donsker class when $v$ has distribution $V=YX$ . Let

\displaystyle\mathcal{F}_{\delta}^{(i)}

\displaystyle\coloneqq\left\{f_{\theta,\sigma}^{(i)}(\cdot)\mid\left\|{\theta-u^{\star}}\right\|_{2}\leq\delta,~{}\sigma\in\mathcal{F}_{\mathsf{link}}\right\},

so it is evidently sufficient to prove that $\mathcal{F}_{\delta}^{(1)}$ forms a Donsker class.

We use bracketing and entropy numbers [44] to control the $\mathcal{F}_{\delta}^{(i)}$ . Recall that for a function class $\mathcal{F}$ , an $\epsilon$ -bracket of $\mathcal{F}$ in $L^{q}(\mathbb{P})$ is a collection of functions $\{(l_{i},u_{i})\}$ such that for each $f\in\mathcal{F}$ , there exists $i$ such that $\ell_{i}\leq f\leq u_{i}$ and $\left\|{u_{i}-l_{i}}\right\|_{L^{q}(\mathbb{P})}\leq\epsilon$ . The bracketing number $N_{[\,]}(\epsilon,\mathcal{F},L^{q}(\mathbb{P}))$ is the cardinality of the smallest such $\epsilon$ -bracket, and the bracketing entropy is

\displaystyle J_{[\,]}(\mathcal{F},L^{q}(\mathbb{P}))\coloneqq\int_{0}^{\infty}\sqrt{\log N_{[\,]}(\epsilon,\mathcal{F},L^{q}(\mathbb{P}))}d\epsilon.

To show that $\mathcal{F}_{\delta}^{(i)}$ is Donsker, it is sufficient [44, Ch. 2.5.2] to show that $J_{[\,]}(\mathcal{F}_{\delta}^{(i)},L^{2}(\mathbb{P}))<\infty$ . Our approach to demonstrate that $\mathcal{F}_{\delta}^{(1)}$ is Donsker is thus to construct an appropriate $\epsilon$ -bracket of $\mathcal{F}_{\delta}^{(1)}$ , which we do by first covering $\ell_{2}$ -balls in $\mathbb{R}^{d}$ , then for vectors $\theta\in\mathbb{R}^{d}$ , constructing a bracketing of the induced function class $\{f_{\theta,\sigma}^{(1)}\}_{\sigma\in\mathcal{F}_{\mathsf{link}}}$ , which we combine to give the final bracketing of $\mathcal{F}_{\delta}^{(1)}$ .

We proceed with this two-stage covering and bracketing. Let $\epsilon,\gamma>0$ be small numbers whose values we determine later. Define the $\ell_{2}$ ball $\mathbb{B}_{2}^{d}=\{x\in\mathbb{R}^{d}\mid\left\|{x}\right\|_{2}\leq 1\}$ , and for any $0<\epsilon<\delta$ , let $\mathcal{N}_{\epsilon}$ be a minimal $\epsilon$ -cover of $\delta\mathbb{B}_{2}^{d}$ in the Euclidean norm of size $N=N(\epsilon,\delta\mathbb{B}_{2}^{d},\left\|{\cdot}\right\|_{2})$ , $\mathcal{N}_{\epsilon}=\{\theta_{1},\ldots,\theta_{N}\}$ , so that for any $\theta$ with $\left\|{\theta}\right\|_{2}\leq\delta$ there exists $\theta_{i}\in\mathcal{N}_{\epsilon}$ with $\left\|{\theta-\theta_{i}}\right\|_{2}\leq\epsilon$ . Standard bounds [46, Lemma 5.7]) give

\displaystyle\log N(\epsilon,\delta\mathbb{B}_{2}^{d},\left\|{\cdot}\right\|_{2})\leq d\log\left({1+\frac{2\delta}{\epsilon}}\right).

(33)

For simplicity of notation and to avoid certain tedious negations, we define the “flipped” monotone function family

\mathcal{F}_{\mathsf{flip}}\coloneqq\{g:\mathbb{R}\to\mathbb{R}\mid g(t)=\sigma(-t)\}_{\sigma\in\mathcal{F}_{\mathsf{link}}}.

Now, for any $\theta\in\mathbb{R}^{d}$ , let $\mu_{\theta}$ denote the pushforward measure of $\langle V,\theta\rangle=Y\langle X,\theta\rangle$ . For $\theta\in\mathbb{R}^{d}$ , we then let $\mathcal{N}_{[\,],\gamma,\theta}$ be a minimal $\gamma$ -bracketing of $\mathcal{F}_{\mathsf{flip}}$ in the $L^{4}(\mu_{\theta})$ norm. That is, for $N=N_{[\,]}(\gamma,\mathcal{F}_{\mathsf{flip}},L^{2}(\mu_{\theta}))$ , we have $\mathcal{N}_{[\,],\gamma,\theta}=\{(l_{\theta,i},u_{\theta,i})\}_{i=1}^{N}$ , and for each $\sigma\in\mathcal{F}_{\mathsf{link}}$ , there exists $i=i(\sigma)$ such that

\displaystyle l_{\theta,i}(t)\leq\sigma(-t)\leq u_{\theta,i}(t)~{}~{}\mbox{and}~{}~{}\left\|{u_{\theta,i}-l_{\theta,i}}\right\|_{L^{4}(\mu_{\theta})}\leq\gamma.

Because elements of $\mathcal{F}_{\mathsf{flip}}\subset\mathbb{R}\to[0,1]$ are monotone, van der Vaart and Wellner [44, Thm. 2.7.5] guarantee there exists a universal constant $K<\infty$ such that

\displaystyle\sup_{Q}\log N_{[\,]}(\gamma,\mathcal{F}_{\mathsf{flip}},L^{4}(Q))\leq\frac{K}{\gamma},

(34)

and in particular, $\log N_{[\,]}(\gamma,\mathcal{F}_{\mathsf{flip}},L^{4}(\mu_{\theta}))\leq\frac{K}{\gamma}$ for each $\theta\in\mathbb{R}^{d}$ .

With the covering $\mathcal{N}_{\epsilon}$ and induced bracketing collections $\mathcal{N}_{[\,],\gamma,\theta}$ , we now turn to a construction of the actual bracketing of the class $\mathcal{F}_{\delta}^{(1)}$ . For any $\theta\in\mathbb{R}^{d}$ and bracket $(l_{\theta,i},u_{\theta,i})\in\mathcal{N}_{[\,],\gamma,\theta}$ , define the functionals $\widehat{l}_{\theta,j},\widehat{u}_{\theta,j}:\mathbb{R}^{d}\to\mathbb{R}$ by

	$\displaystyle\widehat{l}_{\theta,j}(v)$	$\displaystyle\coloneqq\left({v_{1}}\right)_{+}\max\left\{l_{\theta,j}(\langle v,\theta\rangle)-\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,0\right\}-\left({-v_{1}}\right)_{+}\min\left\{u_{\theta,j}(\langle v,\theta\rangle)+\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,1\right\},$
	$\displaystyle\widehat{u}_{\theta,j}(v)$	$\displaystyle\coloneqq\left({v_{1}}\right)_{+}\min\left\{u_{\theta,j}(\langle v,\theta\rangle)+\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,1\right\}-\left({-v_{1}}\right)_{+}\max\left\{l_{\theta,j}(\langle v,\theta\rangle)-\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,0\right\}.$

The key is that these functions form a bracketing of $\mathcal{F}_{\delta}^{(1)}$ :

Lemma G.6.

Define the set

\mathcal{B}_{\epsilon,\gamma}\coloneqq\left\{\left(\widehat{l}_{\theta_{i},j},\widehat{u}_{\theta_{i},j}\right)\mid\theta_{i}\in\mathcal{N}_{\epsilon},~{}1\leq j\leq N_{[\,]}(\gamma,\mathcal{F}_{\mathsf{flip}},L^{4}(\mu_{\theta_{i}}))\right\}.

Then $\mathcal{B}_{\epsilon,\gamma}$ is a

2\mathsf{L}\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/2}\cdot\epsilon+\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/4}\cdot\gamma

bracketing of $\mathcal{F}_{\delta}^{(1)}$ with cardinality at most $\log\textup{card}(\mathcal{B}_{\epsilon,\gamma})\leq\frac{K}{\gamma}+d\log(1+\frac{\delta}{\epsilon})$ .

Proof.

Let $f_{\theta,\sigma}^{(1)}(v)\in\mathcal{F}_{\delta}^{(1)}$ . Take $\theta_{i}\in\mathcal{N}_{\epsilon}$ satisfying $\left\|{\theta-\theta_{i}}\right\|_{2}\leq\epsilon$ and $(l_{\theta_{i},j},u_{\theta,j})\in\mathcal{N}_{[\,],\gamma,\theta_{i}}$ such that $l_{\theta_{i},j}(t)\leq\sigma(-t)\leq u_{\theta_{i},j}(t)$ for all $t$ , where $\left\|{u_{\theta_{i},j}-l_{\theta_{i},j}}\right\|_{L^{4}(\mu_{\theta_{i}})}\leq\gamma$ . We first demonstrate the bracketing guarantee

\displaystyle\widehat{l}_{\theta_{i},j}(v)\leq f_{\theta,\sigma}^{(1)}(v)=v_{1}\sigma(-\langle v,\theta\rangle)\leq\widehat{u}_{\theta_{i},j}(v)~{}~{}\mbox{for~{}all~{}}v\in\mathbb{R}^{d}.

For the upper bound, we have

	$\displaystyle f_{\theta,\sigma}^{(1)}(v)=v_{1}\sigma(-\langle v,\theta\rangle)$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(i)}}}{{\leq}}\left({v_{1}}\right)_{+}\min\left\{\sigma(-\langle v,\theta_{i}\rangle)+\mathsf{L}\|\langle v,\theta_{i}-\theta\rangle\|,1\right\}-\left({-v_{1}}\right)_{+}\max\left\{\sigma(-\langle v,\theta_{i}\rangle)-\mathsf{L}\|\langle v,\theta_{i}-\theta\rangle\|,0\right\}$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(ii)}}}{{\leq}}\left({v_{1}}\right)_{+}\min\left\{\sigma(-\langle v,\theta_{i}\rangle)+\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,1\right\}-\left({-v_{1}}\right)_{+}\max\left\{\sigma(-\langle v,\theta_{i}\rangle)-\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,0\right\}$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(iii)}}}{{\leq}}\left({v_{1}}\right)_{+}\min\left\{u_{\theta_{i},j}(\langle v,\theta_{i}\rangle)+\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,1\right\}-\left({-v_{1}}\right)_{+}\max\left\{l_{\theta_{i},j}(\langle v,\theta_{i}\rangle)-\mathsf{L}\left\\|{v}\right\\|_{2}\epsilon,0\right\}$
		$\displaystyle=\widehat{u}_{\theta_{i},j}(v),$

where step (i) follows from the $\mathsf{L}$ -Lipschitz continuity of $\sigma$ , (ii) from the Cauchy-Schwarz inequality and that $\left\|{\theta-\theta_{i}}\right\|_{2}\leq\epsilon$ , while step (iii) follows by the construction that $l_{\theta_{i},j}(t)\leq\sigma(-t)\leq u_{\theta_{i},j}(t)$ for all $t\in\mathbb{R}$ . Similarly, we obtain the lower bound

\displaystyle f_{\theta,\sigma}^{(1)}(v)=v_{1}\sigma(-\langle v,\theta\rangle)\geq\widehat{l}_{\theta_{i},j}(v),

again valid for all $v\in\mathbb{R}^{d}$ .

The second part of the proof is to bound the distance between the upper and lower elements in the bracketing. By definition, $\widehat{u}_{\theta_{i},j}-\widehat{l}_{\theta_{i},j}$ has the pointwise upper bound

\displaystyle\left({\widehat{u}_{\theta_{i},j}(v)-\widehat{l}_{\theta_{i},j}(v)}\right)^{2}

\displaystyle\leq\left({|v_{1}|\left({u_{\theta_{i},j}(\langle v,\theta_{i}\rangle)-l_{\theta_{i},j}(\langle v,\theta_{i}\rangle)+2\mathsf{L}\left\|{v}\right\|_{2}\epsilon}\right)}\right)^{2}.

Recalling that $V=YX$ , by the Minkowski and Cauchy-Schwarz inequalities, we thus obtain

	$\displaystyle\left\\|{\widehat{u}_{\theta_{i},j}(V)-\widehat{l}_{\theta_{i},j}(V)}\right\\|_{L^{2}(\mathbb{P})}$	$\displaystyle\leq\left\\|{\|V_{1}\|\left({u_{\theta_{i},j}(\langle V,\theta_{i}\rangle)-l_{\theta_{i},j}(\langle V,\theta_{i}\rangle)}\right)}\right\\|_{L^{2}(\mathbb{P})}+\left\\|{\|V_{1}\|\cdot 2\mathsf{L}\left\\|{V}\right\\|_{2}\epsilon}\right\\|_{L^{2}(\mathbb{P})}$
		$\displaystyle\leq\left\\|{\|V_{1}\|}\right\\|_{L^{4}(\mathbb{P})}\cdot\left({\left\\|{u_{\theta_{i},j}(\langle V,\theta_{i}\rangle)-l_{\theta_{i},j}(\langle V,\theta_{i}\rangle)}\right\\|_{L^{4}(\mathbb{P})}+2\mathsf{L}\epsilon\cdot\left\\|{\left\\|{V}\right\\|_{2}}\right\\|_{L^{4}(\mathbb{P})}}\right).$

Noting the trivial bounds $\left\|{|V_{1}|}\right\|_{L^{4}(\mathbb{P})}\leq\left\|{X}\right\|_{L^{4}(\mathbb{P})}<\infty$ and the assumed bracketing distance

\displaystyle\left\|{u_{\theta_{i},j}(\langle V,\theta_{i}\rangle)-l_{\theta_{i},j}(\langle V,\theta_{i}\rangle)}\right\|_{L^{4}(\mathbb{P})}

\displaystyle=\left\|{u_{\theta_{i},j}-l_{\theta_{i},j}}\right\|_{L^{4}(\mu_{\theta_{i}})}\leq\gamma,

we have the desired bracketing distance $\|{\widehat{u}_{\theta_{i},j}-\widehat{l}_{\theta_{i},j}}\|_{L^{2}(\mathbb{P})}\leq 2\mathsf{L}\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/2}\epsilon+\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/4}\gamma$ .

The final cardinality bound is immediate via inequalities (33) and (34). ∎

Lemma G.6 will yield the desired entropy integral bound. Fix any $t>0$ , and note that if we take $\epsilon=\epsilon(t)\coloneqq t/(4\mathsf{L}\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/2})$ and $\gamma=\gamma(t)\coloneqq t/(2\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/4})$ , then the set $\mathcal{B}_{\epsilon,\gamma}$ is a $t$ -bracketing of $\mathcal{F}_{\delta}^{(1)}$ in $L^{2}$ , and moreover, we have the cardinality bound

\displaystyle\log N_{[\,]}(t,\mathcal{F}_{\delta}^{(1)},L^{2}(\mathbb{P}))

\displaystyle\leq d\log\left(1+\frac{2\delta}{\epsilon(t)}\right)+\frac{K}{\gamma(t)}\leq\frac{8\mathsf{L}d\delta\cdot\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/2}+2K\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/4}}{t}.

Additionally, as covering numbers are necessarily integer, we have $\log N_{[\,]}(t)=0$ whenever $t>(8\mathsf{L}d\delta\cdot\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/2}+2K\mathbb{E}[\left\|{X}\right\|_{2}^{4}]^{1/4})/\log 2$ . This gives the entropy integral bound

\displaystyle J_{[\,]}(\mathcal{F}_{\delta}^{(1)},L^{2}(\mathbb{P}))

\displaystyle=\int_{0}^{\infty}\sqrt{\log N_{[\,]}(t,\mathcal{F}_{\delta}^{(1)},L^{2}(\mathbb{P}))}dt<\infty,

and consequently (cf. [43, Thm. 19.5] or [44, Ch. 2.5.2]), $\mathcal{F}_{\delta}^{(1)}$ is a Donsker class. A completely identical argument shows that $\mathcal{F}_{\delta}^{(i)},i=2,3,\dots,d$ are Donsker, and so $\mathcal{F}_{\delta}$ is a Donsker class, completing the proof of the first claim in Lemma G.5.

To complete the proof of Lemma G.5, we need to show that

\displaystyle\mathbb{G}_{n,m}(\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\vec{\sigma}_{n}}-\nabla_{\theta}\ell_{u^{\star},\vec{\sigma}^{\star}})=\frac{1}{m}\sum_{j=1}^{m}\mathbb{G}_{n}^{(j)}(\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\sigma_{n,j}}-\nabla_{\theta}\ell_{u^{\star},\sigma^{\star}_{j}})\stackrel{{\scriptstyle p}}{{\rightarrow}}0,

where $\mathbb{G}_{n}^{(j)}$ denotes the empirical process on $(X_{i},Y_{ij})_{i=1}^{n}$ . Notably, because $m$ is finite, it is sufficient to show that

\mathbb{G}_{n}^{(j)}(\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\sigma_{n,j}}-\nabla_{\theta}\ell_{u^{\star},\sigma^{\star}_{j}})\stackrel{{\scriptstyle p}}{{\rightarrow}}0,~{}~{}~{}j=1,\ldots,m.

To that end, we suppress dependence on $j$ for notational simplicity and simply write $\mathbb{G}_{n}$ and $\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\sigma_{n}}$ , where $\left\|{\sigma_{n}-\sigma^{\star}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ . For any Donsker class $\mathcal{F}\subset\mathcal{X}\to\mathbb{R}^{d}$ and $\epsilon>0$ , we have

\limsup_{\delta\downarrow 0}\limsup_{n\to\infty}\mathbb{P}\left(\sup_{\left\|{f-g}\right\|_{L^{2}(\mathbb{P})}\leq\delta}\mathbb{G}_{n}(f-g)\geq\epsilon\right)=0,

(see [14, Thm. 3.7.31]), and so in turn it is sufficient to prove that

\left\|{\nabla\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\sigma_{n}}-\nabla\ell_{u^{\star},\sigma^{\star}}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0.

(35)

To demonstrate the convergence (35), let $M$ be finite, and note that for any fixed $\theta,\sigma\in\mathcal{F}_{\mathsf{link}}$ that for $V=YX$ we have

	$\displaystyle\left\\|{\nabla\ell_{\theta,\sigma}-\nabla\ell_{u^{\star},\sigma^{\star}}}\right\\|_{L^{2}(\mathbb{P})}^{2}$	$\displaystyle=\mathbb{E}\left[{\left\\|{V\sigma(-\langle V,\theta\rangle)-V\sigma^{\star}(-\langle V,u^{\star}\rangle)}\right\\|_{2}^{2}}\right]$
		$\displaystyle\leq\mathbb{E}\left[{\left\\|{V}\right\\|_{2}^{2}1\!\left\{{\left\\|{V}\right\\|_{2}\geq M}\right\}}\right]+M^{2}\left\\|{\sigma(-\langle V,\theta\rangle)-\sigma^{\star}(-\langle V,u^{\star}\rangle)}\right\\|_{L^{2}(\mathbb{P})}^{2}$
		$\displaystyle\leq\mathbb{E}\left[{\left\\|{X}\right\\|_{2}^{2}1\!\left\{{\left\\|{X}\right\\|_{2}\geq M}\right\}}\right]+2M^{2}\left\\|{\sigma(-\langle V,u^{\star}\rangle)-\sigma^{\star}(-\langle V,u^{\star}\rangle)}\right\\|_{L^{2}(\mathbb{P})}^{2}$
		$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad~{}+2M^{2}\left\\|{\sigma(-\langle V,u^{\star}\rangle)-\sigma(-\langle V,\theta\rangle)}\right\\|_{L^{2}(\mathbb{P})}^{2}$

by the triangle inequality. As

\displaystyle\left\|{\sigma(-\langle V,u^{\star}\rangle)-\sigma(-\langle V,\theta\rangle)}\right\|_{L^{2}(\mathbb{P})}\leq\mathsf{L}\left\|{\theta-u^{\star}}\right\|_{2}\cdot\left\|{V}\right\|_{L^{2}(\mathbb{P})}=\mathsf{L}\left\|{\theta-u^{\star}}\right\|_{2}\cdot\left\|{X}\right\|_{L^{2}(\mathbb{P})}

and $\widehat{\theta}^{\textup{sp}}_{n,m}-u^{\star}\stackrel{{\scriptstyle p}}{{\rightarrow}}0$ and

\displaystyle\left\|{\sigma_{n}(-\langle V,u^{\star}\rangle)-\sigma^{\star}(-\langle V,u^{\star}\rangle)}\right\|_{L^{2}(\mathbb{P})}=\left\|{\sigma_{n}-\sigma^{\star}}\right\|_{L^{2}(\mathbb{P})}\stackrel{{\scriptstyle p}}{{\rightarrow}}0

by assumption, it follows that for any $\epsilon>0$ that

\displaystyle\mathbb{P}\left(\left\|{\nabla_{\theta}\ell_{\widehat{\theta}^{\textup{sp}}_{n,m},\sigma_{n}}-\nabla_{\theta}\ell_{u^{\star},\sigma^{\star}}}\right\|_{L^{2}(\mathbb{P})}\geq\mathbb{E}[\left\|{X}\right\|_{2}^{2}1\!\left\{{\left\|{X}\right\|_{2}\geq M}\right\}]+\epsilon\right)\to 0.

Taking $M\uparrow\infty$ gives the convergence (35), completing the proof.

Appendix H Proofs for semiparametric approaches

H.1 Proof of Lemma 5.1

As $\sigma$ and $\sigma^{\star}$ are $\mathsf{L}$ -Lipschitz, we may without loss of generality assume that $\mathsf{L}=1$ and so $\left\|{\sigma_{j}^{\prime}}\right\|_{\infty}\leq 1$ and $\left\|{{\sigma^{\star}_{j}}^{\prime}}\right\|_{\infty}\leq 1$ . We can compute the Hessian at any $\theta\in\mathbb{R}^{d}$ and $\vec{\sigma}=(\sigma_{1},\dots,\sigma_{m})$ ,

	$\displaystyle\nabla^{2}L(\theta,\vec{\sigma})$	$\displaystyle=\mathbb{E}\left[\frac{1}{m}\sum_{j=1}^{m}\left(\sigma^{\star}_{j}(-\langle X,u^{\star}\rangle)\sigma_{j}^{\prime}(\langle\theta,X\rangle)+\sigma^{\star}_{j}(\langle X,u^{\star}\rangle)\sigma_{j}^{\prime}(-\langle\theta,X\rangle)\right)XX^{\top}\right]$
		$\displaystyle=\frac{1}{m}\sum_{j=1}^{m}\mathbb{E}\left[\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)XX^{\top}\right],$

where in the last line we use that $\sigma_{j}^{\star}$ and $\sigma_{j}$ are symmetric for $j=1,\dots,m$ . Therefore we can upper bound the distance between Hessians by

\displaystyle\left\|{\nabla^{2}L(\theta,\vec{\sigma})-\nabla^{2}L(u^{\star},\vec{\sigma}^{\star})}\right\|

\displaystyle\leq\frac{1}{m}\sum_{j=1}^{m}\underbrace{\left\|{\mathbb{E}\left[\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)XX^{\top}\right]-\mathbb{E}\left[{\sigma_{j}^{\star}}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}\right]}\right\|}_{:=\delta_{j}},

and thus we only need to prove each quantity $\delta_{j}\to 0$ if $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}((\theta,\vec{\sigma}),(u^{\star},\vec{\sigma}^{\star}))\to 0$ , that is, if $\left\|{\theta-u^{\star}}\right\|_{2}\to 0$ and $\left\|{\sigma_{j}(-Y\langle X,u^{\star}\rangle)-\sigma_{j}^{\star}(-Y\langle X,u^{\star}\rangle)}\right\|_{L^{2}(\mathbb{P})}\to 0$ . In the following, we will show $\delta_{j}\to 0$ under the two different conditions. To further simplify the quantity, we claim it is sufficient to show $\xi_{j}\coloneqq\left\|{\mathbb{E}[\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)XX^{\top}]-\mathbb{E}[\sigma_{j}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}]}\right\|\to 0$ . Indeed, we have

Lemma H.1.

If $\xi_{j}\to 0$ and $\left\|{\sigma_{j}(-Y\langle X,u^{\star}\rangle)-\sigma_{j}^{\star}(-Y\langle X,u^{\star}\rangle)}\right\|_{L^{2}(\mathbb{P})}\to 0$ , then $\delta_{j}\to 0$ .

Proof.

By the triangle inequality and the independent decomposition $X=Zu^{\star}+W$ , we have

	$\displaystyle\delta_{j}$	$\displaystyle\leq\xi_{j}+\left\\|{\mathbb{E}\left[\sigma_{j}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}\right]-\mathbb{E}\left[{\sigma_{j}^{\star}}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}\right]}\right\\|$
		$\displaystyle=\xi_{j}+\left\\|{\mathbb{E}\left[(\sigma_{j}^{\prime}(Z)-{\sigma_{j}^{\star}}^{\prime}(Z))Z^{2}\right]\cdot u^{\star}{u^{\star}}^{\top}}\right\\|=\xi_{j}+\left\|\mathbb{E}\left[(\sigma_{j}^{\prime}(Z)-{\sigma_{j}^{\star}}^{\prime}(Z))Z^{2}\right]\right\|.$

It remains to show $\mathbb{E}\left[(\sigma_{j}^{\prime}(Z)-{\sigma_{j}^{\star}}^{\prime}(Z))Z^{2}\right]\to 0$ . Using the symmetry of $\sigma_{j}$ and $\sigma^{\star}_{j}$ , so $\sigma_{j}^{\prime}(t)=\sigma_{j}^{\prime}(-t)$ , we can replace $Z$ by $|Z|$ . Then integrating by parts, for any $0<\epsilon<M<\infty$ we have

	$\displaystyle\mathbb{E}\left[{\sigma_{j}^{\prime}(Z)Z^{2}\mathds{1}\{\epsilon\leq\|Z\|\leq M\}}\right]$	$\displaystyle=\int_{\epsilon}^{M}\sigma_{j}^{\prime}(z)z^{2}p(z)dz$
		$\displaystyle=\sigma_{j}(M)M^{2}p(M)-\sigma_{j}(\epsilon)\epsilon^{2}p(\epsilon)-\int_{\epsilon}^{M}\sigma_{j}(z)(2zp(z)+z^{2}p^{\prime}(z))dz.$

By our w.l.o.g. assumption that $\left\|{\sigma^{\prime}}\right\|_{\infty}\leq 1$ , we have $|\mathbb{E}[\sigma_{j}^{\prime}(Z)Z^{2}]-\mathbb{E}[\sigma_{j}^{\prime}(Z)Z^{2}\mathds{1}\{\epsilon\leq|Z|\leq M\}]|\leq\mathbb{E}[Z^{2}\mathds{1}\{|Z|<\epsilon\text{ or }|Z|>M\}]$ . Thus, recognizing the trivial bound $\left\|{\sigma_{j}}\right\|_{\infty}\leq 1$ , we have

	$\displaystyle\left\|\mathbb{E}\left[(\sigma_{j}^{\prime}(Z)-{\sigma_{j}^{\star}}^{\prime}(Z))Z^{2}\right]\right\|$	$\displaystyle\leq 2\mathbb{E}[Z^{2}\mathds{1}\{\|Z\|<\epsilon\text{ or }\|Z\|>M\}]+2\left({\epsilon^{2}p(\epsilon)+M^{2}p(M)}\right)+$
		$\displaystyle\qquad+\underbrace{\left\|\int_{\epsilon}^{M}\sigma_{j}(z)(2zp(z)+z^{2}p^{\prime}(z))-\int_{\epsilon}^{M}\sigma_{j}^{\star}(z)(2zp(z)+z^{2}p^{\prime}(z))\right\|}_{(\star)}.$

We show for any fixed $0<\epsilon<M<\infty$ , $(\star)\to 0$ . Applying the Cauchy-Schwarz inequality twice, we have the bounds

	$\displaystyle\left\|\int_{\epsilon}^{M}(\sigma_{j}(z)-\sigma_{j}^{\star}(z))zp(z)dz\right\|$	$\displaystyle\leq\left\\|{\sigma_{j}(Z)-\sigma_{j}^{\star}(Z)}\right\\|_{L^{2}(\mathbb{P})}\cdot\sqrt{\mathbb{E}[Z^{2}]}\to 0,$
	$\displaystyle\left\|\int_{\epsilon}^{M}(\sigma_{j}(z)-\sigma_{j}^{\star}(z))z^{2}p^{\prime}(z)dz\right\|$	$\displaystyle\leq\left\\|{\sigma_{j}(Z)-\sigma_{j}^{\star}(Z)}\right\\|_{L^{2}(\mathbb{P})}\cdot\sqrt{\int_{\epsilon}^{M}z^{4}\left({\frac{p^{\prime}(z)}{p(z)}}\right)^{2}p(z)dz}$
		$\displaystyle\leq\left\\|{\sigma_{j}(Z)-\sigma_{j}^{\star}(Z)}\right\\|_{L^{2}(\mathbb{P})}\cdot\sup_{z\in[\epsilon,M]}\left\|\frac{p^{\prime}(z)}{p(z)}\right\|\sqrt{\mathbb{E}[Z^{4}]}\to 0,$

where for the final inequality we use that $p(z)$ is nonzero and continuously differentiable.

As $(\star)\to 0$ , we evidently have $\limsup|\mathbb{E}[(\sigma_{j}^{\prime}(Z)-{\sigma_{j}^{\star}}^{\prime}(Z))Z^{2}]|\leq 2\mathbb{E}[Z^{2}(\mathds{1}\{|Z|<\epsilon\}+\mathds{1}\{|Z|>M\})]+2(\epsilon^{2}p(\epsilon)+M^{2}p(M))$ for arbitrary $0<\epsilon<M<\infty$ . Using the assumptions that $\mathbb{E}[Z^{2}]\leq\mathbb{E}[\left\|{X}\right\|_{2}^{2}]<\infty$ and $\lim_{z\to s}z^{2}p(z)=0$ for $s\in\{0,\infty\}$ , we conclude the proof by taking $\epsilon\to 0$ and $M\to\infty$ . ∎

Finally we prove $\xi_{j}\coloneqq\left\|{\mathbb{E}[\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)XX^{\top}]-\mathbb{E}[\sigma_{j}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}]}\right\|\to 0$ under the two conditions in the statement of Lemma 5.1: that $\sigma^{\prime}_{j}$ are Lipschitz or $X$ has a continuous density.

Condition 1. The links have Lipschitz derivatives

We apply Jensen’s inequality to write

	$\displaystyle\xi_{j}\leq\mathbb{E}\left[\|\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)-\sigma_{j}^{\prime}(-Y\langle X,u^{\star}\rangle)\|\cdot\left\\|{XX^{\top}}\right\\|\right]$	$\displaystyle=\mathbb{E}\left[\|\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)-\sigma_{j}^{\prime}(-Y\langle X,u^{\star}\rangle)\|\cdot\left\\|{X}\right\\|_{2}^{2}\right]$
		$\displaystyle\leq\left\\|{\sigma_{j}^{\prime}}\right\\|_{\textup{Lip}}\left\\|{\theta-u^{\star}}\right\\|_{2}\cdot\mathbb{E}\left[\left\\|{X}\right\\|_{2}^{3}\right],$

where $\left\|{\cdot}\right\|_{\textup{Lip}}$ denotes the Lipschitz constant of its argument. Taking $\theta\to u^{\star}$ completes the proof for this case.

Condition 2. The covariates have continuous density

Let $X$ have density $q(x)$ . We rewrite the convergence $\theta\to u^{\star}$ instead as $\theta=Vu^{\star}$ where $V\to I_{d}$ is invertible. We again divide the expectation into large $\left\|{X}\right\|_{2}$ part and small $\left\|{X}\right\|_{2}$ part. Let $M<\infty$ be large enough that $\mathbb{E}[\left\|{X}\right\|_{2}^{2}\mathds{1}\{\left\|{X}\right\|_{2}>M\}]\leq\epsilon$ . Then using $\|{\sigma_{j}^{\prime}}\|_{\infty}<\infty$ , we obtain

	$\displaystyle\xi_{j}$	$\displaystyle\leq\left\\|{\mathbb{E}\left[(\sigma_{j}^{\prime}(-Y\langle X,\theta\rangle)-{\sigma_{j}}^{\prime}(-Y\langle X,u^{\star}\rangle))XX^{\top}\right]\mathds{1}\{\left\\|{X}\right\\|_{2}\leq M\}}\right\\|+2\mathsf{L}^{\prime}\mathbb{E}\left[{\left\\|{X}\right\\|_{2}^{2}\mathds{1}\{\left\\|{X}\right\\|_{2}>M\}}\right]$
		$\displaystyle\leq\left\\|{\left({V^{-\top}\mathbb{E}\left[\sigma_{j}^{\prime}(-\langle V^{\top}X,u^{\star}\rangle)V^{\top}XXV\right]V-\mathbb{E}\left[{\sigma_{j}}^{\prime}(-\langle X,u^{\star}\rangle)XX^{\top}\right]}\right)\mathds{1}\{\left\\|{X}\right\\|_{2}\leq M\}}\right\\|+2\left\\|{\sigma_{j}^{\prime}}\right\\|_{\infty}\epsilon.$

By the linear transformation of variables $X^{\prime}\coloneqq V^{\top}X$ , the first term in the above display is

	$\displaystyle\left\\|{V^{-\top}\mathbb{E}\left[\sigma_{j}^{\prime}(-Y\langle V^{\top}X,u^{\star}\rangle)V^{\top}XXV\right]V-\mathbb{E}\left[{\sigma_{j}}^{\prime}(-Y\langle X,u^{\star}\rangle)XX^{\top}\right]}\right\\|$
	$\displaystyle=\left\\|{\det(V^{-1})\cdot V^{-\top}\left({\int_{\left\\|{V^{\top}x}\right\\|_{2}\leq M}\sigma_{j}^{\prime}(-x^{\top}u^{\star})xx^{\top}p(V^{-\top}x)dx}\right)V-\int_{\left\\|{x}\right\\|_{2}\leq M}\sigma_{j}^{\prime}(-x^{\top}u^{\star})xx^{\top}p(x)dx}\right\\|.$

Since $p(x)$ is absolutely continuous on any compact set, the above term converges to $0$ as $V\to I_{d}$ . Finally, as $\epsilon>0$ was arbitrary, we take $M\to\infty$ to conclude that $\xi_{j}\to 0$ .

H.2 Proof of Proposition 6

We apply Theorem 4. We first give the specialization of $C_{m,\vec{\sigma}^{\star}}$ that the assumptions of the proposition imply, recognizing that as ${\sigma^{\textup{lr}}}^{\prime}=\sigma^{\textup{lr}}(1-\sigma^{\textup{lr}})$ , we have

\displaystyle C_{m,\vec{\sigma}^{\star}}

\displaystyle=\frac{\sum_{j=1}^{m}\mathbb{E}[\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z)(1-\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z))]}{(\sum_{j=1}^{m}\mathbb{E}[{\sigma^{\textup{lr}}}^{\prime}(\alpha_{j}^{\star}Z)])^{2}}=\left({\sum_{j=1}^{m}\mathbb{E}[\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z)(1-\sigma^{\textup{lr}}(\alpha_{j}^{\star}Z))]}\right)^{-1}.

We now argue that we can actually invoke Theorem 4, which requires verification of Assumption A5. Because for any $M<\infty$ , the link functions $t\mapsto\sigma^{\textup{lr}}(\alpha t)$ have Lipschitz continuous derivatives for $|\alpha|\leq M$ , so when $\vec{\sigma}_{n}$ has form $\vec{\sigma}_{n}=[\sigma^{\textup{lr}}(\alpha_{n,j}\cdot)]_{j=1}^{m}$ , Lemma 5.1 implies the continuity of the mapping $(\theta,\vec{\sigma})\mapsto\nabla_{\theta}^{2}L(\theta,\vec{\sigma})$ for $d_{\mathcal{F}_{\mathsf{link}}^{\mathsf{sp}}}$ at $(u^{\star},\vec{\sigma}^{\star})$ . Then recognizing that by Lipschitz continuity of $\sigma^{\textup{lr}}$ we have

\left\|{\vec{\sigma}_{n}(Z)-\vec{\sigma}^{\star}(Z)}\right\|_{L^{2}(\mathbb{P})}\leq\left\|{\alpha_{n}-\alpha^{\star}}\right\|_{2}^{2}\left\|{Z}\right\|_{L^{2}(\mathbb{P})}^{2}\stackrel{{\scriptstyle p}}{{\rightarrow}}0

whenever $\alpha_{n}\in\mathbb{R}^{m}$ satisfies $\alpha_{n}\stackrel{{\scriptstyle p}}{{\rightarrow}}\alpha^{\star}$ , we obtain the proposition.

Appendix I Proofs of nonparametric convergence results

In this technical appendix, we include proofs of the results from Section 5.2 as well as a few additional results, which are essentially corollaries of results on localized complexities and nonparametric regression models, though we require a few modifications because our setting is slightly non-standard.

I.1 Preliminary results

To set notation and to make reading it self-contained, we provide some definitions. The $L^{r}(P)$ norm of a function or random vector is $\left\|{f}\right\|_{L^{r}(P)}=(\int|f|^{r}dP)^{1/r}$ , so that its $L^{2}(P_{n})$ -norm is $\left\|{f}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}=\frac{1}{n}\sum_{i=1}^{n}f(X_{i})^{2}$ . We consider the following abstract nonparametric regression setting: we have a function class $\mathcal{F}\subset\{\mathbb{R}\to\mathbb{R}\}$ with $f^{\star}\in\mathcal{F}$ , and our observations follow the model

Y_{i}=f^{\star}(X_{i})+\xi_{i},

(36)

but instead of observing $(X_{i},Y_{i})$ pairs we observe $(\widetilde{X}_{i},Y_{i})$ pairs, where $\widetilde{X}_{i}$ may not be identical to $X_{i}$ (these play the roll of $\langle u^{\star},X_{i}\rangle$ versus $\langle u_{n}^{\textup{init}},X_{i}\rangle$ in the results to come). We assume that $\xi_{i}$ are bounded so that $\sup\xi-\inf\xi\leq 1$ , independent, and satisfy the conditional mean-zero property that $\mathbb{E}[\xi_{i}\mid X_{i}]=0$ (though $\xi_{i}$ may not be independent of $X_{i}$ ). For a (thus far unspecified) function class $\mathcal{F}$ we set

\widehat{f}=\operatorname*{argmin}_{f\in\mathcal{F}}P_{n}(Y-f(\widetilde{X}))^{2}.

We now demonstrate that the error $\|{\widehat{f}-f^{\star}}\|_{L^{2}(\mathbb{P}_{n})}^{2}=\frac{1}{n}\sum_{i=1}^{n}(\widehat{f}(X_{i})-f^{\star}(X_{i}))^{2}$ can be bounded by a combination of the errors $\widetilde{X}_{i}-X_{i}$ and local complexities of the function class $\mathcal{F}$ . Our starting point is a local complexity bound analagous to the localization results available for in-sample prediction error in nonparametric regression [cf. 46, Thm. 13.5]. To present the results, for a function class $\mathcal{H}$ we define the localized $\xi$ -complexity

\mathcal{R}_{n}(u;\mathcal{H})\coloneqq\mathbb{E}\left[\sup_{h\in\mathcal{H},\|{h}\|_{L^{2}(\mathbb{P}_{n})}\leq u}\left|n^{-1}\sum_{i=1}^{n}\xi_{i}h(x_{i})\right|\right],

where we treat the expectation conditionally on $X_{i}$ and $\xi_{i}$ are random (i.e. $\xi_{i}=Y_{i}-f^{\star}(X_{i})$ ). For the model (36), we define the centered class $\mathcal{F}^{\star}=\{f-f^{\star}\mid f\in\mathcal{F}\}$ , which is star-shaped as $\mathcal{F}$ is a convex set.¹¹1 A set $\mathcal{H}$ is star-shaped if for all $h\in\mathcal{H}$ , if $\alpha\in[0,1]$ then $\alpha h\in\mathcal{H}$ . We say that $\delta$ satisfies the critical radius inequality if

\frac{1}{\delta}\mathcal{R}_{n}(\delta;\mathcal{F}^{\star})\leq\delta.

(37)

With this, we can provide a proposition giving a high-probability bound on the in-sample prediction error of the empirical estimator $\widehat{f}$ , which is essentially identical to [46, Thm. 13.5], though we require a few modifications to address that we observe $\widetilde{X}_{i}$ and not $X_{i}$ and that the noise $\xi_{i}$ are bounded but not Gaussian.

Proposition 7.

Let $\mathcal{F}$ be a convex function class, $\delta_{n}>0$ satisfy the critical inequality (37), and let $\gamma^{2}=\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}(f(X_{i})-f(\widetilde{X}_{i}))^{2}$ . Then for $t\geq\delta_{n}$ ,

\mathbb{P}\left(\|{\widehat{f}-f^{\star}}\|_{L^{2}(\mathbb{P}_{n})}^{2}\geq 30t\delta_{n}+25\gamma\max\{\gamma,\|{\xi}\|_{L^{2}(\mathbb{P}_{n})}\}\mid X_{1}^{n},\widetilde{X}_{1}^{n}\right)\leq\exp\left(-\frac{nt\delta_{n}}{4}\right).

See Section I.3 for a proof of the proposition.

Revisiting the critical radius inequality (37), we can also apply [46, Corollary 13.7], which allows us to use an entropy integral to guarantee bounds on the critical radius. Here, we again fix $X_{1}^{n}=x_{1}^{n}$ , and for a function class $\mathcal{H}$ we let $\mathcal{B}_{n}(\delta;\mathcal{H})=\{h\in\textup{Star}(\mathcal{H})\mid\left\|{h}\right\|_{L^{2}(\mathbb{P}_{n})}\leq\delta\}$ . Let $N_{n}(t;\mathcal{B})$ denote the $t$ -covering number of $\mathcal{B}$ in $\left\|{n\cdot}\right\|_{L^{2}(\mathbb{P}_{n})}$ -norm. Then modifying a few numerical constants, we have

Corollary 7 (Wainwright [46], Corollary 13.7).

Let the conditions of Proposition 7 hold. Then for a numerical constant $C\leq 16$ , any $\delta\in[0,1]$ satisfying

\frac{C}{\sqrt{n}}\int_{\delta^{2}/4}^{\delta}\sqrt{\log N_{n}(t;\mathcal{B}_{n}(\delta,\mathcal{F}^{\star}))}dt\leq\frac{\delta^{2}}{2}

satisfies the critical inequality (37).

As an immediate consequence of this inequality, we have the following:

Corollary 8.

Assume $|x_{i}|\leq b$ for all $i\in[n]$ and that $\mathcal{F}$ is contained in the class of $M$ -Lipschitz functions with $f(0)=0$ (or any other fixed constant). Then for a numerical constant $c<\infty$ , the choice

\delta_{n}=c\left(\frac{Mb}{n}\right)^{1/3}

satisfies the critical inequality (37).

Proof.

The covering numbers $N_{\infty}$ for the class $\mathcal{F}$ of $M$ -Lipschitz functions on $[-b,b]$ satisfying $f(0)=0$ in supremum norm $\left\|{f}\right\|_{\infty}=\sup_{x\in[-b,b]}|f(x)|$ satisfy $\log N_{\infty}(t;\mathcal{F})\lesssim\frac{Mb}{t}$ [cf. 46, Example 5.10]. Using that $N_{n}\leq N_{\infty}$ , we thus have

\int_{\delta^{2}/4}^{\delta}\sqrt{\log N_{n}(t;\mathcal{B}_{n}(\delta,\mathcal{F}^{\star}))}\lesssim\int_{\delta^{2}/4}^{\delta}\sqrt{\frac{Mb}{t}}dt=2\sqrt{Mb}\left(\sqrt{\delta}-\delta/4\right)\leq 2\sqrt{Mb\delta}

whenever $\delta\leq 1$ . For a numerical constant $c>0$ it suffices in Corollary 7 to choose $\delta$ satisfying $c\frac{1}{\sqrt{n}}\sqrt{Mb\delta}\leq\delta^{2}$ , or $\delta=c(\frac{Mb}{n})^{1/3}$ . ∎

I.2 Proof of Proposition 5

We assume without loss of generality that $m=1$ , as nothing in the proof changes except notationally (as we assume $m$ is fixed). We apply Proposition 7 and Corollary 8. For notational simplicity, let $Q$ denote the measure on $\mathbb{R}$ that $Y\langle u^{\star},X\rangle$ induces for $X\sim P$ , and $Q_{n}$ similarly for $P_{n}$ . We first show that $\|{\sigma_{n}-\sigma_{\star}}\|_{L^{2}(Q_{n})}$ converges. First, we recall [28, Lemma 3] that $\max_{i\leq n}n^{-1/k}\left\|{X_{i}}\right\|_{2}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}0$ as $n\to\infty$ . Thus there is a (random) $B<\infty$ such that $\max_{i\leq n}\left\|{X_{i}}\right\|_{2}\leq Bn^{1/k}$ for all $n$ . Therefore, Corollary 8 implies that the choice $\delta_{n}=c(\frac{\mathsf{L}B}{n^{1-1/k}})^{1/3}$ satisfies the critical inequality (37), and taking $\gamma^{2}_{n}=\frac{\mathsf{L}^{2}}{n}\sum_{i=1}^{n}\langle u_{n}^{\textup{init}}-u^{\star},X_{i}\rangle^{2}\leq M^{2}\epsilon_{n}^{2}\left\|{X}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}$ , we have that for $t\geq\delta_{n}$ ,

\|{\sigma_{n}-\sigma_{\star}}\|_{L^{2}(\mathbb{P}_{n})}^{2}\lesssim t\delta_{n}+\mathsf{L}^{2}\epsilon_{n}^{2}\left\|{X}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}~{}~{}~{}\mbox{with~{}probability~{}at~{}least}~{}1-\exp\left(-\frac{nt\delta_{n}}{4}\right)

on the event that $\max_{i\leq n}\left\|{X_{i}}\right\|_{2}\leq Bn^{1/k}$ for all $n$ , where we have conditioned (in the probability) on $X_{i}$ . As $\left\|{X}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\mathbb{E}[\left\|{X}\right\|_{2}^{2}]$ , we may choose $t=\delta_{n}\gg 1/n^{1/3}$ and find that the Borell-Cantelli lemma then implies that with probability $1$ , there is a random $C<\infty$ such that

\left\|{\sigma_{n}-\sigma^{\star}}\right\|_{L^{2}(Q_{n})}^{2}\leq C\left(n^{\frac{2}{3k}-\frac{2}{3}}+\epsilon_{n}^{2}\right)

(38)

for all $n$ with probability $1$ .

Finally we argue that $\left\|{\sigma_{n}-\sigma^{\star}}\right\|_{L^{2}(Q)}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}0$ . Let $b<\infty$ be otherwise arbitrary, and let $\mathcal{G}_{b}$ be the collection of $2\mathsf{L}$ -Lipschitz functions on $[-b,b]$ with $\left\|{g}\right\|_{\infty}\leq 1$ and $g(0)$ for $g\in\mathcal{G}_{b}$ , noting that $\sigma_{n}-\sigma^{\star}$ restricted to $[-b,b]$ evidently belongs to $\mathcal{G}_{b}$ . Then we have

$\displaystyle\left\\|{\sigma_{n}-\sigma_{\star}}\right\\|_{L^{2}(Q)}^{2}$
	$\displaystyle=\left\\|{\sigma_{n}-\sigma_{\star}}\right\\|_{L^{2}(Q_{n})}^{2}+\int_{\|t\|\leq b}(\sigma_{n}(t)-\sigma_{\star}(t))^{2}(dQ(t)-dQ_{n}(t))+\int_{\|t\|>b}(\sigma_{n}(t)-\sigma_{\star}(t))^{2}(dQ(t)-dQ_{n}(t))$
	$\displaystyle\leq\left\\|{\sigma_{n}-\sigma_{\star}}\right\\|_{L^{2}(Q_{n})}^{2}+\sup_{g\in\mathcal{G}_{b}}\|Qg^{2}-Q_{n}g^{2}\|+Q([-b,b]^{c})+Q_{n}([-b,b]^{c}),$	(39)

where the inequality used that $\left\|{\sigma_{n}-\sigma_{\star}}\right\|_{\infty}\leq 1$ by construction. The first term in inequality (39) we have already controlled. We may control the second supremum term almost immediately using Dudley’s entropy integral and a Rademacher contraction inequality. Indeed, we have

\mathbb{E}[\sup_{g\in\mathcal{G}_{b}}|Q_{n}g^{2}-Qg^{2}|]\stackrel{{\scriptstyle(i)}}{{\leq}}2\mathbb{E}[\sup_{g\in\mathcal{G}_{b}}|Q_{n}^{0}g^{2}|]\stackrel{{\scriptstyle(ii)}}{{\leq}}4\mathbb{E}[\sup_{g\in\mathcal{G}_{b}}|Q_{n}^{0}|]\stackrel{{\scriptstyle(iii)}}{{\lesssim}}\frac{1}{\sqrt{n}}\int_{0}^{\infty}\sqrt{\log N_{\infty}(t;\mathcal{G}_{b})}dt,

where inequality $(i)$ is a standard symmetrization inequality, $(ii)$ is the Rademacher contraction inequality [24, Ch. 4] applied to the function $t\mapsto t^{2}$ , which is $2$ Lipschitz for $t\in[-1,1]$ , and $(iii)$ is Dudley’s entropy integral bound. As the $\sup$ -norm log-covering numbers of $\mathsf{L}$ -Lipschitz functions on $[-b,b]$ scale as $\frac{\mathsf{L}b}{t}$ for $t\leq\mathsf{L}b$ and are 0 otherwise, we obtain $\mathbb{E}[\sup_{g\in\mathcal{G}_{b}}|Q_{n}g^{2}-Qg^{2}|]\lesssim\frac{\mathsf{L}b}{\sqrt{n}}$ . The bounded-differences inequality then implies that for any $t>0$ ,

\mathbb{P}\left(\sup_{g\in\mathcal{G}_{b}}|(Q_{n}-Q)g^{2}|\geq c\frac{\mathsf{L}b}{\sqrt{n}}+t\right)\leq\mathbb{P}\left(\sup_{g\in\mathcal{G}_{b}}|(Q_{n}-Q)g^{2}|\geq\mathbb{E}[\sup_{g\in\mathcal{G}_{b}}|(Q_{n}-Q)g^{2}|]+t\right)\leq\exp(-cnt^{2}).

Finally, the final term in the bound (39) evidently satisfies $Q([-b,b]^{c})\leq\frac{\mathbb{E}[\left\|{X}\right\|_{2}^{k}]}{b^{k}}$ and $\sup_{b}|Q_{n}([-b,b]^{c})-Q([-b,b])|\leq 2\sqrt{t/n}$ with probability at least $1-e^{-2t^{2}}$ by the DKW inequality. We thus find by the Borel-Cantelli lemma that simultaneously for all $b<\infty$ , with probability at least $1-e^{-nt^{2}}$ we have

\left\|{\sigma_{n}-\sigma_{\star}}\right\|_{L^{2}(Q)}^{2}\leq\left\|{\sigma_{n}-\sigma_{\star}}\right\|_{L^{2}(Q_{n})}^{2}+\frac{C\mathsf{L}b}{\sqrt{n}}+Ct+\frac{\mathbb{E}[\left\|{X}\right\|_{2}^{k}]}{b^{k}},

where $C$ is a numerical constant.

Substituting inequality (38) into the preceding display and taking $b=n^{-\frac{1}{2(k+1)}}$ , we get the result.

I.3 Proof of Proposition 7

We begin with an extension of the familiar basic inequality [e.g. 46, Eq. (13.18)], where we see by convexity that for any $\eta>0$ we have

	$\displaystyle\sum_{i=1}^{n}(Y_{i}-\widehat{f}(X_{i}))^{2}$	$\displaystyle\leq(1+\eta)\sum_{i=1}^{n}(Y_{i}-\widehat{f}(\widetilde{X}_{i}))^{2}+(1+1/\eta)\sum_{i=1}^{n}(\widehat{f}(\widetilde{X}_{i})-\widehat{f}(X_{i}))^{2}$
		$\displaystyle\leq(1+\eta)\sum_{i=1}^{n}(Y_{i}-f^{\star}(\widetilde{X}_{i}))^{2}+(1+1/\eta)\sum_{i=1}^{n}(\widehat{f}(\widetilde{X}_{i})-\widehat{f}(X_{i}))^{2}$
		$\displaystyle\leq(1+\eta)^{2}\sum_{i=1}^{n}(Y_{i}-f^{\star}(X_{i}))^{2}+(2+\eta+1/\eta)\sum_{i=1}^{n}\left[(\widehat{f}(\widetilde{X}_{i})-\widehat{f}(X_{i}))^{2}+(f^{\star}(\widetilde{X}_{i})-f^{\star}(X_{i}))^{2}\right].$

Noting that $Y_{i}=f^{\star}(X_{i})+\xi_{i}$ , algebraic manipulations yield that if $\Delta=[f^{\star}(X_{i})-\widehat{f}(X_{i})]_{i=1}^{n}$ is the error vector, then

\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}-\frac{2}{n}\xi^{T}\Delta+\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\leq(1+\eta)^{2}\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}+\frac{2+\eta+1/\eta}{n}\sum_{i=1}^{n}\left[(\widehat{f}(\widetilde{X}_{i})-\widehat{f}(X_{i}))^{2}+(f^{\star}(\widetilde{X}_{i})-f^{\star}(X_{i}))^{2}\right].

Simplifying, we obtain the following:

Lemma I.1.

Let $\gamma^{2}=\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}(f(X_{i})-f(\widetilde{X}_{i}))^{2}$ and $\Delta=[\widehat{f}(X_{i})-f^{\star}(X_{i})]_{i=1}^{n}$ . Then

	$\displaystyle\left\\|{\Delta}\right\\|_{L^{2}(\mathbb{P}_{n})}^{2}$	$\displaystyle\leq\inf_{\eta}\left\{(2\eta+\eta^{2})\left\\|{\xi}\right\\|_{L^{2}(\mathbb{P}_{n})}^{2}+\frac{2}{n}\xi^{T}\Delta+(4+2\eta+2/\eta)\gamma^{2}\right\}$
		$\displaystyle\leq\frac{2}{n}\xi^{T}\Delta+11\gamma\max\left\{\gamma,\left\\|{\xi}\right\\|_{L^{2}(\mathbb{P}_{n})}\right\}.$

Proof.

The first inequality is algebraic manipulations and uses that our choice of $\eta$ was arbitrary. For the second, we consider two cases: that $\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\geq\gamma$ and that $\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\leq\gamma$ . In the first case, we consider $\eta\leq 1$ , yielding the simplified bound that $\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\leq\frac{2}{n}\xi^{T}\Delta+3\eta\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}+\frac{8\gamma^{2}}{\eta}$ for $\eta\leq 1$ . Taking $\eta=\gamma/\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}$ gives that

\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\leq\frac{2}{n}\xi^{T}\Delta+11\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\gamma.

In the case that $\gamma\geq\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}$ , we choose $\eta=1$ , and the bound simplifies to $\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\leq\frac{2}{n}\xi^{T}\Delta+3\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}+8\gamma^{2}\leq\frac{2}{n}\xi^{T}\Delta+11\gamma^{2}$ . ∎

We now return to the proof of the proposition proper. We begin with an essentially immediate extension of the result [46, Lemma 13.12]. We let $\mathcal{H}$ be an arbitrary star-shaped function class. Define the event

A(u)\coloneqq\left\{\mbox{there~{}exists~{}}g\in\mathcal{H}~{}\mbox{s.t.}~{}\left\|{g}\right\|_{L^{2}(\mathbb{P}_{n})}\geq u~{}\mbox{and}~{}\left|P_{n}\xi g\right|\geq 2\left\|{g}\right\|_{L^{2}(\mathbb{P}_{n})}u\right\},

(40)

which we treat conditionally on $X_{1}^{n}=x_{1}^{n}$ as in the definition of the local complexity $\mathcal{R}_{n}$ . (Here the noise $\xi_{i}$ are still random, taken conditionally on $X_{1}^{n}=x_{1}^{n}$ .)

Lemma I.2 (Modification of Lemma 13.12 of Wainwright [46]).

Let $\mathcal{H}$ be a start-shaped function class and let $\delta_{n}>0$ satisfy the critical radius inequality

\frac{1}{\delta}\mathcal{R}_{n}(\delta;\mathcal{H})\leq\delta.

Then for all $u\geq\delta_{n}$ , we have

\mathbb{P}(A(u))\leq\exp\left(-\frac{nu^{2}}{4}\right).

Deferring the proof of Lemma I.2 to Section I.3.1, we can parallel the argument for [46, Thm. 13.5] to obtain our proposition.

Let $\mathcal{H}=\mathcal{F}^{\star}$ in Lemma I.2. Whenever $t\geq\delta_{n}$ , we have $\mathbb{P}(A(\sqrt{t\delta_{n}}))\leq e^{-nt\delta_{n}/4}$ . We consider the two cases that $\|{\Delta}\|_{L^{2}(\mathbb{P}_{n})}\lessgtr\sqrt{t\delta_{n}}$ . In the former that $\|{\Delta}\|_{L^{2}(\mathbb{P}_{n})}\leq\sqrt{t\delta_{n}}$ , we have nothing to do. In the latter, we have $\widehat{f}-f^{\star}\in\mathcal{F}^{\star}$ while $\|{\Delta}\|_{L^{2}(\mathbb{P}_{n})}>\sqrt{t\delta_{n}}$ , so that if $A(\sqrt{t\delta_{n}})$ fails then we must have

\left|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}\Delta(x_{i})\right|\leq 2\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}\sqrt{t\delta_{n}}.

From the extension of the basic inequality in Lemma I.1 we see that

\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}^{2}\leq 4\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}\sqrt{t\delta_{n}}+11\max\left\{\gamma^{2},\gamma\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\right\}.

Solving for $\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}$ then yields

\left\|{\Delta}\right\|_{L^{2}(\mathbb{P}_{n})}\leq\frac{4\sqrt{t\delta_{n}}+\sqrt{16t\delta_{n}+44\gamma\max\{\gamma,\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\}}}{2}\leq 4\sqrt{t\delta_{n}}+\sqrt{11\gamma\max\{\gamma,\left\|{\xi}\right\|_{L^{2}(\mathbb{P}_{n})}\}}.

Simplifying gives Proposition 7.

I.3.1 Proof of Lemma I.2

Mimicking the proof of [46, Lemma 13.12], we begin with [46, Eq. (13.40)]:

\mathbb{P}(A(u))\leq\mathbb{P}(Z_{n}(u)\geq 2u^{2})~{}~{}\mbox{for}~{}~{}Z_{n}(u)\coloneqq\sup_{g\in\mathcal{H},\|{g}\|_{L^{2}(\mathbb{P}_{n})}\leq u}\left|n^{-1}\sum_{i=1}^{n}\xi_{i}g(x_{i})\right|.

Now note that if $\|{g}\|_{L^{2}(\mathbb{P}_{n})}\leq u$ , then the function $\xi\mapsto|n^{-1}\sum_{i=1}^{n}\xi_{i}g(x_{i})|$ is $u/\sqrt{n}$ -Lipschitz with respect to the $\ell_{2}$ -norm, so that convex concentration inequalities [e.g. 46, Theorem 3.4] imply that $\mathbb{P}(Z_{n}(u)\geq\mathbb{E}[Z_{n}(u)]+t)\leq\exp(-\frac{t^{2}n}{4b^{2}u^{2}})$ whenever $\sup\xi-\inf\xi\leq b$ , and so for $b=1$ we have

\mathbb{P}(Z_{n}(u)\geq\mathbb{E}[Z_{n}(u)]+u^{2})\leq\exp\left(-\frac{nu^{2}}{4}\right).

As $\mathbb{E}[Z_{n}(u)]=\mathcal{R}_{n}(u)$ , we finally use that the normalized complexity $t\mapsto\frac{\mathcal{R}_{n}(t)}{t}$ is non-decreasing [46, Lemma 13.6] to obtain that for $u\geq\delta_{n}$ , $\frac{1}{u}\mathcal{R}_{n}(u)\leq\frac{1}{\delta_{n}}\mathcal{R}_{n}(\delta_{n})\leq\delta_{n}$ , the last inequality by assumption. In particular, we find that for $u\geq\delta_{n}$ we have $\mathbb{E}[Z_{n}(u)]=\mathcal{R}_{n}(u)\leq u\delta_{n}\leq u^{2}$ , and so

\mathbb{P}(Z_{n}(u)\geq 2u^{2})\leq\mathbb{P}(Z_{n}(u)\geq\mathbb{E}[Z_{n}(u)]+u^{2})\leq\exp\left(-\frac{nu^{2}}{4}\right),

as desired.

$\displaystyle\nabla L^{\textup{mv}}_{m}(tu^{\star})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{(i)}}}{{=}}-\mathbb{E}\left[\frac{\mathsf{sign}(Z)}{1+\exp(t\|Z\|)}\rho_{m}(t^{\star}\|Z\|)X\right]+\mathbb{E}\left[\frac{\mathsf{sign}(Z)\exp(t\|Z\|)}{1+\exp(t\|Z\|)}(1-\rho_{m}(t^{\star}\|Z\|))X\right]$
	$\displaystyle=-\mathbb{E}\left[\frac{\mathsf{sign}(Z)Z}{1+\exp(t\|Z\|)}\rho_{m}(t^{\star}\|Z\|)\right]u^{\star}+\mathbb{E}\left[\frac{\mathsf{sign}(Z)Z\exp(t\|Z\|)}{1+\exp(t\|Z\|)}(1-\rho_{m}(t^{\star}\|Z\|))\right]u^{\star}$
	$\displaystyle=\left(\mathbb{E}\left[\frac{\|Z\|\exp(t\|Z\|)}{1+\exp(t\|Z\|)}\right]-\mathbb{E}\left[\|Z\|\rho_{m}(t^{\star}\|Z\|)\right]\right)u^{\star}$
	$\displaystyle=\left(\mathbb{E}\left[{\|Z\|(1-\rho_{m}(t^{\star}\|Z\|))}\right]-\mathbb{E}\left[{\frac{\|Z\|}{1+e^{t\|Z\|}}}\right]\right)u^{\star}=h_{m}(t)u^{\star},$	(6)

	$\displaystyle\mathbb{E}\left[{\nabla_{\theta}\log p_{\theta^{\star}}(X,Y^{+})\cdot\nabla_{\theta}\log p_{\theta^{\star}}(X,Y^{+})^{\top}\mid X=x}\right]$
	$\displaystyle=\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\frac{\nabla_{\theta}\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\nabla_{\theta}\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{\top}}{\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{2}}$
	$\displaystyle\qquad+(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)\cdot\frac{\left({\nabla_{\theta}(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)}\right)\cdot\left({\nabla_{\theta}(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)}\right)^{\top}}{(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)^{2}}$
	$\displaystyle=\frac{\rho_{m}^{\prime}(\|\langle x,\theta^{\star}\rangle\|)^{2}}{\rho_{m}(\|\langle x,\theta^{\star}\rangle\|)(1-\rho_{m}(\|\langle x,\theta^{\star}\rangle\|))}\cdot xx^{\top}$

	$\displaystyle\left\|\ell_{\sigma,\theta}(y\mid x)-\ell_{g,v}(y\mid x)\right\|$	$\displaystyle=\left\|\int_{0}^{y\langle x,\theta\rangle}\sigma(-t)dt-\int_{0}^{y\langle x,v\rangle}g(-t)dt\right\|$
		$\displaystyle\leq\sup_{\|t\|\leq r\left\\|{x}\right\\|_{2}}\|\sigma(t)-g(t)\|+\left\|\int_{y\langle x,v\rangle}^{y\langle x,\theta\rangle}\|\sigma(-t)-g(-t)\|dt\right\|$
		$\displaystyle\leq\left\\|{\sigma-g}\right\\|+1\!\left\{{\left\\|{x}\right\\|_{2}\geq R/r}\right\}+\|\langle x,\theta-v\rangle\|$
		$\displaystyle\leq\left\\|{\sigma-g}\right\\|+\left\\|{x}\right\\|_{2}\left\\|{\theta-v}\right\\|_{2}+1\!\left\{{\left\\|{x}\right\\|_{2}\geq R/r}\right\},$

	$\displaystyle\mathbb{E}\left[\sup_{\left\\|{\theta}\right\\|_{2}\leq r}\sup_{\vec{\sigma}\in\mathcal{F}_{\mathsf{link}}^{m}}\|P_{n}\ell_{\vec{\sigma},\theta}-L(\theta,\vec{\sigma})\|\right]$
		$\displaystyle\lesssim\mathbb{E}\left[\max_{\theta\in\Theta_{\epsilon}}\max_{\vec{g}\in\mathcal{G}^{m}}\left\|P_{n}^{0}\ell_{\vec{g},\theta}\right\|\right]+\epsilon+\mathbb{E}[\left\\|{X_{1}}\right\\|_{2}]\epsilon+\frac{1}{n}\sum_{i=1}^{n}\mathbb{P}(\left\\|{X_{i}}\right\\|_{2}\geq R/r)$
		$\displaystyle\stackrel{{\scriptstyle(i)}}{{\lesssim}}\sqrt{\frac{dmR\mathsf{L}}{\epsilon}\log\left(1+\frac{2r}{\epsilon}\right)}\mathbb{E}\left[\frac{r^{2}}{n}\sum_{i=1}^{n}\left\\|{X_{i}}\right\\|_{2}^{2}\right]^{1/2}+\epsilon+\mathbb{E}[\left\\|{X_{1}}\right\\|_{2}]\epsilon+\frac{\mathbb{E}[\left\\|{X_{i}}\right\\|_{2}^{\gamma}]}{(R/r)^{\gamma}}$
		$\displaystyle\leq Mr\sqrt{\frac{dmR\mathsf{L}}{n\epsilon}\log\left(1+\frac{2r}{\epsilon}\right)}+(M+1)\epsilon+\frac{M^{\gamma}}{(R/r)^{\gamma}},$

	$\displaystyle\overline{\sigma}(t)+\overline{\sigma}(-t)$	$\displaystyle=P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(-\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(i)}}}{{=}}P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ P_{m}\circ\left(1-\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)\right)$
		$\displaystyle\stackrel{{\scriptstyle\mathrm{(ii)}}}{{=}}P_{\overline{m}}^{-1}\circ P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)+P_{\overline{m}}^{-1}\circ\left(1-P_{m}\circ\sigma^{\star}\left(\frac{\left\\|{\theta}\right\\|_{2}}{\left\\|{\overline{\theta}}\right\\|_{2}}t\right)\right)$
		$\displaystyle=1,$

How many labelers do you have? A closer look at gold-standard labels

Abstract

1 Introduction

1.1 Problem formulation

The model with multiple labels.

Classification and calibration.

Estimators.

1.2 Summary of theoretical results and implications

Improved performance using multiple labels for well-specified models

Robustness of majority-vote estimators

Fundamental limits of majority vote

Semi-parametric approaches

Experimental evaluation

1.3 Related work

Notation

2 The well-specified logistic model

2.1 The isotropic Gaussian case

Performance with non-aggregated data.

Corollary 1.

Performance with majority-vote aggregation.

Corollary 2.

2.2 Comparisons for more general feature distributions

Assumption A1.

Assumption A2.

Proposition 1.

Proposition 2.

3 Label aggregation and misspecified model

3.1 Master results

Assumption A3.

Minimizer of the population loss.

Lemma 3.1.

Asymptotic normality with multiple labels.

Theorem 1.

Corollary 3 (The well-specified case).

Asymptotic normality with majority vote.

Theorem 2.

3.2 Robustness to model mis-specification

Multi-label estimator.

Corollary 4.

Majority vote estimator.

Assumption A4.

Proposition 3.

4 Fundamental estimation limits of majority vote labels

4.1 Inconsistency from aggregate labels

Proposition 4.

4.2 Fundamental asymptotic limits in the logistic case

Corollary 5.

Lemma 4.1.

Proof.

Theorem 3.

Corollary 6.

5 Semi-parametric approaches

5.1 Master results

Assumption A5.

Lemma 5.1.

Theorem 4.

5.2 A single index model

Proposition 5.

5.3 Crowdsourcing model

Proposition 6.

6 Experiments

6.1 BlueBirds

6.2 CIFAR-10H

6.3 Crowdsourcing methods on semisynthetic CIFAR-10 labels

7 Discussion

Support

References

Appendix A Technical lemmas

Lemma A.1.

Lemma A.2.

Lemma A.3.

Lemma A.4.

A.1 Proof of Lemma A.1

A.2 Proof of Lemma A.2

A.3 Proof of Lemma A.3

Part 1. Normal approximation for 1−ρm​(t)1-\rho_{m}(t) when t=O​(1/m)t=O(1/\sqrt{m}).

Part 2. Approximating (I) by Gaussian cdf.

Part 3. Upper bounding (II).

A.4 Proof of Lemma A.4

Appendix B Proof of Lemma 3.1

How many labelers do you have?
A closer look at gold-standard labels

Part 1. Normal approximation for $1-\rho_{m}(t)$ when $t=O(1/\sqrt{m})$ .

Large $m$ behavior.

Large $m$ behavior.

Part I: Asymptotics for $B_{m}(t)$ .

Part II: Asymptotics for $A_{m}(t)$ .