Inference via robust optimal transportation: theory and methods

Optimal transportation (OT) dates back to the work of [43], who considered the problem of finding the optimal way to move given piles of sand to fill up given holes of the same total volume. Monge’s problem remained open until it was revisited and solved by Kantorovich, who characterized the optimal transportation plan in relation to the economic problem of optimal allocation of resources. We refer to [57], [53] for a book-length presentation in mathematics, to [47] for a data science view, and to [46] for a statistical perspective.

Within the OT setting, the Wasserstein distance ($W_{p},p\geq 1$) is defined as the minimum cost of transferring the probability mass from a source distribution to a target distribution, for a given transfer cost function. Nowadays, OT theory and $W_{p}$ are of fundamental importance for many scientific areas, including statistics, econometrics, information theory and machine learning. For instance, in machine learning, OT based inference is a popular tool in generative models; see e.g. [3], [23], [45] and references therein. Similarly, OT techniques are widely-applied for problems related to domain adaptation; see [15] and [4]. In statistics and econometrics, estimation methods based on $W_{p}$ are related to minimum distance estimation (MDE); see [5] and [8]. MDE is a research area in continuous development and the estimators based on this technique are called minimum distance estimators. They are linked to M-estimators and may be robust. We refer to [29], [55] and [7] for a book-length statistical introduction. As far as econometrics is concerned, we refer to [34] and [30]. The extant approaches for MDE are usually defined via Kullback-Leibler (KL) divergence, Cressie-Read power divergence, total variation (TV) distance, Hellinger distance, to mention a few. Some of those approaches (e.g. Hellinger distance) require kernel density estimation of the underlying probability density function. Some others (e.g. KL divergence) have poor performance when the considered distributions do not have a common support. Compared to other notions of distance or divergence, $W_{p}$ avoids the mentioned drawbacks, e.g. it can be applied when the considered distributions do not share the same support and inference is conducted in a generative fashion—namely, a sampling mechanism is considered, in which non-linear functions map a low dimensional latent random vector to a larger dimensional space; see [23]. This explains why $W_{p}$ is a popular inference tool; see e.g. [15, 22, 35, 12, 47]; see [25] and [37] for a recent overview of technology transfers among different areas.

In spite of their appealing theoretical properties, OT and $W_{p}$ have two important inference aspects to deal with: (i) implementation cost and (ii) lack of robustness. (i) The computation of the solution to OT problem and of the computation of $W_{p}$ can be demanding. To circumvent this problem, [16] proposes the use of an entropy regularized version of the OT problem. The resulting divergence is called Sinkhorn divergence; see [2] for further details. [23] illustrate the use of this divergence for inference in generative models, via MDE. The proposed estimation method is attractive in terms of performance and speed of computation. However (to the best of our knowledge) the statistical guarantees of the minimum Sinkhorn divergence estimators have not been provided. (ii) Robustness is becoming a crucial aspect for many complex statistical and machine learning applications. The reason is essentially two-fold. First, large datasets (e.g. records of images or large-scale data collected over the internet) can be cluttered with outlying values (e.g. due to recording device failures). Second, every statistical model represents only an approximation to reality. Thus, the need for inference procedures that remain informative even in the presence of small deviations from the assumed statistical (generative) model is in high demand, in statistics and in machine learning. Robust statistics deals with this problem. We refer to [29] and [51] for a book-length discussion on robustness principles.

Other papers have already mentioned the robustness issues of $W_{p}$ and OT; see e.g. [1], [26],[27],[28] [18], [44], [60] and [50]. In the OT literature, [13] proposes to deal with outliers via unbalanced OT, namely allowing the source and target distribution to be non-standard probability distributions. This sensitivity is an undesired consequence of exactly satisfying the marginal constraints in OT transportation problem: intuitively, using Monge’s formulation, the piles of sand and the holes need to have the same total volume, thus the optimal transportation plan is affected by the outlying values inducing a high transportation cost. Relaxing the exact marginal constraints in a way that the transportation plan does not assign large weights to outliers yields the unbalanced OT theory: a mass variation is allowed in the OT problem (intuitively, the piles of sand and the holes do not need to have the same volume) and outlying values can be ignored by the transportation plan. Working along the lines of this approach, [4] define a robust optimal transportation (ROBOT) problem and study its applicability to machine learning. In the same spirit, starting from OT formulation with a regularization term which relaxes the marginal constraints, [44] derive a novel robust OT problem, that they label as ROBOT and they apply it to machine learning. Moreover, building on the Minimum Kantorovich Estimator (MKE) of [5], Mukherjee et al. propose a class of estimators obtained through minimizing the ROBOT and illustrate numerically their performance. In the same spirit, recently, [45] defined a notion of outlier-robust Wasserstein distance which allows for an $\epsilon$-percentage of outlier mass to be removed from contaminated distributions.

Working at the intersection between mathematics, probability, statistics, machine learning and econometrics, we study the theoretical, methodological and computational aspects of ROBOT. Our contributions to these different research areas can be summarized as follows.

We prove that the primal ROBOT problem of [44] defines a robust distance that we call robust Wasserstein distance $W^{(\lambda)}$, which depends on a tuning parameter $\lambda$ controlling the level of robustness. We study the measure theoretic properties of $W^{(\lambda)}$, proving that it induces a metric space whose separability and completeness are proved. Moreover, we derive concentration inequalities for the robust Wasserstein distance and illustrate their practical applicability for the selection of $\lambda$ in the presence of outliers.

These theoretical (in mathematics and probability) findings are instrumental to another contribution (in statistics and econometrics) of this paper: the development of a novel inferential theory based on robust Wasserstein distance. Making use of convex analysis [49] and of the techniques in [8], we prove that the minimum robust Wasserstein distance estimators exist (almost surely) and are consistent (at the reference model, possibly misspecified). Numerical experiments show that the robust Wasserstein distance estimators remain reliable even in the presence of local departures from the postulated statistical model. These results complement the methodological investigations already available in the literature on minimum distance estimation (in particular the MKE of [5] and [8]) and on ROBOT ([4] and [44]). Indeed, beside the numerical evidence available in the mentioned papers, there are (to the best of our knowledge) no statistical guarantees on the estimators obtained by the method of [44] and [4]. Moreover, our results extend the use of OT to conduct reliable inference on random variables with infinite moments.

Another contribution is the derivation of the dual form of ROBOT for application to machine learning problems. This result not only completes the theory in [44], but also provides the stepping stone for the implementation of ROBOT: we explain how our duality can be applied to define robust Wasserstein Generative Adversarial Networks (RWGAN). Numerical exercises provide evidence that RWGAN have better performance than the extant Wasserstein GAN (WGAN) models. Moreover, we illustrate how our ROBOT can be applied to domain adaptation, another popular machine learning problem; see Section 5.4 and Appendix .4.1.

Finally, as far as the approach in [45] is concerned, we remark that their notion of robust Wasserstein distance has a similar spirit to our $W^{(\lambda)}$ (both are based on a total variation penalization of the original OT problem, but yield different duality formulations) and, similarly to our results in §5.3, it is proved to be useful in generative adversarial network (GAN). In spite of these similarities, our investigation goes deeper in the probabilistic and statistical aspects related to inference methods based on $W^{(\lambda)}$. With this regard, some key difference between [45] and our paper are that Nietert et al.: (a) do not study concentration inequalities for their distance; (b) do not provide indications on how to select and control the $\epsilon$-percentage of outlier mass ignored by the OT solution—for a numerical illustration of this aspect in the setting of GAN, see Section 5.3; (c) do not define the estimators related to the minimization of their outlier-robust Wasserstein distance; (d) use regularization in the dual and a constraint in the primal, whereas ROBOT does the reverse: this simplifies our derivation of the dual problem. In the next sections we take care of all these and other aspects for the ROBOT, providing the theoretical and the applied statistician with a novel toolkit for robust inference via OT.

We recall briefly the fundamental notions of OT as formulated by Kontorovich; more details are available in Appendix .1. Let $\mathcal{X}$ and $\mathcal{Y}$ denote two Polish spaces, and let $\mathcal{P}(\mathcal{X})$ represent the set of all probability measures on $\mathcal{X}$. Let $\Pi(\mu,\nu)$ denote the set of all joint probability measures of $\mu\in\mathcal{P(X)}$ and $\nu\in\mathcal{P(Y)}$. Kontorovich’s problem aims at finding a joint distribution $\pi\in\Pi(\mu,\nu)$ which minimizes the expectation of the coupling between $X$ and $Y$ in terms of the cost function $c$. This problem can be formulated as

A solution to Kantorovich’s problem (KP) (1) is called an optimal transport plan. Note that the problem is convex, and its solution exists under some mild assumptions on $c$, e.g., lower semicontinuous; see e.g. [57, Chapter 4].

KP as in (1) has a dual form, which is related to Kantorovich dual (KD) problem

for $\forall(x,y)$, where $C_{b}(\mathcal{X})$ is the set of bounded continuous functions on $\mathcal{X}$. According to Th. 5.10 in [57], if function $c$ is lower semicontinuous, there exists a solution to the dual problem such that the solutions to KD and KP coincide (no duality gap). In this case, the solution is

where the functions $\phi$ and $\psi$ are called $c$-concave and $c$-convex, respectively, and $\phi$ (resp. $\psi$) is called the $c$-transform of $\psi$ (resp. $\phi$). When $c$ is a metric on $\mathcal{X}$, OT problem becomes $\sup\left\{\int_{\mathcal{X}}\psi d\nu-\int_{\mathcal{X}}\psi d\mu:\psi\,\text{is 1-Lipschitz continuous}\right\}$, which is the Kantorovich-Rubenstein duality. Now, let $(\mathcal{X},d)$ denote a complete metric space equipped with a metric $d:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$, and let $\mu$ and $\nu$ be two probability measures on $\mathcal{X}$. Solving the optimal transport problem in (1), with the cost function $c(x,y)=d^{p}(x,y)$, introduces a distance, called Wasserstein distance of order $p$ $(p\geq 1)$:

With regard to (2 ), we emphasize that the ability to lift the ground distance $d(x,y)$ is one of the perks of $W_{p}$ and it makes it a suitable tool in statistics and machine learning; see Remark 2.18 in [47]. Interestingly, this desirable feature becomes a negative aspect as far as robustness is concerned. Intuitively, OT embeds the distributions geometry: when the underlying distribution is contaminated by outliers, the marginal constraints force OT to transport outlying values, inducing an undesirable extra cost, which in turns entails large changes in the $W_{p}$. More in detail, let $\delta_{x_{0}}$ be point mass measure centered at $x_{0}$ (a point in the sample space), and $d(x,y)$ be the ground distance. Then, we introduce the $O\cup I$ framework, in which the majority of the data $(X_{i})_{i\in I}$ are inliers, that is i.i.d. informative data with distribution $\mu$, whilst a few data $(X_{i})_{i\in O}$ are outliers, meaning that they are not i.i.d. with distribution $\mu$. We refer to [39] (and references therein) for more detail on $O\cup I$ and for discussion on its connection to Huber gross-error neighbourhood. Thus, for $\varepsilon>0$, we have $W_{p}({\rm\mu},(1-\varepsilon){\rm\mu}+\varepsilon\delta_{x_{0}})\rightarrow\infty$, as $x_{0}\rightarrow\infty$. Thus, $W_{p}$ can be arbitrarily inflated by a small number of large observations and it is sensitive to outliers. For instance, let us consider $W_{p}$, with $p=1,2$. In Figure 1(a) and 1(b), we display the scatter plot for two bivariate distributions: the distribution in panel (b) has some abnormal points (contaminated sample) compared to the distribution in panel (a) (clean sample). Although only a small fraction of the sample data is contaminated, both $W_{1}$ and $W_{2}$ display a large increase. Anticipating some of the results that we will derive in the next pages, we compute our $W^{(\lambda)}$ (see §3.1) and we notice that its value remains roughly the same in both panels. We refer to §5.1 for additional insights on the resistance to outliers of $W^{(\lambda)}$.

We recall the primal formulation of ROBOT problem, as defined in [44], who introduce a modified version of Kantorovich OT problem and work on the notion of unbalanced OT. Specifically, they consider a TV-regularized OT re-formulation

where $\lambda>0$ is a regularization parameter. In (3), the original source measure $\mu$ is modified by adding $s$ (nevertheless, the first and last constraints ensure that $\mu+s$ is still a valid probability measure). Intuitively, having $\mu(x)+s(x)=0$ means that $x\in{\cal X}$ has strong impact on the OT problem and hence can be labelled as an outlier. The outlier is eliminated from the sample, since the probability measure $\mu+s$ at this point is zero.

[44] prove that a simplified, computationally efficient formulation equivalent to (3) is

which has a functional form similar to (1), but the cost function $c$ is replaced by the trimmed cost function $c_{\lambda}=\min\left\{c,2\lambda\right\}$ that is bounded from above by $2\lambda$. Following [44], we call formulations (3) and (4) ROBOT. Beside the primal formulation, in the following theorem we drive the dual form of the ROBOT, which is not available in [44]. We need this duality for the development our inferential approach and, for ease of reference, we state it in form of theorem—the proof can be obtained from Th. 5.10 in [57] and it is available in Appendix .2.

Note that the difference between the dual form of OT and that of ROBOT is that the latter imposes a restriction on the range of $\psi$. Thanks to this, we may think of using the ROBOT in theoretical proofs and inference problems where the dual form of OT is already applied, with the advantage that ROBOT remains stable even in the presence of outliers. In the next sections, we illustrate these ideas and we start the construction of our inference methodology by proving that to ROBOT is associated a robust distance, which will be the pivotal tool of our approach.

Setting $c(x,y)=d(x,y)$, (2) implies that the corresponding minimal cost $W_{1}(\mu,\nu)$ is a metric between two probability measures $\mu$ and $\nu$. A similar result can be obtained also in the ROBOT setting of (4), if the modified cost function takes the form

Based on Lemma 3.1, we define the robust Wasserstein distance and prove that it is a metric on $\mathcal{P(X)}$.

It is worth noting that $W_{p}$ is well defined only if probability measures have finite $p$-th order moments. In contrast, thanks to the boundedness of the $c_{\lambda}$, our $W^{(\lambda)}$ in (7) can be applied to all probability measures, even those with infinite moments of any order. Making use of $W^{(\lambda)}$ we introduce the following

In this section we state, briefly, some preliminary results. The proofs (which are based on [57]) of these results are available in Appendix .2, to which we refer the reader interested in the mathematical details. Here, we provide essentially the key aspects, which we prefer to state in the form of theorems and corollary for the ease of reference in the theoretical developments available in the next sections.

Equipped with Th. 3.2 and Definition 3.3, it is interesting to connect $W_{1}$ to $W^{(\lambda)}$, via the characterization of its limiting behavior as $\lambda\rightarrow\infty$. Since $W^{(\lambda)}$ is continuous and monotonically increasing with respect to $\lambda$, one can prove (via dominated convergence theorem) that its limit coincides with the Wasserstein distance of order $p=1$. Thus:

Th. 3.4 has a two-fold implication: first, $W^{(\lambda)}\leq W_{1}$; second, for large values of the regularization parameter, $W^{(\lambda)}$ and $W_{1}$ behave similarly. Another important property of $W^{(\lambda)}$ is that weak convergence is entirely characterized by convergence in the robust Wasserstein distance. More precisely, we have

The next result follows immediately from Theorem 3.5.

Finally, we prove the separability and completeness of robust Wasserstein space, when $\mathcal{X}$ is separable and complete. To this end, we state

We remark that our dual form in Th. 2.1 unveils a connection between $W^{(\lambda)}$ and the Bounded Lipschitz (BL) metric discussed in [19, p.41 and p.42]: the two metrics coincide for a suitable selection of $\lambda$. With this regard, we emphaisze that, since the BL meterizes the weak-star topology, Th. 3.2 and Th. 3.5 are in line with the results in Section 3 of [19].

The previous results offer the mathematical toolkit needed to derive some concentration inequalities for $W^{(\lambda)}$ building on the extant results for $W_{1}$. To illustrate this aspect, let us consider the case of Talagrand transportation inequality $T_{p}$. We recall that using $W_{p}$ as a distance between probability measures, transportation inequalities state that, given $\alpha>0$, a probability measure $\mu$ on $X$ satisfies $T_{p}(\alpha)$ if the inequality

holds for any probability measure $\nu$, with $H(\mu|\nu)$ being the Kullback-Leibler divergence. For $p=1$, it has been proven that $T_{1}(\alpha)$ is equivalent to the existence of a square-exponential moment; see e.g. [10]. Now, note that, from Th. 3.4, we have $W^{(\lambda)}\leq W_{1}$: we can immediately claim that if $T_{1}(\alpha)$ holds, then we have

Thus, we state

The proof follows along the lines as the proof of Th. 2.1 in Bolley et al. combined with Theorem 2.1 which yields

We remark that condition $n\geq n_{0}\max(\varepsilon^{-\left(d^{\prime}+2\right)},1)$ implies that Th. 3.8 has an asymptotic nature. Nevertheless, Bolley et al. prove that

hods for any $n$ with $C(\epsilon)$ being a (large) constant depending on $\epsilon$. The argument in (8) implies

for any $n\in\mathbb{N}$. This yields a concentration inequality which is valid for any sample size.

We obtained the results in Th. 3.8 using an upper bound on $W_{1}$. The simplicity of our argument comes with a cost: for Th. 3.8 to hold, the underlying measure $\mu$ needs to satisfy some conditions required for $W_{1}$, like e.g. some moment restrictions. To obtain bounds that do not need these stringent conditions, in the next theorem we derive novel non asymptotic upper bounds working directly on $W^{(\lambda)}$. The techniques that we apply to obtain our new results put under the spotlight the role of $\lambda$ to control the concentration in the $O\cup I$ framework.

Some remarks are in order. (i) In the case where $O=\varnothing$, (9) holds without any assumption on $\mu$ as the variance term $\sigma^{2}$ is always bounded, even without assuming a finite second moment of $X_{i}$. This strongly relaxes the exponential square moment assumption required for $W_{1}(\mu,\hat{\mu}_{n})$ in Th. 3.8.

(ii) (10) illustrates that the presence of contamination entails the extra term $2\lambda|O|/n$: one can make use of the tuning parameter $\lambda$ to mitigate the influence of outlying values.

(iii) In (10), it is always possible to replace the term

by either ${\rm E}[W^{(\lambda)}(\mu,\hat{\mu}_{n})]$ or ${\rm E}[W^{(\lambda)}(\mu,\hat{\mu}^{\prime}_{n})]$, where $\mu_{n}^{\prime}=n^{-1}\sum_{i=1}^{n}\delta_{X_{i}^{\prime}}$ and $X_{1}^{\prime},\ldots,X_{n}^{\prime}$ are i.i.d. with common distribution $\mu$. This can be proved noticing that it is possible to bound $W^{(\lambda)}(\mu,\hat{\mu}_{n})$ exploiting the fact that $c_{\lambda}$ is bounded by $2\lambda$, so Th. 2.1 implies

(iv) Delving more into the link between (10) and (11), consider that, for any $\psi\in C_{b}(\mathcal{X})$,

As $\text{range}(\psi)\leqslant 2\lambda$, for any $i\in O$, it holds that

Therefore, we have, for any $\psi\in C_{b}(\mathcal{X})$,

Taking the suprema over $\psi$ in $C_{b}(\mathcal{X})$ yields (11). From (11) and the triangular inequality, we have that (adding and subtracting $\frac{|I|}{n}W^{(\lambda)}(\mu,\hat{\mu}^{(I)}_{n})$)

Inequality (10) follows by plugging (9) into this last bound. Finally, combining (10) and (11), we obtain

Beside the concentrations in Th. 3.9, we consider the problem of providing an upper bound to ${\rm E}[W^{(\lambda)}(\mu,\hat{\mu}_{n}^{(I)})]$. Following [9], we call this quantity the mean rate of convergence. For the sake of exposition, we assume that all data are inliers, that is $X_{1},\ldots,X_{n}$ are i.i.d. with common distribution $\mu$—the case with outliers can be obtained using (11). As in Th. 3.8, we notice that it is always possible to use $W^{(\lambda)}\leqslant W_{1}$ from Th. 3.4 and derive bounds for ${\rm E}[W^{(\lambda)}(\mu,\hat{\mu}_{n})]$ applying the results for ${\rm E}[W_{1}(\mu,\hat{\mu}_{n})]$; see e.g. [9], [21, 40, 20]. Beside this option, here we derive novel results for ${\rm E}[W^{(\lambda)}(\mu,\hat{\mu}_{n})]$. In what follows, we assume that $\mu$ is supported in $B(0,K)$, the ball in Euclidean distance of $\mathbb{R}^{d}$ with radius $K$ and that the reference distance is the Euclidean distance in $\mathbb{R}^{d}$. Then we state:

The bound is obtained using (i) the duality formula derived in Th. 2.1, (ii) an extension of Dudley’s inequality in the spirit of [9], and (iii) a computation of the entropy number of a set of $1$-Lipschitz functions defined on the ball $B(0,K)$ and taking value in $[-\lambda,\lambda]$. The technical details are available in Appendix .2. Here, we remark the key role of $\lambda$ when $d=1$ (univariate case), which allows to bound the mean rate of convergence. For $d\geq 2$, our bounds could have been obtained also from [9]. However, the obtained results can be substantially larger than our upper bounds when $\lambda\ll K$: Th. 3.10 provides a refinement (related to the boundedness of $c_{\lambda}$) of the extant bounds; see also Proposition 9 in [45] for a similar result.

Let us consider a probability space $(\Omega,\mathcal{F},{\rm P})$. On this probability space, we define random variables taking values on $\mathcal{X}\subset\mathbb{R}^{d},d\geq 1$ and endowed with the Borel $\sigma$-algebra. We observe $n\in\mathbb{N}$ i.i.d. data points $\{X_{1},\ldots,X_{n}\}$, which are distributed according to $\mu_{\star}^{(n)}\in\mathcal{P}\left(\mathcal{X}^{n}\right)$. A parametric statistical model on $\mathcal{X}^{n}$ is denoted by $\{\mu_{\theta}^{(n)}\}_{\theta\in\Theta}$ and it is a collection of probability distributions indexed by a parameter $\theta$ of dimension $d_{\theta}$. The parameter space is $\Theta\subset\mathbb{R}^{d_{\theta}},d_{\theta}\geq 1$, which is equipped with a distance $\rho_{\Theta}$. We let $Z_{1:n}$ represent the observations from $\mu_{\theta}^{(n)}$. For every $\theta\in$ $\Theta$, the sequence $\left(\hat{\mu}_{\theta,n}\right)_{n\geq 1}$ of random probability measures on $\mathcal{X}$ converges (in some sense) to a distribution $\mu_{\theta}\in\mathcal{P}(\mathcal{X})$, where

with $Z_{1:n}\sim\mu_{\theta}^{(n)}$. Similarly, we will assume that the empirical measure $\hat{\mu}_{n}$ converges (in some sense) to some distribution $\mu_{\star}\in\mathcal{P}(\mathcal{X})$ as $n\rightarrow\infty$. We say that the model is well-specified if there exists $\theta_{\star}\in\Theta$ such that $\mu_{\star}\equiv\mu_{\theta_{\star}}$; otherwise, it is misspecified. Parameters are identifiable: $\theta_{1}=\theta_{2}$ is implied by $\mu_{\theta_{1}}=\mu_{\theta_{2}}$.

Our estimation method relies on selecting a parametric model in $\{\mu_{\theta}^{(n)}\}_{\theta\in\Theta}$, which is the closest, in robust Wasserstein distance, to the true model $\mu_{\star}$. Thus, minimum $W^{(\lambda)}$ estimation refers to the minimization, over the parameter $\theta\in\Theta$, of the robust Wasserstein distance between the empirical distribution $\hat{\mu}_{n}$ and the reference model distribution $\mu_{\theta}$. This is similar to the approach described in [5] and [8], who derive minimum Kantorovich estimators by making use of $W_{p}$.

More formally, the minimum robust Wasserstein estimator (MRWE) is defined as

When there is no explicit expression for the probability measure characterizing the parametric model (e.g. in complex generative models, see [23] and § 5.2 for some examples), the computation of MRWE can be difficult. To cope with this issue, in lieu of (13), we propose using the minimum expected robust Wasserstein estimator (MERWE) defined as

where the expectation $\mathrm{E}_{m}$ is taken over the distribution $\mu^{(m)}_{\theta}$. To implement the MERWE one can rely on Monte Carlo methods and approximate numerically ${\rm E}_{m}[W^{(\lambda)}\left(\hat{\mu}_{n},\hat{\mu}_{\theta,m}\right)]$. Replacing the robust Wasserstein distance with the $W_{p}$ in (14), one obtains the minimum Wasserstein estimator (MWE) and the minimum expected Wasserstein estimator (MEWE), studied in [8].

Intuitively, the consistency of the MRWE and MERWE can be conceptualized as follows. We expect that the empirical measure converges to $\mu_{\star}$, in the sense that $W^{(\lambda)}(\hat{\mu}_{n},\mu_{\star})\rightarrow 0$ as $n\rightarrow\infty$; see Th. 3.5. Therefore, the $\arg\min$ of $W^{(\lambda)}(\hat{\mu}_{n},\mu_{\star})$ should converge to

assuming its existence and unicity. The same can be said for the minimum of the MERWE, provided that $m\rightarrow\infty$. If the reference parametric model is correctly specified (e.g. no data contamination), $\theta^{\ast}$ is the limiting object of interest and it is the minimizer of $W^{(\lambda)}(\mu_{\star},\mu_{\theta})$. In the case of model misspecification (e.g. wrong parametric form of $\mu_{\theta}$ and/or presence of data contamination), $\theta^{\ast}$ is not necessarily the parameter that minimizes the KL divergence between the empirical measure and the measure characterizing the reference model. We emphasize that this is at odd with the standard misspecification theory (see e.g. [59]) and it is due to the fact that we replace the KL divergence (which yields non robust estimators) with our novel robust Wasserstein distance.

To formalize these arguments, we introduce the following set of assumptions, which are standard in the literature on MKE; see [8].

Then we state the following

To prove Th. 4.4 we need to show that the sequence of functions $\theta\mapsto W^{(\lambda)}\left(\hat{\mu}_{n}(\omega),\mu_{\theta}\right)$ epi-converges (see Definition .5 in Appendix .2) to $\theta\mapsto W^{(\lambda)}\left(\mu_{\star},\mu_{\theta}\right)$. The result follows from [49]. We remark that Th. 4.4 generalizes the results of [5] and [8], where the model is assumed to be well specified ([5]) and moments of order $p\geq 1$ are needed ([8]).

Moving along the same lines as in Th. 2.1 in [8], we may prove the measurability of MRWE; see Th. .6 in Appendix A. These results for the MRWE provide the stepping stone to derive similar theorems for the MERWE $\hat{\theta}_{n,m}^{\lambda}$. To this end, the following assumptions are needed.

For a generic function $f$, let us define

Then, we state the following

Considering the case where the data and $n$ are fixed, the next result shows that the MERWE converges to the MRWE, as $m\rightarrow\infty$. In the next assumption, the observed empirical distribution is kept fix and $\varepsilon_{n}=\inf_{\theta\in\Theta}W^{(\lambda)}\left(\hat{\mu}_{n},\mu_{\theta}\right)$.

To complement the insights obtained looking at Figure 1, we build on the notion of sensitive curve, which is an empirical tool that illustrates the stability of statistical functionals; see [54] and [29] for a book-length presentation. We consider a similar tool to study the sensitivity of $W_{1}$ and $W^{(\lambda)}$ in finite samples, both interpreted as functionals of the empirical measure. More in detail, we fix the standard normal distribution as a reference model (for this distribution both $W^{(\lambda)}$ and $W_{1}$ are well-defined) and we use it to generate a sample of size $n=1000$. We denote the resulting sample $\mathbf{X}_{n}:=(X_{1},\ldots,X_{n})$, whose empirical measure is $\hat{\mu}_{n}$. Then, we replace $X_{n}$ with a value $x\in\mathbb{R}$ and form a new set of sample points, denoted by $\mathbf{X}_{n}(x)$, whose empirical measure is $\hat{\mu}^{(x)}_{n}$. We let $T_{n}$ represent the robust Wasserstein distance between an $n$-dimensional sample and $\mathbf{X}_{n}$. So, $T_{n}(\mathbf{X}_{n})$ is the empirical robust Wasserstein distance between $\mathbf{X}_{n}$ and its self, thus it is equal to $0$, while $T_{n}(\mathbf{X}_{n}(x))$ is the empirical robust Wasserstein distance between $\mathbf{X}_{n}(x)$ and $\mathbf{X}_{n}$. Finally, for different values of $x$ and $\lambda$, we compute (numerically)