Anomalous spreading in reducible multitype
branching Brownian motion

Despite the fact that spatial multitype branching processes have a long history, the study of the asymptotic behaviour of their largest displacement has not been considered until recently. As previously mentioned, Biggins [Big12] computed the speed of multitype reducible branching random walks. This process is a discrete-time particles system in which each particle gives birth to offspring in an independent fashion around its position, with a reproduction law that depends on its type, under the assumption that the Markov chain associated to the type of a typical individual in the process is reducible. Ren and Yang [RY14] then considered the asymptotic behaviour of the maximal displacement in an irreducible multitype BBM.

In [Big10], Biggins gives an explicit description of the speed of a reducible two-type branching random walk as the solution of an optimization problem. In the context of the two-type BBM we consider, the optimization problem can be expressed as such:

$$v=\max\left\{pa+(1-p)b:p\in[0,1],\ p\left(\frac{a^{2}}{2\sigma^{2}}-\beta\right)\leq 0,\ p\left(\frac{a^{2}}{2\sigma^{2}}-\beta\right)+(1-p)\left(\frac{b^{2}}{2}-1\right)\leq 0\right\}.$$

(2.1)

This optimization problem can be understood as follows. It is well-known that if $a<\sqrt{2\sigma^{2}\beta}$ and $b\geq\sqrt{2}$, there are with high probability around $e^{pt(\beta-a^{2}/2\sigma^{2})+o(t)}$ particles of type $1$ at time $pt$ to the right of position $pta$, and a typical particle of type $2$ has probability $e^{(1-p)t(1-b^{2}/2)+o(t)}$ of having a descendant to the right of position $(1-p)bt$ at time $(1-p)t$. Therefore, for all $(p,a,b)$ such that

$$p\in[0,1],\ p\left(\frac{a^{2}}{2\sigma^{2}}-\beta\right)\leq 0,\ p\left(\frac{a^{2}}{2\sigma^{2}}-\beta\right)+(1-p)\left(\frac{b^{2}}{2}-1\right)\leq 0,$$

by law of large numbers there should be with high probability particles of type $2$ to the right of the position $t(pa+(1-p)b)$ at time $t$.

If we write $(p^{*},a^{*},b^{*})$ the triplet optimizing the problem (2.1), it follows from classical optimization under constraints computations that:

1. 1.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{I}$, then $p^{*}=1$ and $a^{*}=\sqrt{2\beta\sigma^{2}}$, which is in accordance with Theorem 1.1, as the extremal particle system is dominated by the behaviour of particles of type $1$, and particles of type $2$ contributing to the extremal process are close relatives descendants of a parent of type $1$;

2. 2.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{II}$, then $p^{*}=0$ and $b^{*}=\sqrt{2}$, which is in accordance with Theorem 1.2, as the extremal particle system is dominated by the behaviour of particles of type $2$, that are born at time $o(t)$ from particles of type $1$;

3. 3.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{III}$, then

   $$p^{*}=\frac{\sigma^{2}+\beta-2}{2(1-\sigma^{2})(\beta-1)},\quad a^{*}=\sigma^{2}\sqrt{2\frac{\beta-1}{1-\sigma^{2}}}\ \text{ and }\ b^{*}=\sqrt{2\frac{\beta-1}{1-\sigma^{2}}}.$$

   We then have $v=\frac{\beta-\sigma^{2}}{\sqrt{2(1-\sigma^{2})(\beta-1)}}$, which corresponds to the main result of Theorem 1.3. Additionally, the Lagrange multiplier associated to this optimization problem is $\theta=\sqrt{2\frac{\beta-1}{1-\sigma^{2}}}=b=\frac{a}{\sigma^{2}}$.

In particular, the optimization problem associated to the case $(\beta,\sigma^{2})\in\mathcal{C}_{III}$ can be related to the following interpretation of Theorem 1.3. The extremal process at time $t$ is obtained as the superposition of the extremal processes of an exponentially large number of BBMs of type $2$, starting around time $tp^{*}$ and position $tp^{*}a^{*}$. The number of these BBMs is directly related to the number of particles of type $1$ that displace at speed $a^{*}$, which is known to be proportional to $W_{\infty}(\theta)e^{t(1-(a^{*})^{2}/2\sigma^{2})}$. It explains the apparition of this martingale in Theorem 1.3, whereas the decoration distribution $\mathfrak{D}^{(III)}$ is the extremal process of a BBM of type $2$ conditionally on moving at the speed $b^{*}>v$.

For Theorem 1.1, a similar description can be made. We expect the asymptotic behaviour to be driven by the behaviour of particles of type $1$, therefore the extremal process of particles of type $2$ should be obtained as a decoration of the extremal process of particles of type $1$. However, as we were not able to use result of convergence of extremal processes together with a description of the behaviour of particles at times $t-O(1)$, we do not obtain an explicit value for $c_{(I)}$ and an explicit description of the law $\mathfrak{D}^{(I)}$. However, with similar techniques as the ones used in [ABBS13] or [ABK13], such explicit constructions should be available.

Finally, in the case covered by Theorem 1.2, the above optimization problem indicates that the extremal process of the multitype reducible BBM should be obtained as the superposition of a finite number of BBMs of particles of type $2$, descending from the first few particles of type $2$ to be born. The random variable $\overline{Z}$ is then constructed as the weighted sum of i.i.d. copies of the derivative martingale of a standard BBM and the decoration is the same as the decoration of the original BBM.

To prove Theorems 1.1–1.3, we show that the above heuristic holds, i.e. that with high probability the set of particles contributing to the extremal processes are the one we identified in each case. We then use previously known results of branching Brownian motions to compute the Laplace transforms of the extremal point measures we are interested in.

The solution of the optimization problem (2.1) is also solution of $v=\sup\{a\in\mathbb{R}:g(a)\leq 0\}$, where $g$ is the largest convex function such that

$$\forall|x|\leq\sqrt{2\beta\sigma^{2}},\ g(x)\leq\left(\frac{x^{2}}{2\sigma^{2}}-\beta\right)\quad\text{ and }\quad\forall y\in\mathbb{R},\ g(y)\leq\frac{y^{2}}{2}-1,$$

see [Big10] for precisions. The function $x\mapsto\frac{x^{2}}{2\sigma^{2}}-\beta$ is known as the rate function for particles of type $1$, and $y\mapsto\frac{y^{2}}{2}-1$ is the rate function for particles of type $2$.

We then observe that the three cases described above are the following:

1. 1.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{I}$, then $v=\sqrt{2\beta\sigma^{2}}=\sup\{x\in\mathbb{R}:x^{2}/2\sigma^{2}-\beta\leq 0\}$.

2. 2.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{II}$, then $v=\sqrt{2}=\sup\{y\in\mathbb{R}:\frac{y^{2}}{2}-1\leq 0\}$.

3. 3.

   If $(\beta,\sigma^{2})\in\mathcal{C}_{III}$, then $v>\max(\sqrt{2\beta\sigma^{2}},\sqrt{2})$.

In other words, the anomalous spreading corresponds to the case when the convex envelope $g$ crosses the $x$-axis to the right of the rate functions of particles of type $1$ and $2$.

As mentioned above, Ren and Yang [RY14] studied the asymptotic behaviour of irreducible multitype BBM, and computed the speed at which that process invades its environment. In that case (i.e. when for all pair of types $i$ and $j$, individuals of type $i$ have positive probability of having at least one descendant of type $j$ after some time), this asymptotic behaviour is similar to the one obtained for a single-type BBM, with branching rate and variance obtained by considering the invariant measure of the Markov process describing the type of a typical individual. The notion of anomalous spreading in this case is thus very different, and the ancestral lineage of typical particles in the extremal process will present regular changes of type. As a result, we do not expect an asymptotic behaviour similar to the one observed in Theorem 1.3 to occur in irreducible multitype BBM.

In a different direction, Smadi and Vatutin [SV16] studied the limit in distribution of a critical reducible Galton-Watson process. It is worth noting that similarly to our results, they obtained three different behaviours for the system, with either the domination of particles of the first type, of the second type, or an interplay between the two.

The results presented here, in particular in the anomalous spreading case, are reminiscent of the known asymptotic for the extremal process of time-inhomogeneous branching Brownian motions. This model was introduced by Fang and Zeitouni [FZ12a], and is defined as follows. Given $t\geq 0$, the process is a BBM consisting only of particles of type $1$ until time $t/2$, at which time they all become simultaneously particles of type $2$. It has been showed [FZ12b, Mal15a] that depending on the value of $(\beta,\sigma^{2})$, the position of the maximal displacement at time $t$ can exhibit different types of asymptotic behaviours. In particular, the logarithmic correction exhibit a strong phase transition in the phase space of $(\beta,\sigma^{2})$.

Looking more closely at the convergence of the extremes, Bovier and Hartung [BH14, BH15] obtained the convergence in distribution of the extremal process of the time-inhomogeneous BBM. In particular, for a multitype BBM with parameters $(\beta,\sigma^{2})\in\mathcal{C}_{III}$ such that particles change from type $1$ to type $2$ at time $p^{*}t$, they showed that the extremal process converges towards $\mathcal{E}^{(III)}_{\infty}$, with an extra $\frac{1}{2\theta}\log t$ logarithmic correction for the centring. This is in accordance with our heuristic as we expect that the particles contributing to the extremal process at time $t$ to have been born from particles of type $1$ around time $p^{*}t$.

Generalized versions of time-inhomogeneous BBM have been studied, in which the variance of particles evolves continuously over time [MZ16, Mal15b]. In that case, the maximal displacement grows at constant speed with a negative correction of order $t^{1/3}$. It would be interesting to construct a multitype BBM, possibly with an infinite number of types, that would exhibit a similar phenomenon.

Observe that similarly to the standard BBM, the multitype BBM can be associated to a reaction diffusion equation in the following way. Let $f,g:\mathbb{R}\to[0,1]$ be measurable functions, we define for all $x\in\mathbb{R}$:

	$\displaystyle u(t,x)$	$\displaystyle=\mathbf{E}^{(1)}\left(\prod_{u\in\mathcal{N}^{1}_{t}}f(X_{u}(t)+x)\prod_{u\in\mathcal{N}^{2}_{t}}g(X_{u}(t)+x)\right)$
	$\displaystyle v(t,x)$	$\displaystyle=\mathbf{E}^{(2)}\left(\prod_{u\in\mathcal{N}^{1}_{t}}f(X_{u}(t)+x)\prod_{u\in\mathcal{N}^{2}_{t}}g(X_{u}(t)+x)\right)=\mathbf{E}^{(2)}\left(\prod_{u\in\mathcal{N}^{2}_{t}}g(X_{u}(t)+x)\right)$

where $\mathbf{P}^{(1)}$ (respectively $\mathbf{P}^{(2)}$) is the law of the multitype BBM starting from one particle of type $1$ (resp. $2$), and we use the fact that particles of type $2$ only produce offspring of type $2$, with the usual convention $\prod_{u\in\emptyset}f(X_{u}(t)+x)=1$.

As under $\mathbf{P}^{(2)}$, the process behaves as a standard BBM, the function $v$ is a solution of the classical F-KPP reaction-diffusion equation

$$\partial_{t}v=\frac{1}{2}\Delta v-v(1-v)\quad\text{with }v(0,x)=g(x).$$

(2.2)

To obtain the partial differential equation satisfied by $u$, we observe that under law $\mathbf{P}^{(1)}$ one of the three following events might happen during the first $\mathrm{d}t$ units of time:

• •

  with probability $\beta\mathrm{d}t+o(\mathrm{d}t)$, the original particle of type $1$ branches into two offspring of type $1$ that start i.i.d. processes with law $\mathbf{P}^{(1)}$;

• •

  with probability $\alpha\mathrm{d}t+o(\mathrm{d}t)$ the particle of type $1$ branches into one offspring of type $1$ and one of type $2$, that start independent processes with law $\mathbf{P}^{(1)}$ and $\mathbf{P}^{(2)}$ respectively;

• •

  with probability $1-(\beta+\alpha)\mathrm{d}t$, the particle of type $2$ diffuses as the Brownian motion $\sigma B$ with diffusion constant $\sigma^{2}$

As a result, we have

	$\displaystyle u(t+\mathrm{d}t,x)$	$\displaystyle=\beta\mathrm{d}tu(t,x)^{2}+\alpha\mathrm{d}tu(t,x)v(t,x)+(1-(\beta+\alpha)\mathrm{d}t)\mathbf{E}\left(u(t,x-\sigma B_{\mathrm{d}t})\right)+o(\mathrm{d}t)$
		$\displaystyle=u(t,x)+\mathrm{d}t\left(\frac{\sigma^{2}}{2}\Delta u(t,x)-\beta u(1-u)-\alpha u(1-v)\right).$

This, together with (2.2) show that $(u,v)$ is a solution of the following coupled F-KPP equation

$$\begin{cases}&\partial_{t}u=\frac{\sigma^{2}}{2}\Delta-\beta u(1-u)-\alpha u(1-v)\\ &\partial_{t}v=\frac{1}{2}\Delta v-v(1-v)\\ &u(0,x)=f(x),\quad v(0,x)=g(x).\end{cases}$$

(2.3)

This non-linear coupling of F-KPP equation was introduced by Holzer [Hol14]. In that article, the author conjectured this partial differential equation to exhibit an anomalous spreading phenomenon, and conjectured a phase diagram for the model [Hol14, Figure 1]. Our main results confirm this conjecture, and the diagram we obtain in Figure 1 exactly matches (up to an adaptation of the notation $\sigma^{2}\rightsquigarrow 2d$, $\beta\rightsquigarrow\alpha$ and $\alpha\rightsquigarrow\beta$) the one obtained by Holzer. Additionally, Theorems 1.1–1.3 give the position of the front of $v_{t}$ in (2.3).

When starting with well chosen initial conditions $f$,$g$ (for example such that there exists $A>0$ satisfying $f(x)=g(x)=1$ for $x<-A$ and $f(x)=g(x)=0$ for $x>A$), we obtain the existence of a function $v_{t}$ such that for all $x\in\mathbb{R}$,

$$\lim_{t\to\infty}(u(t,x-v_{t}),v(t,x-v_{t}))=(w_{1}(x),w_{2}(x)),$$

where $(w_{1},w_{2})$ is a travelling wave solution of the coupled PDE and:

1. 1.

   if $(\beta,\sigma^{2})\in\mathcal{C}_{I}$, then $v_{t}=\sqrt{2\beta\sigma^{2}}t-\frac{3}{2\sqrt{2\beta/\sigma^{2}}}\log t$;

2. 2.

   if $(\beta,\sigma^{2})\in\mathcal{C}_{II}$, then $v_{t}=\sqrt{2}t-\frac{3}{2\sqrt{2}}\log t$;

3. 3.

   if $(\beta,\sigma^{2})\in\mathcal{C}_{III}$, then $v_{t}=vt$, with $v$ defined in Theorem 1.3.

Holzer further studied a linearised version of (2.3) in [Hol16], and showed the presence of an anomalous spreading property in that context. However, the phase diagram in that case is of a different nature as the one we obtain in Figure 1. We believe that the phase diagram of this linearised PDE equation should be related to first moment estimates on the number of particles above a given level in the multitype BBM.

The equation (2.3) should also be compared to the partial differential equation studied in [BC14]. In that article, they considered a population with a family of traits indexed by a parameter $\theta\in(\theta_{\min},\theta_{\max})$, that modifies the motility of particles. This was proposed as a model for the invasion of cane toads is Australia, as that population consists of faster individuals, that sacrifice part of their reproduction power as a trade off, and slower individuals that reproduce more easily. The multitype BBM we consider here could then be thought of as some toy-model for this partial differential equation.

We recall that Theorems 1.1–1.3 cover the asymptotic behaviour of the two-type reducible BBM assuming that $(\beta,\sigma^{2})\in\mathcal{C}_{I}\cup\mathcal{C}_{II}\cup\mathcal{C}_{III}$. However, it does not give the asymptotic behaviour of this process when $(\beta,\sigma^{2})$ belongs to the boundary of this set. Understanding the behaviour of the process at these points could help understanding the phase transitions occurring between the different areas of the state space. This would allow results similar to the ones developed in [BH20] for time-inhomogeneous BBM to be considered in reducible multitype BBM.

We conjecture the following behaviours for the branching Brownian motion at the boundary between areas $\mathcal{C}_{I}$ and $\mathcal{C}_{III}$.

Indeed, in this situation, particles $u$ of type $2$ contributing to the extremal process are expected to satisfy $t-T(u)=O(t^{1/2})$. Therefore, the extremal process keeps an intensity driven by the derivative martingale of particles of type $1$, and the decoration point measure is given by the extremal process of a BBM of particles of type $2$ conditioned to travel at speed $\sqrt{2\beta\sigma^{2}}>\sqrt{2}$.

Similarly, at the boundary between areas $\mathcal{C}_{II}$ and $\mathcal{C}_{III}$, the following behaviour is expected.

There, we used the fact that particles $u$ of type $2$ contributing to the extremal process are expected to satisfy $T(u)=O(t^{1/2})$.

In the case when $\beta<1$ and $\sigma^{2}\beta=1$, which corresponds to the boundary between cases $\mathcal{C}_{I}$ and $\mathcal{C}_{II}$, the picture is less clear as at all time $s$ between $0$ and $t$, particles should have the same probability to reach the maximal position, at least to the first order, as the BBM of particles of type $1$ and of particles of type $2$ have same speed.

Further generalisations of the model we consider in this article could be considered. A more general reducible multitype branching Brownian motions with a finite number of states would be expected to exhibit a similar behaviour. One could also allow particles to have different drift coefficients in addition to the different variance terms and branching rates. In that situation, one expects an optimization problem similar to the one studied in [Mal15a] to appear, with a similar resolution of proving that the trajectory followed by particles reaching the maximal position is the same as the one inferred from the solution of the optimization problem.

Proving Theorems 1.1–1.3 for two-type reducible branching Brownian motions in which particles of type $1$ and type $2$ split into a random number of children at each branching event, say $L_{1}$ for particles of type $1$ and $L_{2}$ for particles of type $2$ would be a other natural generalisation of our results. A natural condition to put on the reproduction laws to obtain the asymptotic behaviour observed in Theorem 1.3 is

$$\mathbf{E}(L_{1}\log L_{1})+\mathbf{E}(L_{2}\log L_{2})<\infty.$$

It is worth noting that anomalous spreading might occur even if $\mathbf{E}(L_{2})<1$, i.e. even if the genealogical tree of a particle of type $2$ is subcritical and grows extinct almost surely.

While we only take interest here in the asymptotic behaviour of the extremal particles in this article, we believe that many other features of multitype branching Brownian motions might be of interest, such as the growth rate of the number of particles of type $2$ to the right of $at$ for $a<v$, the large deviations of the maximal displacement $M_{t}$ at time $t$, or the convergence of associated (sub)-martingales.

We begin by introducing the additive martingales of the BBM. For all $\theta\in\mathbb{R}$, the process

$$W_{t}(\theta):=\sum_{u\in\mathcal{N}_{t}}e^{\theta X_{u}(t)-t\left(\tfrac{\theta^{2}}{2}+1\right)},\quad t\geq 0$$

(3.2)

is a non-negative martingale. It is now a well-known fact that the martingale $(W_{t}(\theta),t\geq 0)$ is uniformly integrable if and only if $|\theta|<\sqrt{2}$, and in that case it converges towards an a.s. positive limiting random variable

$$W_{\infty}(\theta):=\lim_{t\to\infty}W_{t}(\theta).$$

(3.3)

Otherwise, we have $\lim_{t\to\infty}W_{t}(\theta)=0$ a.s. This result was first shown by [Nev88]. It can also be obtained by a specific change of measure technique, called the spinal decomposition. This method was pioneered by Lyons, Pemantle and Peres [LPP95] for the study of the martingale of a Galton-Watson process, and extended by Lyons [Lyo97] to spatial branching processes setting.

For all $|\theta|<\sqrt{2}$ the martingale limit $W_{\infty}(\theta)$ is closely related to the number of particles moving at speed $\theta$ in the BBM. For example, by [Big92, Corollary 4], for all $h>0$ we have

$$\lim_{t\to\infty}t^{1/2}e^{t\left(\frac{\theta^{2}}{2}-1\right)}\sum_{u\in\mathcal{N}_{t}}\mathbbm{1}_{\left\{|X_{u}(t)-\theta t|\leq h\right\}}=\frac{2\sinh(\theta h)}{\theta}W_{\infty}(\theta)\quad\text{a.s.}$$

(3.4)

This can be thought of as a local limit theorem result for the position of a particle sampled at random at time $t$, where a particle at position $x$ is sampled with probability proportional to $e^{\theta x}$. A Donsker-type theorem was obtained in [Pai18, Section C] for this quantity, see also [GKS18]. In particular, for any continuous bounded function $f$, one has

$$\lim_{t\to\infty}\sum_{u\in\mathcal{N}_{t}}f\left(\tfrac{X_{u}(t)-\theta t}{t^{1/2}}\right)e^{\theta X_{u}(t)-t\left(\frac{\theta^{2}}{2}+1\right)}=\frac{W_{\infty}(\theta)}{\sqrt{2\pi}}\int_{\mathbb{R}}e^{-\frac{z^{2}}{2}}f(z)\mathrm{d}z\text{ a.s.}$$

(3.5)

This justifies the fact that the variable $W_{\infty}(\theta)$ appears in the limiting distribution of the extremal process in the anomalous spreading case, by the heuristics described in Section 2.1.

To prove Theorem 1.3, we use the following slight generalization of the above convergence.