Fluctuation theorems from Bayesian retrodiction

As mentioned, studies of irreversibility involve a comparison between a forward ($F$) and a reverse ($R$) process Sei (05). Consider the evolution of a system from an initial time $t=0$ to a final time $t=\tau>0$. Let us assume, for simplicity, that the system possesses a finite state space $\mathcal{A}$. Suppose now that both forward and reverse processes, respectively, are given to us in terms of two suitable joint probability distributions, $P_{F}(x,y)$ and $P_{R}(x,y)$, respectively, where $x\in\mathcal{A}$ labels the state of the system at time $t=0$, while $y\in\mathcal{A}$ labels the state of the system at time $t=\tau$. Concretely, $P_{F}(x,y)$ denotes the probability that the system starts in state $x$ at time $t=0$ and ends in state $y$ at time $t=\tau$ under the forward process, while $P_{R}(x,y)$ denotes the probability that the system starts in state $y$ at time $t=\tau$ and ends in state $x$ at time $t=0$ under the reverse process. Notice that at this point we are not yet preoccupied with the problem of determining what makes the two processes one the “reverse” of the other. This will be the main issue in the rest of the paper, but for the time being we are just considering $P_{F}(x,y)$ and $P_{R}(x,y)$ as given.

If really $P_{F}(x,y)$ and $P_{R}(x,y)$ are one the reverse of the other, then irreversibility may be quantified in terms of how much the two distributions differ, in agreement with the idea that a reversible situation should correspond to the case in which $P_{F}(x,y)=P_{R}(x,y)$ for all $x,y\in\mathcal{A}$. In mathematical statistics, the degree of “disagreement” of two distributions is captured in terms of information divergences. Here we focus on the family of $f$-divergences, defined as¹¹1In fact, $f$-divergences are usually defined in terms of $1/r$, see e.g. LM (08). However, for the sake of the present discussion, formulas are more easily recognizable with the alternative definition. Csi (67)

where $f:\mathbb{R}^{+}\to\mathbb{R}$ is a function to be specialized further in what follows. For the time being, it suffices to notice that for $f(r)=\ln r$ one recovers the Kullback-Leibler divergence KL (51), i.e., the usual relative entropy, whereas for $f(r)=r^{\alpha}$ one recovers the family of Hellinger-Rényi divergences LM (08). We will come back to these particular choices at the end of this section.

The forward-reverse ratio will be henceforth denoted

A priori, one should pay attention to the pairs $(x,y)$ such that this ratio is not well-defined. If necessary, this problem can be dealt with on a case-by-case basis, but there is no need to burden the notation already at this point. Besides, common sense demands (and our later definition will vindicate it) that if a pair $(x,y)$ is assigned a null probability under the forward process, it should be so also under the reverse process, and vice versa. Thus, ultimately, $r(x,y)$ will be ill-defined only for pairs $(x,y)$ that don’t contribute to any average because $P_{F}(x,y)=P_{R}(x,y)=0$.

Eq. (1) suggests to interpret $f(r(x,y))$ itself as a random variable, whose realization is denoted $\omega_{F}(x,y)$ for the forward process. The same random variable, when evaluated for the reverse process, is denoted $\omega_{R}(x,y)$. For consistency, these two must be the same function, though evaluated on different arguments: more explicitly,

simply due to the fact that when considering the reverse variable $\omega_{R}$, the roles of forward and reverse processes are exchanged and the ratio is thus inverted.

A Crooks-type fluctuation relation is a relation between the probability density functions $\mu$ of $\omega$ for the forward and the backward processes:

If $f:\mathbb{R}^{+}\to\mathbb{R}$ is invertible, everything is well defined, and moreover there exists another function $g$ such that $f(1/r)=g(f(r))$, i.e., $\omega_{R}(x,y)=g(\omega_{F}(x,y))$. More explicitly, $g(u)=f(\frac{1}{f^{-1}(u)})$. Plugging these definitions into (5), we obtain

and thus, with $u=g(\omega)$, we have a Crooks-type relation of the form

Moreover, since $\int_{\mathbb{R}^{+}}\mu_{R}(g(\omega))|g^{\prime}(\omega)|d\omega=\int_{\mathbb{R}^{+}}\mu_{R}(u)du=1$, there follows the corresponding Jarzynski-like relation

The above relation can be verified by unraveling our notations: since $\omega$ here is $\omega_{F}$, we have $g(\omega)=\omega_{R}$, and $f^{-1}(\omega_{R})=1/r$. Thus $\left\langle f^{-1}(g(\omega))\right\rangle_{F}=\left\langle 1/r\right\rangle_{F}=\sum_{x,y}\frac{1}{r(x,y)}P_{F}(x,y)=\sum_{x,y}P_{R}(x,y)=1$. Both this direct proof and the derivation from (6) show that, from our perspective, the Jarzynski-like relation (7) expresses the normalization of the reverse process.

The usual choice made in the literature is

with $z\neq 0$. In this case, we have $f^{-1}(\omega)=e^{z\omega}$, $f(1/r)=-\frac{1}{z}\ln r$, and thus $g(\omega)=-\omega$. The relations (6) and (7) become the familiar ones, namely,

and

Notice that one could have imposed the condition $\omega_{R}(x,y)=-\omega_{F}(x,y)$ from the start, in a way somehow reminiscent of an “arrow-of-time” variable. In our notation, this condition is equivalent to impose that the function $f$ satisfies $f(1/r)=-f(r)$, which leads to $f(r)=\frac{1}{z}\ln r$ for an arbitrary $z$.

As a second example, consider

with $\alpha\neq 0$. Then $f^{-1}(\omega)=\omega^{1/\alpha}$, $f(1/r)=r^{-\alpha}$, and therefore $g(\omega)=1/\omega$. The fluctuation relations (6) and (7) for random variables that satisfy $\omega_{R}=1/\omega_{F}$ are therefore

Other examples may look exotic, but nonetheless possible: for instance, by choosing $\omega=f(r)=e^{\kappa r}$, with $\kappa\neq 0$, one obtains $\omega_{R}=e^{\kappa^{2}/\ln\omega_{F}}$, that is, $\ln\omega_{R}\ln\omega_{F}=\kappa^{2}$.

We begin this section by reviewing the theory of Bayesian retrodiction. For simplicity, let us consider two random variables $X$ and $Y$ with states labeled by the indices $x$ and $y$, both taken from a finite set $\mathcal{A}$. Let $P_{XY}(x,y)$ denote their joint distribution. If one’s knowledge on $Y$ is updated to a definite value $y=y^{*}$, the Bayes–Laplace rule tells that the agents should update their belief on $X$ according to $P^{\prime}_{X}(x)=P_{X|Y}(x|y^{*})$. In other words, the joint probability is updated as

The above formula, which constitutes the standard formulation of the Bayes–Laplace rule, is silent however about those situations, most common in real scenarios, in which the update does not result in a definite value $y^{*}$, but is itself described in terms of another probability distribution $P_{Y}^{\prime}(y)$. In the face of such “soft evidence”, the update should follow Jeffrey’s conditioning Jef (65)

Jeffrey based this update on his “rule of probability kinematics”. The coefficients $P_{X|Y}(x|y)$ are seen as defining a channel that propagates the soft belief acquired about $y$ back onto $x$. It was later noticed that Jeffrey’s update can also be obtained from Bayes–Laplace rule using Pearl’s “method of virtual evidence”  Pea (88); Jay (03); CD (05); Jac (19). From this viewpoint, the soft evidence on $Y$ is consequence of another definite evidence on a “virtual” variable $Z$, that has no direct influence on $X$ (i.e. $X\to Y\to Z$ forms a Markov chain). When $Z$ is updated to a definite value $z^{*}$, one updates the belief on $Y$ to $P^{\prime}_{Y}(y)\equiv P_{Y|Z}(y|z)\,\delta_{z,z^{*}}$ and Eq. (10) is recovered.

The Bayesian update we just described becomes retrodiction when the variables $X$ and $Y$ are given a diachronic meaning: $X$ represents the system’s state at the initial time $t=0$, while $Y$ represents the system’s state at the final time $\tau>0$.

As already noticed, if we know the forward transition probability $P_{Y|X}(y|x)$, henceforth denoted as $\varphi(y|x)$, any prior knowledge $p(x)$ on $X$ can be forward-propagated using it. Hence, the forward process will be

where $p(x)$ can be chosen arbitrarily. Now we want to define the reverse transition probability $\hat{\varphi}(x|y)$, with the same functionality: it should be possible to use it to back-propagate onto $X$ any prior knowledge $q(y)$ that one obtains about $Y$. In other words, the reverse process will be

where $q(y)$ can be chosen arbitrarily.

The reverse transition is obtained from Jeffrey’s conditioning (10), but the knowledge of the forward transition alone is not enough Wat (65). One needs to define a reference prior $P_{X}(x)$. In the context of irreversibility, a natural choice is to set the reference prior equal to a steady (viz. invariant) state $\gamma(x)$, that is, a distribution such that

for all $y\in\mathcal{A}$. This choice coincides with what is customarily done in the theory of Markov chains when defining the reverse chain Nor (97). Notice that, while every transition matrix possesses at least one steady state, the steady state may not be unique: in such a case, to any choice of a steady state there will correspond a different reverse transition, hence a different fluctuation relation.

Starting from a steady state, the reverse transition $\hat{\varphi}(x|y)$ is defined by the relation $\gamma(y)\hat{\varphi}(x|y)=\gamma(x)\varphi(y|x)$. We see that $\gamma$ is an invariant distribution for the reverse process too (in other words: by choosing the steady state as prior, we define a reference process that does not distinguish between forward and reverse evolution). Implicitly restricting the analysis to the pairs $(x,y)$ with strictly positive weight (i.e., $\gamma(x)\varphi(y|x)>0$), we obtain

Plugging this together with Eqs. (11) and (12) into (2), we find

that is, the forward-reverse ratio $r(x,y)$, which is the crucial quantity in the study of irreversibility as noticed in Section II, depends on the stochastic transition $\varphi(y|x)$ only through its steady state $\gamma$.

We start by noticing that the condition

holds if and only if the transition is doubly-stochastic, viz. if in addition to the compulsory normalisation $\sum_{y}\varphi(y|x)=1$, it also holds $\sum_{x}\varphi(y|x)=1$. Indeed, comparing with (13), we see that (20) holds if and only if the steady state can be chosen as the uniform distribution $\gamma(x)\propto 1$. It is then trivial to check that doubly stochastic channels always admit such a steady state, while channels that are not doubly stochastic do not have a uniform steady state.

A particularly important case of doubly stochastic transition is classical Hamiltonian dynamics, which is deterministic, viz. there is a one-to-one correspondence between the initial state $x$ and the final state $y\equiv x^{\prime}$. The evolution $x\to x^{\prime}$ is then represented by the transition $\varphi(x^{\prime}|x)$, with $\varphi(x^{\prime}|x)=1$ if $x^{\prime}$ is the final state corresponding to the initial state $x$, and $\varphi(x^{\prime}|x)=0$ otherwise. Thus, for any classical Hamiltonian process one can choose $\gamma$ as uniform.

For instance, the scenario considered by Bochkov and Kuzovlev BK (77) is of this type. They specify a class of driven Hamiltonians such that, on any given forward trajectory $x\rightarrow x^{\prime}$, the non-driven term $H_{0}$ satisfies $H_{0}(x^{\prime})=H_{0}(x)+E(x,x^{\prime})$, with $E$ determined by the driving protocol. By choosing thermal priors $p(x)\propto e^{-\beta H_{0}(x)}$ and $q(x^{\prime})\propto e^{-\beta H_{0}(x^{\prime})}$, with $\beta=1/k_{B}T$, they obtain

Since the process is deterministic, the statistics (and in particular the fluctuation relations) carry over from the initial conditions to the whole trajectory. From this observation, rich physical consequences follow: one can derive many-point relations, relations for the currents (e.g. Onsager), linear response results (in the limit when the driving forces go to zero), etc. BK (77). As we presented it, all this can be seen as starting with retrodiction.

Another example that fits in this subsection is that of a generic time-dependent Hamiltonian, but with microcanonical priors instead of the most frequently used thermal ones THM (08). Staying with a discrete alphabet again (the generalisation follows immediately), such priors read: $p(x)=N(E)^{-1}$ if $E_{x}=E$ (and zero otherwise) and $q(x^{\prime})=N(E^{\prime})^{-1}$ if $E_{x^{\prime}}=E^{\prime}$ (and zero otherwise), where $N(E)$ and $N(E^{\prime})$ are the degeneracies of the two levels. Thus, for the processes with non-zero probability, the forward-reverse ratio reads

where $S(E):=k_{B}\ln N(E)$ coincides with Boltzmann’s entropy formula.

We consider now a system composed of two subsystems. For notational purposes, let us denote the microstates of the first system at initial and final time by the labels $x$ and $x^{\prime}$, respectively; and those of the second system analogously by the labels $w$ and $w^{\prime}$. For the purpose of the present example, all labels belong to discrete sets. If the joint evolution is Hamiltonian, we can borrow from the previous discussion and conclude that

To construct the forward and reverse processes (11) and (12), one should specify the prior distributions $p(x,w)$ and $q(x^{\prime},w^{\prime})$.

This notation opens the possibility of coarse-graining over one of the subsystems. For definiteness, we describe the particular case studied in Jar (00). In this narrative, the second system consists of one or several heat reservoirs. It is rather natural therefore to stipulate that:

1. 1.

   Both priors are in product form, i.e., $p(x,w)=p(x)P(w)$ and $q(x^{\prime},w^{\prime})=q(x^{\prime})Q(w^{\prime})$;

2. 2.

   The reservoirs’ priors $P(w)$ and $Q(w^{\prime})$ are thermal distributions at inverse temperature $\beta^{-1}=k_{B}T$, i.e., $P(w)\propto e^{-\beta E_{w}}$ and $Q(w^{\prime})\propto e^{-\beta E_{w^{\prime}}}$, where $E_{w}$ and $E_{w^{\prime}}$ are the energies of the reservoir’s microstates $w$ and $w^{\prime}$. This condition implies

   $\displaystyle\frac{P(w)}{Q(w^{\prime})}=e^{\beta(E_{w^{\prime}}-E_{w})}=:e^{\Delta S/k_{B}}$

   (24)

   where $\Delta S$ is the entropy generated.

With these two assumptions on the prior distributions, one can compute the following marginal conditional probability:

The above represents the forward transition probability that the system, if starting in microstate $x$, will end up in microstate $x^{\prime}$ generating in the process an amount of entropy equal to $\Delta S$. Notice that the coarse-grained transition $\varphi(x^{\prime},\Delta S|x)$ computed in (25) only depends on the reservoir’s prior $P(x)$. In other words, while the reservoir’s prior distribution $P(x)$ is fixed, the system’s prior remains arbitrary.

Analogously, but starting from the retrodicted transition (23), we obtain

where in (26) we used relation (23), while in (27) we used relation (24). The above represents the reverse transition probability that the system, if starting in microstate $x^{\prime}$, will end up in microstate $x$ generating in the process an amount of entropy equal to $-\Delta S$. Summarizing, we have recovered

which is the main result of Jar (00). It is worth stressing that, as written, $\Delta S$ plays the role of an additional random variable coarse-graining the reservoir’s microstates: that is, $\varphi(x^{\prime},\Delta S|x)$ describes a transition from input $x$ to output $(x^{\prime},\Delta S)$. Thus, the object at the denominator in (28) is not the complete Bayesian reverse of the numerator, but rather a partial (viz. “hybrid” Jar (00)) reversal.

However desirable it would be to derive everything from the Hamiltonian dynamics of a closed (possibly composite) system, information about the latter is often lacking. This is when the approach based on Bayesian retrodiction really shows its power: it allows one to make inferences based on whatever partial information is available.

As a first example of this type, we consider the stochastic thermodynamics setting that led to Crooks’ theorem Cro (98). A general process in discrete-time stochastic thermodynamics is modeled as a sequence of external driving protocols (the work steps) alternating with periods during which the system is allowed to equilibrate with an ideal heat bath (the relaxation steps) Cro (98); SSBC (12). The changes in the system’s internal energy during each work step are counted as work done on the system, while the changes happening during each relaxation steps are counted as heat absorbed by the system. For simplicity, we consider here just a two-step process: one work step followed by one relaxation step.

Let us denote the system’s initial state by $x$, and let $E_{x}$ be the system’s initial energy. The system is (deterministically) driven to another energy $E^{\prime}_{x}$. This constitutes the work step. The relaxation step, which is not deterministic, is modeled using a transition conditional probability $\varphi(y|x)$, where $y$ labels the system’s state after the relaxation step. Assuming that during the relaxation step the system’s Hamiltonian does not change, the system’s final energy, after the relaxation step is over, is $E^{\prime}_{y}$.

By definition, an invariant distribution for $\varphi(y|x)$ is the thermal distribution $\gamma(x)\propto e^{-\beta E^{\prime}_{x}}$. This, via (13), leads to the retrodicted transition

Choosing as priors the thermal distributions, that is, $p(x)=e^{\beta(F-E_{x})}$ and $q(y)=e^{\beta(F^{\prime}-E^{\prime}_{y})}$, with $F,F^{\prime}$ the corresponding Helmholtz free energies, the ratio (14) becomes

where in the last passage we used the assumption that the system’s internal energy change during the work step is work $W$ done on the system. The fluctuation relation (8) then immediately gives

in accordance with Cro (98). Notice that in our retrodictive derivation we did not require any particular condition (such as the condition of detailed balance) for the relaxation step $\varphi(y|x)$, apart from it preserving the thermal distribution, which is implicit in the definition of “relaxation”.

The paradigm for quantum fluctuation relations is provided by Tasaki’s two-measurement setup Tas (00). Here, a $d$-level quantum system is prepared in the state $\rho_{0}$ but immediately subjected to a von Neumann projective measurement of the initial Hamiltonian $H_{0}=\sum_{x}\epsilon_{x}|\epsilon_{x}\rangle\langle\epsilon_{x}|$. The system, now collapsed onto $|\epsilon_{x}\rangle$ after observation of $\epsilon_{x}$, next undergoes a perfectly adiabatic work protocol: its Hamiltonian is driven from $H_{0}$ to $H_{\tau}=\sum_{y}\eta_{y}|\eta_{y}\rangle\langle\eta_{y}|$, but the system is otherwise perfectly isolated from the surrounding environment. At the end of the driving protocol, during which only mechanical work has been exchanged with the system, the system is subjected to a second energy measurement, this time of the final Hamiltonian $H_{\tau}$.

Let us analyse Tasaki’s setup with our tools. Denoting the unitary evolution resulting from the driving protocol by $U_{0\to\tau}$, the forward (predictive) transition probability is given by

It is easy to verify that $\varphi(y|x)$ is doubly stochastic (see Subsection V.1). This is a consequence of the following three facts: (i) that the underlying quantum process is unitary; (ii), that $\Tr[|\epsilon_{x}\rangle\langle\epsilon_{x}|]=\Tr[|\eta_{y}\rangle\langle\eta_{y}|]=1$; and (iii), that $\sum_{x}|\epsilon_{x}\rangle\langle\epsilon_{x}|=\sum_{y}|\eta_{y}\rangle\langle\eta_{y}|=\openone$. As noted in Subsection V.1, the reverse (retrodictive) transition is given by

At the same time, the corresponding quantum retrodiction, given in Eqs. (17)-(19), is in perfect agreement with a narrative involving time-reversals, as in e.g. Sag (12): one first prepares the eigenstates of $H_{\tau}$, then evolves them “backwards in time” using $U_{0\to\tau}^{\dagger}$, and finally measures $H_{0}$. While such a narrative seems simple and appealing to intuition, it is not directly operational, as the actual implementation of time-reversals is far from straightforward QDS⁺ (19); GLM⁺ (20). Moreover, as we will see in what follows, when the evolution between the two energy measurements is not Hamiltonian, a naive argument using time-reversal can lead to inconsistencies.

In any case, with Eq. (30) at hand, fluctuation relations can be easily derived. One only needs to specify the two priors $p(x)$ and $q(y)$, which in the two-measurement setup is tantamount to specifying two quantum states $\rho_{0}$ and $\sigma_{\tau}$, since from them we have $p(x)=\langle\epsilon_{x}|\rho_{0}|\epsilon_{x}\rangle$ and $q(y)=\langle\eta_{y}|\sigma_{\tau}|\eta_{y}\rangle$. Simply by choosing $\rho_{0}$ as the thermal state for $H_{0}$ and $\sigma_{\tau}$ as the thermal state for $H_{\tau}$ (by assuming that the temperature is the same), we find that

where in the final passage we identified the system’s energy difference as work $W$ done on the system, due to the assumption of adiabaticity during the driving protocol. From here, fluctuation relations analogous to Crooks’ theorem and Jarzynski’s equality quickly follow.

Tasaki’s setup invites various generalizations. Several references ALMZ (13); Ras (13); RZ (14); GPM (15) have considered the situation in which the initial and final measurements are as in Tasaki’s arrangement but the unitary evolution is replaced by a general CPTP linear map $\mathcal{E}$, leading to the forward transition

If the linear map $\mathcal{E}$ is not unital, that is, if it does not preserve the unit matrix (viz. $\mathcal{E}(\openone)\neq\openone$), then the above transition probability is not doubly stochastic in general. For the retrodictive transition, this means that $\hat{\varphi}(x|y)\neq\varphi(y|x)$, and indeed Eq. (13) contains the extra factor $\gamma(x)/\gamma(y)$ coming from Bayes–Laplace rule.

Nonetheless, a common approach prescribes to stay put with Eq. (30) and work with the non-normalized conditional distribution $\tilde{\varphi}(x|y):=\varphi(y|x)$ instead. Obviously, $\tilde{\varphi}(x|y)$ does not admit a realization as in (16), simply because it is not a well-formed conditional distribution. Nonetheless, it is still possible to construct a “ratio”

and formally go through all the calculations as in the normalized case. However, as a result of considering a reverse “process” that is not properly normalized, the resulting Jarzynski–like relation does not average to 1 as one would like (and as it happens in (7)), but to $\sum_{x,y}\tilde{\varphi}(x|y)q(y)$. This quantity is known as the efficacy ALMZ (13), but we recognize it here as a mathematical artifact arising from an ill-defined reverse transition.

The formalism presented in this work allows a simple treatment of the two-measurement process also for arbitrary quantum channels. The forward transition (31) coincides with Eq. (15) for $\rho_{0}^{x}:=|\epsilon_{x}\rangle\langle\epsilon_{x}|$ and $\Pi^{y}_{\tau}:=|\eta_{y}\rangle\langle\eta_{y}|$. Let $\gamma(x)$ be an invariant distribution for $\varphi(y|x)$. Further, let predictive and retrodictive prior distributions be thermal distributions for some energy levels $\epsilon_{x}$ and $\eta_{y}$, so that $p(x)\propto e^{-\beta\epsilon_{x}}$ and $q(y)\propto e^{-\beta\eta_{y}}$. The corresponding initial and final Hamiltonians are defined as $\sum_{x}\epsilon_{x}|\epsilon_{x}\rangle\langle\epsilon_{x}|$ and $\sum_{y}\eta_{y}|\eta_{y}\rangle\langle\eta_{y}|$, respectively.

The ratio is defined as usual, that is,

where now we find an extra term $\Delta\Phi:=\Phi_{y}-\Phi_{x}:=\frac{1}{\beta}\ln\gamma(x)-\frac{1}{\beta}\ln\gamma(y)$. This term is understood as the difference of a nonequilibrium potential that adds to the difference of equilibrium free energy $\Delta F:=F_{\tau}-F_{0}$. Finally, introducing a total stochastic entropy production $\Sigma:=\beta(\Delta E-\Delta\Phi-\Delta F)$, one easily obtains the detailed relation

and the corresponding integral relation $\left\langle e^{-\Sigma}\right\rangle_{F}=1$. In this way, we can see how a correct application of the Bayes–Laplace inversion formula automatically takes into account nonequilibrium potentials. These, usually introduced as corrections HS (01); MHP (15), are now recognizable as the remnants of Bayesian inversion.