Mismatch cost of computing: from circuits to algorithms

Recent decades have seen significant advancements in non-equilibrium statistical physics, particularly in stochastic thermodynamics seifert2012stochastic ; parrondo2015thermodynamics ; wolpert2019stochastic and open quantum thermodynamics kolchinsky2021dependence . These fields extend classical statistical physics to systems far from thermal equilibrium, including those with many degrees of freedom and thermal and quantum fluctuations, which are critical in small-scale systems. Stochastic thermodynamics provides the mathematical tools to describe energy dissipation, or entropy production (EP), in various systems, including biological processes, nano-machines, and single-molecule experiments.

The early work of Szilard, Landauer, and Bennett highlighted the thermodynamic properties of computation szilard1964decrease ; landauer1961irreversibility ; bennett1982thermodynamics . Their contributions laid the foundation for the modern understanding that information is physical and that physical processes, including computation, are governed by thermodynamic laws. Stochastic thermodynamics applies not only to nano-scale systems but also to systems of any size, providing a framework for describing entropy production in processes far from thermal equilibrium, with many rapidly evolving degrees of freedom. This makes stochastic thermodynamics an ideal framework for analyzing the thermodynamics of real-world digital devices.

In the field of computer science, the efficiency of algorithms has traditionally been measured in terms of two primary resources: space and time arora2009computational . These measures, known as space complexity and time complexity, capture the amount of memory and the duration of computation, respectively, required to solve a problem. While these dimensions are fundamental to software development and system design, they do not account for the thermodynamic costs inherent in computation. Space and time complexity focus on the logical and operational aspects of algorithms, but they overlook the physical resources required for executing computations, especially in terms of energy dissipation. While these dimensions are fundamental to software development and system design, we argue that a third dimension “THERMO"-complexity should also be considered. The thermodynamic perspective is not merely an extension but a necessity for fully understanding the costs of computation.

A central quantity in this analysis is the mismatch cost wolpert2020minimal ; kolchinsky2021dependence ; kolchinsky2017dependence , a lower bound to entropy production in physical systems that implement a given stochastic process. Mismatch cost is universal in that it applies to any system, regardless of whether the process is discrete or continuous in time, Markovian or not, quantum or classical, or whether the state space is countable or uncountable. More importantly, it provides a strictly positive lower bound to entropy production in periodic processes, a feature that is characteristic of most modern digital devices ouldridge2023thermodynamics ; manzano2024thermodynamics . This makes mismatch cost a crucial quantity for understanding the thermodynamic limits of real-world computing systems.

Additionally, mismatch cost applies to modular systems, where computational variables are statistically coupled without being physically linked boyd_different_2018 ; wolpert2020thermodynamics ; wolpert2019stochastic ; wolpert2020uncertainty . This concept strengthens the second law of thermodynamics, offering a thermodynamic lower bound on EP for processes that are either periodic or modular—features common in digital computation.

This paper has several key contributions:

First, in Sec. II.2, we show that mismatch cost can be a significant contribution to the total EP at the macroscopic scale, in contrast to traditional lower bounds on EP which primarily focus on the nanoscale, such thermodynamic uncertainity relations (TURs) and speed limits.

Second, in Sec. II.3, we analyse the mismatch cost to systems that are coarse-grained in time and/or space, showing that even coarser resolution, the mismatch cost still provides a lower bound on the microscopic entropy production. This broadens the applicability of mismatch cost to any computational machine beyond the idealized, fine-grained models typically used in stochastic thermodynamics. This makes mismatch cost well-suited to quantify the thermodynamic cost for arbitrary computational machines.

Finally, in Sec. III and IV, we apply mismatch cost to two fundamental models of computation: Boolean circuits and Random Access Stored Program machines (a simplified model of microprocessors). We derive the mismatch cost for both models, providing concrete examples of how this thermodynamic measure can be used to assess the efficiency of computational systems.

The prior distribution is defined as the initial distribution that minimizes $\sigma(p)$, denoted as $q_{0}:=\mathrm{arg}\min\{\sigma(p)\}$. The mismatch cost quantifies the portion of the cost arising from a mismatch between the actual initial distribution $p$ and a prior distribution $q$ that minimizes the cost (1), and it is given by:

where $D(p\|q)=\sum_{x}p(x)\ln\frac{p(x)}{q(x)}$ is the KL divergence between distributions $p$ and $q$. It has been shown that any cost function of the form (1) can be written in terms of the mismatch cost kolchinsky2017dependence ; kolchinsky2021dependence ; wolpert2020thermodynamics :

where $\sigma_{\text{res}}:=\min\{\sigma(p)\}=\sigma(q)$ is called the residual cost. There are a couple of important things to note about the decomposition (3). First, $\sigma_{\mathrm{res}}$ is inherent to the dynamics of the process and does not depend on the actual initial distribution $p$ over the states of the system. Second, on the other hand, $\mathrm{MC}(p_{0})$ is a function of the actual initial distribution $p_{0}$ and it has a very limited dependence on the details of the underlying physical process primarily arising from the prior distribution $q_{0}$. Moreover, due to the second law, $\sigma_{res}\geq 0$ always holdsseifert2012stochastic ; van2015ensemble , and therefore, mismatch cost always provides a lower bound to the EP:

Equality is achieved when $\sigma_{res}=0$, indicating that the underlying physical process is thermodynamically reversible for a certain initial distribution. Calculating $\sigma(p)$ requires full knowledge of the system’s dynamics, which is rarely available for real physical systems. On the other hand, calculating mismatch cost only requires the knowledge of the map $G$ and the prior distribution $q_{0}$. Therefore, for a specified computation $G$, and given prior distribution $q_{0}$ for the physical substrate on which the computation is run, the mismatch cost formula (4) provides a lower bound on the EP. Moreover, if the same process is repeated many times over the same interval, the actual distribution over the states of the systems is $p_{n}=G^{n}p_{0}$. Since the underlying process is the same, the associated prior distribution would remain the same for each iteration, only the actual distribution over the states of the system modifies with each iteration. The mismatch cost in the $i^{th}$ iteration of the process is,

Mismatch cost in Eq. (6) is called periodic mismatch cost. The total mismatch cost for $n$-iterations of the process is obtained by summing over all the iterations

It is important to note that even if the system initially begins with its actual distribution matching the prior distribution, the mismatch cost is zero only for the first iteration. After this initial step, the actual distribution over the system’s states will deviate from the prior distribution, leading to a non-zero mismatch cost in subsequent iterations. Thus, Eq. (7) establishes an unavoidable non-zero EP in any repeated process. Figure 2 illustrates a straightforward example of a map $G$ that is applied repeatedly over a system’s state space, showing the mismatch cost contribution to the EP in each iteration.

In the following sections, we examine the properties of $q_{0}$ and analyze its dependence on the details of the underlying physical system. Additionally, we demonstrate that the mismatch cost can constitute a substantial portion of the total EP, highlighting the practical utility of the bound in Eq. (4).

It’s clear from the definition (1) that the prior distribution $q_{0}=\mathrm{arg}\min_{p_{0}}\{\sigma(p_{0})\}$ depends on $G$ and $f_{i}$, where $f_{i}$ is thermodynamically interpretation as the average heat flow from the system to the environment during the evolution over $[t_{0},t_{1}]$, when the system is initialized in state $x_{0}=i$.

It’s important to mention that the prior $q_{0}$ is always in the interior of the probability simplex $\Delta(X)$. This is proved in Appendix (A). In other words, the prior distribution assigns non-zero probability to every state in the state space $X$. By ensuring that $q_{0}$ remains in the interior of the simplex, the mismatch cost remains well-behaved and finite for any initial distribution $p$.

Moreover, if $f_{x}$ values are sufficiently large for all $x$ relative to the change in system’s Shannon entropy in (1), then $f_{x}$ will primarily determine $q_{0}$. In such cases, if $f_{x}$ is not uniform across all states—meaning $f_{x}$ is higher for some states than others—the associated prior distribution $q_{0}$ will take on lower values for those higher-weighted states compared to the rest.

Put differently, the prior distribution will shift closer to the boundary of the probability simplex. The greater the non-uniformity of $f_{x}$ across states, the nearer the prior will be to the simplex edge. In that case, any typical actual distribution $p_{0}$ on the simplex that is not close to the edge would yield a significantly high value of KL-divergence $D(p_{0}\|q_{0})$. This intuitive idea is formalized in Appendix C to prove that the mismatch cost could get at least as large as the difference between maximum and minimum value of $f_{x}$ minus $\log|X|$.

Here, $\mathrm{MC}^{*}$ is the worst-case mismatch cost, i,e., when the actual distribution $p_{0}$ is furthest away from the prior distribution. Remember that thermodynamically, $f_{x}$ represents the average heat flow terms. Result (8) highlights that in the physical scenario where the heat flow terms are significantly large compared to $\ln|X|$, the mismatch cost could amount to a significant portion of the total EP. This situation is particularly relevant for mesoscopic processes where the heat generated is large enough to be measurable, in contrast to microscopic stochastic systems where the heat generated is typically negligible.

There are other known lower bounds on EP, such as the Thermodynamic Uncertainty Relation (TUR)s barato2015thermodynamic ; gingrich2016dissipation ; horowitz2020thermodynamic and speed limits shiraishi2018speed ; vo2020unified . The TUR provides a bound on the trade-off between EP and the precision of a thermodynamic current. For a system with current $J$, the TUR takes the form:

where $\langle J\rangle$ is the mean current, $\mathrm{Var}(J)$ is its variance, and $\langle\sigma\rangle$ is the mean EP. This inequality bounds the variance of observable quantities in terms of the EP, making it a useful tool for stochastic processes. However, the TUR does not apply to deterministic processes, such as computational algorithms, and does not explicitly address the role of initial distributions.

Speed limits, on the other hand, place constraints on the minimum time required for a system to evolve between two states, given the amount of energy dissipated in the process. Other recent studies have developed techniques for estimating EP using limited system observables kawai2007dissipation ; roldan2010estimating ; bisker2017hierarchical ; harunari2022learn ; pietzonka2024thermodynamic and for analyzing EP based on the dynamics of coarse-grained systems degunther2024fluctuating ; gomez2008lower .

Mismatch cost differs from both the TUR and speed limits in that it focuses on the additional EP resulting from the mismatch between the actual initial distribution $p_{0}$ and the prior distribution $q_{0}$. Unlike TUR and speed limits, the mismatch cost framework is general enough to apply to both stochastic and deterministic processes, including computational algorithms and logical circuits. In this sense, mismatch cost provides a more versatile tool for analyzing EP across a wide range of physical and information-theoretic systems, offering insights that complement the TUR and speed limits by directly addressing the effects of non-optimal initial conditions.

The mismatch cost lower bound on EP associated with a computation depends significantly on the time resolution at which the computation process is observed. For example, when a circuit runs, advancing the computation across it’s gate, one could calculate the mismatch cost at a finest time resolution, where the contribution from each gate’s operation is accounted for individually as it runs. Alternatively, a coarser time resolution might involve calculating the mismatch cost for groups of gates after they complete their operations, or even at the coarsest level, where the mismatch cost is determined solely from the initial and final states of the entire circuit. Understanding how mismatch cost behaves under varying levels of temporal granularity is therefore crucial.

Consider a computation occurring over the time interval $[t_{0},t_{1}]$. This interval can be partitioned into two subintervals, $[t_{0},\tau]$ and $[\tau,t_{1}]$, where $\tau$ is an intermediate time satisfying $t_{0}<\tau<t_{1}$. Let $G_{1}$ be the map that evolves an initial distribution $p_{0}$ to a distribution at the intermediate time $p_{\tau}$, i.e. $p_{\tau}=G_{1}p_{0}$ during $[t_{0},\tau]$, and let $G_{2}$ operate on the second time interval, with $p_{1}=G_{2}p_{\tau}$ during $[\tau,t_{1}]$. The EP during the time interval $[t_{0},t_{1}]$, denoted as $\mathrm{EP}_{[t_{0},t_{1}]}(p_{0})$, can be expressed as the sum of the EP generated during the sub-intervals $[t_{0},\tau]$ and $[\tau,t_{1}]$. Formally:

By definition, the residual cost $\mathrm{R}_{[t_{0},\tau]}$ over the interval $[t_{0},\tau]$ is

Since $\mathrm{EP}_{[t_{0},\tau]}(p_{0})=\mathrm{MC}_{[t_{0},\tau]}+\mathrm{R}_{[t_{0},\tau]}$ and $\mathrm{EP}_{[\tau,t_{1}]}(p_{\tau})=\mathrm{MC}_{[\tau,t_{1}]}+\mathrm{R}_{[\tau,t_{1}]}$, therefore

If the process begins with an initial distribution that matches the prior distribution for the entire interval $[t_{0},t_{1}]$, such that $\mathrm{R}_{[t_{0},t_{1}]}=\mathrm{EP}_{[t_{0},t_{1}]}(q_{0})$, then at the intermediate time $\tau$, the prior distribution evolves to $q_{\tau}=G_{1}q_{0}$, where $G_{1}$ represents the transition dynamics up to time $\tau$. Since the EP over the full interval can be broken into contributions from the two subintervals, we can write:

It is important to note that, in general, $q_{\tau}$ does not serve as the prior for the process during the remainder of the interval $[\tau,t_{1}]$. As a result, the residual cost $\mathrm{R}_{[\tau,t_{1}]}\geq\mathrm{EP}_{[\tau,t_{1}]}(q_{\tau})$. Consequently, the residual cost over the full interval $[t_{0},t_{1}]$ satisfies the inequality:

Using inequality (14) with Eq. (II.3) results in

The inequality above implies that the mismatch cost calculated at a coarser time resolution is always lower than or equal to the sum of the mismatch costs calculated at finer time resolutions over the same interval.

An analogous consideration for comparing mismatch costs under spatial coarse-graining does not hold. In the case of spatial resolutions, one can either analyze the mismatch cost at a finer resolution or at a coarser resolution, but a direct comparison between the two is not possible. Nonetheless, the mismatch cost calculated at a coarser spatial resolution continues to provide a valid lower bound on the entropy production at the finer resolution. This is elaborated upon in Appendix B.

Building on this observation, the following sections systematically apply the mismatch cost formula to Boolean circuits, deriving lower bounds on the EP associated with each step of computation within a circuit.

Formally, a loop-free circuit is a directed acyclic graph (DAG) denoted by $(V,E,\mathcal{X})$. $V$ is the set of nodes or gates and $E$ denotes the set of directed edges connecting nodes. The direction of each edge indicates the dependency between nodes, allowing one to enumerate the nodes in $V$ in a specific sequence. A topological ordering of the nodes satisfies the condition that for every directed edge from node $\mu$ to node $\nu$, node $\mu$ precedes node $\nu$ in the ordering ($\mu<\nu$). The state space of a node $\mu\in V$ is denoted by $X_{\mu}$, and its state is represented by $x_{\mu}\in X_{\mu}$. The joint state space of the circuit is given by $\mathcal{X}=\bigotimes_{\mu}X_{\mu}$.

A node $x_{\mu}$ has incoming edges from nodes known as its parent nodes and outgoing edges to nodes known as its children nodes. The joint state of the parent nodes is denoted by $\mathrm{pa}(x_{\mu})$, and the joint state of the children nodes is denoted by $\mathrm{ch}(x_{\mu})$. The input nodes, or root nodes, have no incoming edges from other nodes. The output gates, or leaf nodes, have no outgoing edges to other nodes. Running a gate $x_{\mu}$ means updating it’s state based on the state of its parent nodes. This dependency is expressed with the conditional distribution $\pi_{\mu}(x_{\mu}|\mathrm{pa}(x_{\mu}))$.

Gates can also be organized into groups known as layers, based on their connectivity, and a similar topological ordering can be applied to these layers. To preserve the generality of the analysis, in what follows, $\text{x}_{A}$ may represent either a single gate or an entire layer of gates, where $A$ denotes the layer’s position in the topological order. One can also discuss the parent layer and child layer, with dependencies expressed as $\pi_{A}(\text{x}_{A}\mid\text{pa}(\text{x}_{A}))$. The entire sequence is called an execution cycle and running a gate or a layer of gates is a step in execution cycle.

Running a circuit can be conceptualized as sequentially updating it gate-by-gate or layer-by-layer. This approach reflects how circuits are implemented in real-world scenarios. Another key observation is that circuits in all computational devices are reused repeatedly. Each time a circuit is used, it undergoes the same sequential process: starting with the new input values, followed by updating the subsequent layers, and so on. We will incorporate this observation into our analysis. We also assume that after each complete run of the circuit, the values of each gate remain as they were and are not reinitialized to any special state before beginning the next cycle.

Let’s consider that input are sampled from the distribution $p_{in}(\text{x}_{in})$. This input distribution induces a distribution over the joint state of the circuit. The joint distribution over the circuit after it’s first run is

We will keep refereeing back to the total joint distribution in (16). Since the gates are not re-initialized to any special state before the circuit is used for the next run, the joint distribution over the state of circuit is $p_{0}(\textbf{x})=p(\text{x}_{\mathrm{in}},\text{x}_{\mathrm{nin}})$ given by (16). Before re-using the circuit, values of the input nodes are over-written with new values which are sampled from $p_{\mathrm{in}}(\text{x}_{\mathrm{in}})$. Meanwhile, other non-input nodes remain unchanged. Therefore, after overwriting inputs, distribution over the joint state of the circuit is,

where $p(\text{x}_{\mathrm{nin}})=\sum_{\mu\in V_{\mathrm{in}}}p(\text{x}_{\mathrm{in}},\text{x}_{\mathrm{nin}})$ is the marginal distribution over non-input nodes.

During the process of overwriting new input values, the input nodes evolve independent of the rest of the non-input nodes which remain unchanged. Therefore, updating input nodes with new values is a sub-system process where the sub-system $\text{x}_{\mathrm{in}}$ evolves independently of $\text{x}_{\mathrm{nin}}$ while the later remains unchanged. The prior distribution for this subsystem process is product distribution, expressed as $q_{0}(\text{x}_{\mathrm{in}})q_{0}(\text{x}_{\mathrm{nin}})$. Since new values of the input nodes are sampled from $p_{\mathrm{in}}(x_{\mathrm{in}})$, this prior distribution evolves to $p_{\mathrm{in}}(\text{x}_{\mathrm{in}})q_{0}(\text{x}_{\mathrm{nin}})$.

while the actual distribution evolves from $p(\text{x}_{\mathrm{in}},\text{x}_{\mathrm{nin}})$ to $p(\text{x}_{\mathrm{in}})p(\text{x}_{\mathrm{nin}})$:

Since new input values are totally independent of the states of the rest of the gates, $\mathcal{I}_{1}(\text{x}_{\mathrm{in}};\text{x}_{\mathrm{nin}})=0$, and the drop in mutual information is

The mismatch cost of overwriting the input values is

After overwriting the input nodes with new values, the gates in the next immediate layer are updated based on these inputs, followed by the subsequent layers, and so on. The gates in each layer can be updated either one at a time, which we refer to as a gate-by-gate implementation, or all at once, which is termed as a layer-by-layer implementation. Each step in the execution cycle modifies the probability distribution over the state of the circuit.

Let’s assume we are in the middle of the execution cycle. Using our previous notation, let $\text{x}_{A}$ represent the state of the gate or layer of gates that is about to be updated based on the values of its parent nodes. Let $\text{x}_{B}$ denote the joint state of all the gates that have already been updated based on the new inputs, including the input nodes themselves. The set $B$ also includes the parent layer(s) of $A$. Finally, let $\text{x}_{C}$ denote all the gates that are to be updated after $\text{x}_{A}$, that is, all the gates whose update occurs following the update of $\text{x}_{A}$. $\text{x}_{A}$ and $\text{x}_{C}$ are correlated form the previous run of the circuit, while $\text{x}_{B}$ which has values from the new run, is independent of $\text{x}_{A}$ and $\text{x}_{C}$. Therefore, the distribution over the state of circuit right before updating $\text{x}_{A}$ is

where $p(\text{x}_{A},\text{x}_{C})=\sum_{\mu\in B}p_{T}(\textbf{x})$ is the marginal distribution over $\text{x}_{A}$ and $\text{x}_{C}$. The distribution over $\text{x}_{B}$ is $p(\text{x}_{B})=\left(\prod_{\mu\in B\backslash V_{\mathrm{in}}}\pi_{\mu}(x_{\mu}|\mathrm{pa}(x_{\mu}))\right)p_{\mathrm{in}}(\text{x}_{\mathrm{in}})$ and it’s same as marginalizing over $A$ and $C$, $p(\text{x}_{B})=\sum_{\mu\in A\cup C}p_{T}(\textbf{x})$.

When $\text{x}_{A}$ is updated based on the new values of it’s parent which are in $B$, it gets correlated with $\text{x}_{B}$ and at the same time, it’s new value is independent of the value of $\text{x}_{C}$ from the previous run. Therefore, after updating $\text{x}_{A}$, the new distribution is

where $p(\text{x}_{B},\text{x}_{A})=\left(\prod_{\mu\in B\cup A\backslash V_{\mathrm{in}}}\pi_{\mu}(x_{\mu}|\mathrm{pa}(x_{\mu}))\right)p_{\mathrm{in}}(\text{x}_{\mathrm{in}})=\sum_{\mu\in C}p_{T}(\textbf{x})$ and $p(\text{x}_{C})=\sum_{\mu\in A\cup B}p_{T}(\textbf{x})$.

We express this change in the distribution when $\text{x}_{A}$ updates by writing

When $\text{x}_{A}$ is updated based on the values of its parents $\mathrm{pa}(\text{x}_{A})$, the rest of the nodes are unchanged. This makes it a subsystem process where $\text{x}_{A}$ and $\mathrm{pa}(\text{x}_{A})$ form a subsystem. The prior distribution therefore is a product distribution

where $\text{x}_{E}$ denotes the joint state of all the gates in the circuit except for $\text{x}_{A}$ and $\mathrm{pa}(\text{x}_{A})$. $q_{0}(\textbf{x})$ evolves to

where $q_{1}(\text{x}_{A},\mathrm{pa}(\text{x}_{A}))=\pi_{A}(\text{x}_{A}|\mathrm{pa}(\text{x}_{A}))q_{0}(\mathrm{pa}(\text{x}_{A}))$. Using Eq. (24), (25), (27), and (25), the mismatch cost of updating the node $\text{x}_{A}$ is

which after simplification (details in the Appendix D) becomes

where $\mathcal{I}(\text{x}_{A};\mathrm{ch}(\text{x}_{A}))$ is the mutual information between $\text{x}_{A}$ and its children $\mathrm{ch}(\text{x}_{A})$

The total mismatch cost of running the entire circuit with sequential gate-by-gate or layer-by-layer execution is the sum of subsystem mismatch costs. For a gate-by-gate run, the total mismatch cost is

where $\mathrm{MC}_{\mu}:=D(p_{0}(\text{x}_{\mu},\mathrm{pa}(\text{x}_{\mu}))||q_{0}(\text{x}_{\mu},\mathrm{pa}(\text{x}_{\mu})))-D(p_{0}(\mathrm{pa}(\text{x}_{\mu}))||q_{0}(\mathrm{pa}(\text{x}_{\mu})))$. In a layer-by-layer run, the sum is over all the layers

As discussed in the Sec. II.3, the mismatch cost of gate-by-gate implementation of the circuit is always going to be lower bounded by the mismatch cost calculated for layer-by-layer implementation of the circuit.

In most programming languages, statements are executed sequentially unless altered by control flow statements such as loops or conditionals. Each codeline represents a single instruction, executed in the order it appears in the code. The interpreter or compiler tracks which codeline is currently being executed, aiding in program flow understanding and debugging. At the hardware level, this is managed by a program counter (PC) or instruction pointer (IP), a register that holds the memory address of the next instruction to execute. The PC is updated after each instruction, pointing to the next, unless a control flow statement (e.g., jump operations) modifies the sequence.

In a microprocessor, commands are stored in memory as binary codes, each corresponding to a specific instruction (e.g., the "JMP" instruction in the Intel 8080 was represented by an 8-bit code). The memory addresses are sequential, and the program counter increments by one unless a jump operation occurs.

The following simple example in pseudocode illustrates this process:

function result = add(a, b)
    % Start at codeline 1
    current_codeline = 1;

    % Codeline 2: Add a and b
    result = a + b;
    current_codeline = current_codeline + 1;

    % Codeline 3: Return the result
end

The algorithm involves four variables, two input variable a and b, a line counter current_codeline, and an output variable result. The line counter, along with the algorithm’s variables, is sufficient to describe the state of execution, as it tracks which part of the program is currently running.

We formally define an algorithm $\mathcal{A}$ by the tuple $(\textbf{x},i,\mathcal{S_{A}},G_{\mathcal{A}})$, where x represents the set of all variables, including the program counter, which define the state of the algorithm at any given step $i\in\{0,\dots,k\}$. Together, $i$ and x can be written as $\textbf{x}_{i}$. $\mathcal{S_{A}}$ represents the state space of the algorithm, which is the set of all valid states x for the algorithm. $G_{\mathcal{A}}:\mathcal{S_{A}}\to\mathcal{S_{A}}$ is a transition map that updates the state of the algorithm from $\textbf{x}_{i}$ to $\textbf{x}_{i+1}$:

The algorithm starts with $\textbf{x}_{0}$, with it’s program counter being at 0. When the algorithm halts, the map $G_{\mathcal{A}}$ reaches a fixed point. The dynamics of the algorithm induces a corresponding evolution on the probability distribution over the state space $\mathcal{S_{A}}$. To formalize this, we enumerate each state $\textbf{x}_{i}\in\mathcal{S_{A}}$ and construct a transition matrix $G$ with elements $G_{nm}$, where $G_{nm}$ represents the probability of transitioning from state $m$ to state $n$. If state $m$ does not transition to state $n$, then $G_{nm}=0$. For deterministic algorithms, $G_{nm}$ takes values of 0 or 1, reflecting the deterministic nature of state transitions. For probabilistic algorithms, $G_{nm}$ represents the probability of transitioning from state $m$ to state $n$. The discrete-time evolution of the probability distribution over the state space is described by the linear map $G$:

where $p_{i}$ denotes the probability distribution over the state space after step $i$. As the algorithm progresses and eventually halts, this distribution converges to a steady-state distribution.

We can apply the periodic mismatch cost discussed in Eq. (7). If algorithm starts with an initial distribution $p_{0}$. The mismatch cost for the first iteration is expressed as:

The distribution over the states of the algorithm evolves to $p_{1}=Gp_{0}$, but the prior distribution for the periodic process remains the same. Therefore, the mismatch cost in the second iteration is

The mismatch cost for the $i$-th iteration is given by:

If it takes $n$ iteration for the algorithm to halt, the total mismatch cost for running the entire algorithm is the sum of the mismatch costs across all iterations:

When choosing an initial distribution $p_{0}$, it is crucial to account for the properties of the variables involved in the algorithm. Typically, an algorithm starts with its line counter set to 0, and includes both input and non-input variables. Non-input variables may consist of loop counters, flags, and other auxiliary variables initialized to specific values. As a result, the algorithm’s initial state sets these special variables, like the line counter and loop counters, to their initialized values with certainty. Meanwhile, the input variables are sampled from a distribution $p_{\mathrm{in}}$, and the non-input variables are assumed to retain their values from the previous execution. For the purposes of this discussion, we assume that non-input variables retain the values at the end of the previous execution. Thus, the initial distribution $p_{0}$ over the state space of the algorithm is primarily determined by $p_{\mathrm{in}}$, the distribution over the input variables.

Similar to circuits, it’s important to consider the thermodynamic cost associated with updating the input variables with new values. When the algorithm is executed, its state variables retain the values from the previous execution. To reuse the algorithm for new inputs, the input variables must be rewritten, and the special variables need to be reinitialized to their starting values.

Let $p_{f}(\mathbf{x})$ represent the joint distribution of the algorithm’s state after the previous run. We will use the following notation: $\mathbf{x}_{\mathrm{in}}$ denotes the input variables, $\mathbf{x}_{\mathrm{nin}}$ represents the non-input variables, and $\mathbf{x}_{\mathrm{sp}}$ indicates the special variables, including the line counter.

Before the new input values are overwritten and the special variables are reinitialized, the state is given by the joint distribution:

Note that the initial distribution $p_{0}(\mathbf{x})$ is always a joint distribution, as all variables are statistically coupled at the end of the algorithm’s execution. After overwriting the input variables, sampled from $p_{in}(\mathbf{x}_{{\mathrm{in}}})$, and reinitializing the special variables, the distribution decomposes as:

where $p_{f}(\mathbf{x}_{\mathrm{nin}})=\sum_{\mathbf{x}_{{\mathrm{in}}},\mathbf{x}_{\mathrm{sp}}}p_{f}(\mathbf{x}_{{\mathrm{in}}},\mathbf{x}_{\mathrm{nin}},\mathbf{x}_{\mathrm{sp}})$ is the marginal distribution over the non-input variables, and $\delta(\mathbf{x}_{\mathrm{sp}})$ indicates that the special variables are reinitialized with probability 1.

The mismatch cost due to this overwriting process is given by:

where $q_{0}(\mathbf{x})$ is the prior distribution before overwriting, and $q_{1}(\mathbf{x})=p_{in}(\mathbf{x}_{{\mathrm{in}}})\delta(\mathbf{x}_{\mathrm{sp}})q_{0}(\mathbf{x}_{\mathrm{nin}})$ is the updated prior distribution after overwriting. This results in the following mismatch cost:

The thermodynamic cost of overwriting before reusing a computational system is not confined to algorithms or circuits. It also applies to any physical system involving overwriting, such as memory systems. For example, dynamic random-access memory (DRAM), solid-state drives (SSD), and flash memory all require periodic refreshing or updating of their stored information, leading to similar considerations of mismatch cost and heat dissipation.