Antifragile Perimeter Control: Anticipating and Gaining from Disruptions with Reinforcement Learning

Alleviating urban network congestion can be realized through various traffic control strategies. Since Daganzo (2007) proved the existence of the Macroscopic Fundamental Diagram (MFD) theoretically and Geroliminis and Daganzo (2008) demonstrated the presence of the MFD with empirical data, the relationship between traffic flow and density has been established through the aggregation of individual microscopic data points. This relationship has paved the path for the development of control strategies on a macroscopic level, enabling more computationally feasible real-time control strategies for large-scale networks (Knoop et al., 2012), such as perimeter control (Keyvan-Ekbatani et al., 2012; Geroliminis et al., 2013; Kouvelas et al., 2017; Yang et al., 2017), congestion pricing (Zheng et al., 2012; Zheng and Geroliminis, 2016; Genser and Kouvelas, 2022), route guidance (Yildirimoglu et al., 2015; Fu et al., 2022), and many more.

Perimeter control is among the strategies that have attracted immense attention and research efforts. Real-world implementation as shown in Ambühl et al. (2018) also demonstrated its applicability as an effective approach to regulating urban traffic. By refraining the incoming vehicles from adjacent regions into a protected zone, the traffic density in the protected area remains below the critical density, and a satisfactory level of service can be upheld (Keyvan-Ekbatani et al., 2012). Geroliminis et al. (2013) proposed an optimal perimeter control method using Model Predictive Control (MPC) and proved its effectiveness compared to a greedy controller in a cordon network. One major issue with the previous works is the MFD heterogeneity. To tackle this challenge, a substantial amount of effort has been made in investigating the partitioning algorithms so that a well-defined MFD can be referred to for a certain subordinated network (Ambühl et al., 2019; Saedi et al., 2020). However, MFDs in the real world can hardly be well-defined, as demonstrated in Ambühl et al. (2021) with loop detector data over a year. Wang et al. (2015) and Ji et al. (2015) also showed that adverse weather conditions and traffic incidents can alter the shape of the MFDs. Even a recovery from the peak-hour congestion may lead to a hysteresis (Gayah and Daganzo, 2011). These phenomena could potentially violate the mathematical model that serves as the foundation for the established model-based perimeter controllers.

To tackle the parameter uncertainties in the models caused by real-world disturbances, recent years have also witnessed a growing trend towards utilizing non-parametric learning-based approaches in traffic control (Nguyen et al., 2018). Among different ML algorithms, RL has been researched extensively in transportation operations, e.g., traffic light control (Wei et al., 2019; Chen et al., 2020), dynamic pricing (Wang et al., 2022), delay management (Zhu et al., 2021). Particularly for perimeter control, recent works from Ni and Cassidy (2019) and Zhou and Gayah (2021) have illustrated the capability of RL algorithms to achieve similar or even superior performance compared to the established control methods, with the advantage of the exact system dynamics being agonistic.

Before introducing antifragility, two associated terms robustness and resilience commonly used in evaluating traffic control strategies and how they can be induced with RL are introduced. As explained in Tang et al. (2020), these terms are sometimes used interchangeably across transportation research works and no unanimous is yet agreed upon. Therefore, we follow the principles proposed by Zhou et al. (2019), that robustness is concerned with assessing a system’s capacity to preserve its initial state and resist performance deterioration in the presence of uncertainty and disturbances, while resilience emphasizes the ability and speed of a system to recover from major disruptions to the original state. Note in this work we use disruptions to refer to the negative events more significant than daily disturbances but less severe than natural hazards or large-scale power outages that completely paralyze the whole traffic network. While researchers have endeavored to produce review articles in either the applications of RL in traffic control (Haydari and Yılmaz, 2022; Haghighat et al., 2020) or robustness/resilience in transportation systems (Zhou et al., 2019; Tamvakis and Xenidis, 2012), a collection of using RL algorithms in transportation to achieve robust or resilient urban networks remains absent. Here we reviewed some of the existing studies that apply RL to regulate urban traffic that have demonstrated robustness or resilience and are summarized in Table 1. By studying how robustness and resilience can be induced in RL algorithms, we can induce antifragility in our traffic control design through similar approaches.

Table 1 also summarizes the RL setups to demonstrate the property of being either robust or resilient of each traffic control strategy. In these papers, some authors proved the superior robustness or resilience of their proposed methods directly through testing against the established methods as benchmarks, whereas others induced such properties by adding specific terms (italicized in Table 1) in the state space of the RL-algorithm. As a result, the algorithms are given additional information related to disturbances or disruptions. For instance, Rodrigues and Azevedo (2019) induced robustness by adding the elapsed time since the last green signal for each phase, Tan et al. (2020) experimented with speed or pressure (residual queue) as an additional state representation in the state space of the RL algorithm, Chu et al. (2020) supplemented the control policies of neighboring intersections as additional information to the agents, and Zhou and Gayah (2023) used an extra binary congestion indicator in the state space. The analysis of the state-of-the-art RL studies considers reversals or sudden changes in the state-action-reward dynamics, which evokes unanticipated uncertainty. The problem in these contexts is often to respond to unexpected results appropriately since they might indicate a shift in the environment. In this case, exploration refers to the process of looking for new information to improve the RL agent’s understanding of the traffic dynamics under disruptions, which would then be used to identify better courses of action.

Ever since the concept of antifragility was proposed, it has become an increasingly popular concept in many disciplines, such as economy (Manso et al., 2020), biology (Kim et al., 2020), medicine (Axenie et al., 2022), and robotics (Axenie and Saveriano, 2023). However, current studies on antifragility in engineering mostly pertain to post-disaster treatment and reconstruction efforts, for example, in Fang and Sansavini (2017); Priyadarshini et al. (2022). Recent work in Sun et al. (2024) proved the fragile nature of road transportation networks and provided insights for antifragile network analysis and design. Leveraging the potential of antifragility for the operation of transportation systems is still a novel and unexplored notion.

In Taleb (2012), two levels of antifragility have been indicated: proto-antifragility and antifragility. To differentiate these two terms more clearly, we rename the latter and the more important one as progressive antifragility. In contrast to robustness or resilience, proto-antifragility describes systems that can improve performance from experiencing disruptions of similar magnitude. An illustrated example can be the biological process of hormesis. In this sense, proto-antifragility resembles the concept of adaptiveness, research works such as Fang and Sansavini (2017) are related to proto-antifragility of infrastructure under hazardous events. The higher level of antifragility, i.e., the progressive antifragility, emphasizes the concave response of the system regarding an increasing magnitude of disruptions, which has been proven in Sun et al. (2024) that urban road networks are progressively fragile by nature. A graphical comparison between robustness, resilience, adaptiveness, and antifragility is shown in Fig. 1. As previously mentioned, robustness refers to a system’s resistance to disturbances, without considering whether the system ultimately recovers. In contrast, resilience evaluates how quickly the system recovers from disruptions and the extent of performance loss, as illustrated by the shaded areas. A proto-antifragile (adaptive) system alleviates performance degradation after similar disruptions but does not necessarily guarantee continuous improvement as disruption magnitude increases. On the other hand, progressive antifragility ensures that the system experiences a damped increase in performance loss with linearly increasing disruptions, which can result from either the system’s innate antifragile characteristics or the implementation of antifragile control strategies. And a system demonstrating progressive antifragility through antifragile control is not inherently proto-antifragile on its own (Taleb and Douady, 2013). Likewise, fragility refers to systems that suffer from exponentially growing loss when faced with linearly increasing disruptions (Taleb and Douady, 2013), which is characterized by convexity and can be mathematically formulated with Jensen’s inequality $E[g(X)]\geq g(E[X])$. On the contrary, the nonlinear relationship between external stressors and responses for antifragile systems is concave with $E[g(X)]\leq g(E[X])$.

Researchers have also proposed methods to incentive antifragile property of a system by emphasizing the derivatives to capture the temporal evolution patterns of the system dynamics, i.e., how fast the system state deviates towards a possible black swan event, and the curvature of this deviation (Taleb and Douady, 2013; Taleb and West, 2023; Axenie et al., 2022). With this additional information, the system can anticipate ongoing disruptions and be more responsive to drastic changes. Similar to the function of redundancy in resilience (Tan et al., 2019; Kamalahmadi et al., 2022), redundancy can also be added in the system to induce antifragility de Bruijn et al. (2020); Johnson and Gheorghe (2013); Munoz et al. (2022). Other feasible approaches also include time-scale separation and attractor dynamics (Axenie et al., 2022; Axenie, 2022).

In RL algorithms, an agent or multiple agents interact with a preset environment and improve the performance of decision-making, defined as action $a_{t}$ in an action space $\mathcal{A}$, based on the observable state $s_{t}$ in the state space $\mathcal{S}$ and the reward, defined as $r_{t}=\mathcal{R}(s_{t},a_{t})$, where $\mathcal{R}$ is the reward function. The improvement of decision-making is commonly realized through a deep neural network as a function approximator. The RL algorithm applied in this work is Deep Deterministic Policy Gradient (DDPG) as proposed in Lillicrap et al. (2015). By applying an actor-critic scheme, DDPG can manage a continuous action space instead of only choosing from a limited set of discrete values as in the Deep Q-Network (DQN) algorithm (Mnih et al., 2013), which are commonly applied as Table 1 shows. Also, Zhou and Gayah (2021) has demonstrated that an RL algorithm with continuous action space can achieve better performance compared to discrete action space. The DDPG algorithm can be divided into two main components, namely the actor and the critic, which are updated at each step through policy gradient and Q-value, respectively. The scheme of the DDPG algorithm applied in this paper is schematically illustrated in Figure 3.

The state $s_{t}\in\mathcal{S}$ is defined distinctively according to different methods applied in this work. Our proposed method consists of three terms, the vehicle accumulation regarding the OD pair $n_{ij}(t)$, the change rate of vehicle accumulation at each time step $dn_{ij}(t)$, also referred to as the first derivative, and the change rate of change rate $d^{2}n_{ij}(t)$, or the second derivative or curvature. In Zhou and Gayah (2021), a state $s_{t}$ defined as $[n_{ij}(t),q_{ij}(t)]$ is adopted. However, since traffic demand in the real world is hardly measurable, $q_{ij}(t)$ would be an unobservable state for the agent. The action $a_{t}\in\mathcal{A}$ is defined the same as the control variables $u_{ij}(t)$. For the reward $r_{t}$, while Zhou and Gayah (2021) uses merely the completion rate, in our proposed algorithm, the reward is defined with an additional $r_{red}(t)$ term, as Eq. 4 shows.

The actor-network is represented by $\mu(\cdot)$ and it determines the best action $a_{t}$, which is the percentage of vehicles that are allowed to travel across the periphery, based on the current state $s_{t}$ and the weight parameters at a certain time step $t$.

The nature of the best action $a_{t}$ was also explored in the previous work of Axenie (2022), where in the framework of variable structure control, the authors demonstrate the need for a discontinuous signal. The critic network, denoted by $Q(\cdot)$, takes the responsibility to evaluate whether a specific state-action pair at a certain time step yields the maximal possible discounted future reward $Q(s_{t},a_{t})$. A common technique used in DDPG is to create a target actor network $\mu^{\prime}(\cdot)$ and a target critic network $Q^{\prime}(\cdot)$, which are a copy of the original actor and critic network but updated posteriorly to stabilize the training process and prevent overfitting (Zhang et al., 2021). The target maximal discounted future reward for the target critic network can be calculated as in Eq. 5.

Similar to DQN, the critic network can be updated by calculating the temporal difference between the predicted reward and the target reward and minimizing the loss for a mini-batch $N$ sampled from the replay buffer:

Afterward, the actor network can be updated with sampled deterministic policy gradient (Silver et al., 2014):

First developed in Horgan et al. (2018), a similar Ape-X architecture as applied in Zhou and Gayah (2021) is also adopted in this work, which allows for multiple generations of simulations to be included and centrally learned for the best policy.

When training the agent, we add disruptions into the simulation environment starting from a certain training episode, such as surging traffic demand or MFD disruption. These various forms of volatility ought to optimally elicit various learning and decision-making processes. The RL agent would include the results of their prior choices to create and update their weight parameters in a situation with high outcome volatility and be capable of generating and updating expectations after sensing a change in a high-volatility environment.

In this work, following the same idea of modifying the state $\mathcal{S}$ as in the research works in Tabel 1, we add additional terms based on derivatives (Taleb and Douady, 2013) and redundancy (de Bruijn et al., 2020), in both the state $\mathcal{S}$ and the reward function $\mathcal{R}$ of the RL algorithm.

The DDPG algorithm introduced in Zhou and Gayah (2021) for perimeter control requires knowledge of vehicle accumulation $n_{ij}(t)$ and a given demand profile $q_{ij}(t)$, which can be regarded as an estimate of average daily traffic demand, since the exact real-time demand is almost impossible to acquire. Therefore, we reduce the information requirement by replacing $q_{ij}(t)$ with the first and second derivatives of vehicle accumulation, denoted $dn_{ij}(t)$ and $d^{2}n_{ij}(t)$. These two derivatives can be calculated directly from $n_{ij}(t)$ and the value from previous time steps without excessive knowledge of $q_{ij}(t)$. Moreover, while the robustness of including $q_{ij}(t)$ against model stochasticity has been demonstrated in Zhou and Gayah (2021), the algorithm remains vulnerable to significant disruptions that cause substantial deviations from the average demand profile. Incorporating the derivative terms brings the advantage of feeding additional information similar to the demand profile $q_{ij}(t)$, but can easily adapt under the presence of disruptions.

However, when $dn_{ij}(t)$ and $d^{2}n_{ij}(t)$ are incorporated into the state $\mathcal{S}$ of the algorithm, significant oscillations in the perimeter control variables can be quite often observed, especially under scenarios with disruptions. As perimeter control is composed of coordinated traffic lights at the border of different regions, oscillating actions will result in significantly varying green splits between consecutive cycles. Even though oscillations may have little impact based on certain key performance indicators, the operation of traffic lights in the real world should be as stable as possible. Therefore, a damping term $r_{dam}(t)$ is introduced into the reward function to penalize potential oscillatory actions. The exponent $\xi$ determines how fast the penalty decays when the change of control variables becomes small, indicating a large $\xi$ mostly penalizes the agent when the oscillation is substantial.

For the reward function $\mathcal{R}$, the trip completion at each time step acts as the main component of our proposed method. The term $r_{red}(t)$ in the objective function $J_{\rm RL}$ in Equation (4) acts as an additional term to build up redundancy in the system. Similar to the creation of the additional term in the state space, we create redundancy also through the calculation of the derivates, but instead of the derivatives of the vehicle accumulation, we calculate the derivatives of the traffic state. This creates a one-to-one correspondence between the derivatives of the vehicle accumulation and the derivatives of the traffic state. To explain this antifragile term in the reward, we summarize $r_{red}(t)$ as the sum of two terms, with $H(t)$ being an overall term representing the first derivative of the traffic state and $\Delta H(t)$ representing the second derivative:

Here, $H(t)$ and $\Delta H(t)$ can be expanded as:

$h_{i}(t)$ and $\Delta h_{i}(t)$ are the first and second numerical derivatives of the traffic states on the MFD, $h_{i}(t)$ is defined as the difference of trip completion over vehicle accumulation at the end of a time step versus at the beginning of the same time step, as in Eq. 14 shows, and the second derivative $\Delta h_{i}(t)$ is calculated as the difference between the first derivatives of two consecutive time steps, as in Eq. 15 shows:

Since in the RL algorithms implemented in this paper, all variables involved in the deep neural network should be normalized to facilitate the training process, meaning the exact values of the derivatives are not of importance, $\omega_{h}$ and $\omega_{\Delta h}$ are introduced as the weight constants for the first and second derivatives to regulate their impact on the reward side $\mathcal{R}$.

The binary variable $\alpha_{i}(t)$ was designed in the first derivative to reward the agent when moving towards the desired direction on the MFD. For instance, the derivative of any data point in the congested zone of the MFD is negative. In this case, when the vehicle accumulation is still getting larger, a penalty will be applied. However, if the vehicle accumulation is decreasing through perimeter control, this binary $\alpha_{i}(t)$ variable will turn it into a reward. For the second derivative, an additional binary variable is not necessary since the two consecutive first derivatives are able to determine whether $\Delta h_{i}(t)$ is either positive or negative.

The term $f(n_{i}(t),n_{i,\rm crit},n_{i,\rm cap})$ is a reduction factor to constrain the impact of the $r_{red}(t)$ term when the accumulation is either on a very lower level (empty network) or on a very higher level (gridlock). The area near the critical accumulation is where the $r_{red}(t)$ term should have the greatest impact. Here we use a modified trigonometric function to realize this purpose. It should be noted that other functions, such as normal distribution, are also valid for achieving the same purpose.

After considering all the modifiers above, we show $H(t)$ and $\Delta H(t)$ using a single MFD as an example in Figure 4 and 5. The first derivative $H(t)$, in Figure 4(a), rewards the agent more when it’s moving towards the critical accumulation to maximize its trip completion rate. However, when the number of vehicles approaches the critical accumulation, this term drops significantly and becomes a penalty when it exceeds the critical point. Since the first derivative $H(t)$ is a complementary term in addition to the trip completion in the reward function, we showcase the influence of this term on the MFD after normalization in Figure 4. With increasing weight coefficient $\omega_{h}$, the critical accumulation of the modified MFD becomes marginally smaller compared to the original MFD, and the reward that the RL agent can receive also decreases faster after the accumulation exceeds the critical accumulation. Although the trip completion still follows the original MFD in the simulation environment, The agent learns to get more rewards following the modified MFD. In this way, redundant overcompensation has been established to prevent accumulation from exceeding the critical accumulation when disruption takes place unexpectedly.

An interesting note is that estimation uncertainty, also known as second-order uncertainty, is another factor that affects disruptions that take place unexpectedly. This is in fact the imprecision of the learner’s current beliefs about the environment, and what the antifragile terms capture. This amount reduces with sampling if beliefs are acquired by learning as opposed to instruction (e.g. anticipation through redundant overcompensation). When estimating uncertainty is substantial, unlikely samples could partly be attributed to the agent’s false assumptions about the environment’s structure rather than a change in that structure (e.g. around critical accumulation).

The second derivative $\Delta H(t)$ is shown in Figure 5. The x-axis is the vehicle accumulation, same as in Figure 4(a), while the y-axis represents how fast the traffic state is changing on the MFD. The faster it increases to reach the critical accumulation, the greater the penalty will be applied to the RL agent. This observation is consistent with the redundant overcompensation and time-scale separation principles formalized in Taleb and Douady (2013) and practically applied in Axenie (2022) and Axenie and Saveriano (2023). On the contrary, if the vehicle accumulation decelerates, a reward will be applied. Similar to $H(t)$, this complementary term $\Delta H(t)$ is also dependent on the normalization factor $\omega_{\Delta h}$.

With $H(t)$ and $\Delta H(t)$, the agent learns to be conservative when regulating the perimeter control variables when the accumulation is about to reach critical, in case disruptions take place. Therefore, as can be concluded, although $H(t)$ and $\Delta H(t)$ apply the same concept of the derivative as the $dn_{ij}(t)$ and $d^{2}n_{ij}$ in the state space $\mathcal{S}$, the purpose of $H(t)$ and $\Delta H(t)$ is preserving redundancy in the system instead of feeding additional information to the agent. This behavior is consistent with the locally discontinuous shape of the action signals $u_{ij}(t)$ applied to the cordon network, as suggested by the control theoretic study of Axenie (2022).

Model Predictive Control (MPC) is a well-established control method with wide applications in regulating dynamic systems in various engineering domains, including transportation and particularly perimeter control, Readers interested in Model Predictive Control (MPC) and its applications in traffic engineering can refer to Geroliminis et al. (2013); Haddad and Mirkin (2017); Zhou and Gayah (2021). The MPC toolkit applied in this paper is introduced in Lucia et al. (2017), which uses the CasADi framework (Andersson et al., 2019) and the NLP solver IPOPT (Wächter and Biegler, 2006).

As introduced in Section 3, we simulate a cordon-shaped urban network with inner and outer regions represented by different MFDs, as Fig. 2(b) shows. These MFDs are largely the same as in Zhou and Gayah (2021) and were originally approximated through the Yokohama loop detector dataset (Geroliminis and Daganzo, 2008). However, a minor modification has been made to allow its differentiability throughout the entire domain, since the MFDs in Zhou and Gayah (2021) are formulated as piecewise functions and are, although continuous, not differentiable. Indifferentiability within MFDs can cause fluctuations when calculating the derivatives and harm the efficacy of the redundancy term. Therefore, the gridlock accumulation of the MFD has been slightly increased with less than $3\%$, so that the whole MFD can be both continuous and differentiable. Other important indicators, e.g., critical accumulation and maximal trip completion, remain the same as in Zhou and Gayah (2021).

Although real-world disruptions can surely happen to both the outer and the inner region and exert a negative influence, we focus on the bottleneck scenarios where the potential of perimeter control can be better reflected. Therefore, for a demand disruption, a surging traffic demand is assumed to be generated from the outer region toward the inner region, whereas for a supply disruption, the MFD of the inner region is lowered as a result of such disruptive events while the MFD of the outer region remains intact. The two types of disruptions also demonstrate different training difficulties for the two RL-based algorithms involved, which will be detailed in Section 6. The traffic demand under no disruption is approximated based on Geroliminis and Daganzo (2008). As the real-world peak hour traffic demand profile has more resemblance to a Gaussian distribution instead of being a simple trapezoidal shape (Mazloumi et al., 2010), the base demand $q_{ij}(t)=[q_{11}(t),q_{12}(t),q_{21}(t),q_{22}(t)]$ is illustrated as the blue curves in Fig. 7(a). It consists of two components, which are a constant value $q_{ij,c}=[q_{11,c},q_{12,c},q_{21,c},q_{22,c}]=[0.3,0.35,0.2,0.25]$ in $veh/s$ and a Gaussian term $q_{ij,N}(t)=C_{ij,N}\mathcal{N}_{ij}(\mu_{ij,N},\sigma_{ij,N})$ with total number of vehicles following Gaussian as $C_{ij,N}=[3000,15000,2000,4000]$ and the mean and variance of the distribution being $\mu_{ij,N}=[1800,1800,1800,1800]$ and $\sigma_{ij,N}=[1200,1500,900,1200]$ in seconds. To simulate demand disruptions, $q_{12,N}$ is increased to reflect traffic flowing from the periphery into the city center. For testing progressive antifragility, a compounding factor of $0.018$ per episode is applied to escalate $q_{12,N}$, with the maximal demand disruption after 25 episodes of incremental disruptions reaching $(1.018^{25}-1)C_{12,N}\mathcal{N}_{12}(\mu_{12,N},\sigma_{12,N})$. For static demand disruptions to validate proto-antifragility, the magnitude is set to be $(1.018^{25/2}-1)C_{12,N}\mathcal{N}_{12}(\mu_{12,N},\sigma_{12,N})$.

Supply disruptions represent adversarial events that negatively affect the MFD, resulting in impaired traffic performance. However, as these events can arise in different forms, such as adverse weather conditions, traffic accidents, road maintenance works, etc., there has been limited research effort dedicated to understanding the exact correlation between such events and their impact on the MFD. Recent work in Ambühl et al. (2020) proposed the concept of infrastructure potential $\lambda$ to represent how efficiently the network infrastructure is used considering the performance loss due to vehicle infrastructure interaction, and a smaller $\lambda$ indicates the infrastructure is utilized with higher efficiency. In this research, a $\lambda$ equal to $0.07$ is chosen for the MFD without disruption representing the inner region, which lies in the common range of $\lambda$ based on data from real-world cities (Ambühl et al., 2020). And for the incremental magnitude of supply disruptions to test progressive antifragility, $\lambda$ is escalated by $0.003$ per episode with the maximal $\lambda$ after 25 episodes being $0.145$, as the red solid curve shows in Fig. 7(b). For static supply disruptions to test proto-antifragility, the magnitude of disruption is set to be half of the maximal disruption. Note as the polynomial MFD and the MFD approximated with $\lambda$ are essentially two different functional forms, there are still marginal deviations between each other after careful calibration of parameters, as illustrated with the blue and gray curves in Fig. 7(b).

The total simulation duration $T$ is 3 hours with each time step $\Delta t$ as $180$ seconds, and the third hour has significantly fewer vehicles so that the network would be able to clear the vehicles accumulated in the previous two hours, if there is no gridlock formed in the simulated network. Another notable constraint is the lower and upper bounds for the perimeter control variable $u_{ij}\in[0.1,0.9]$ to represent the real-world constraints from traffic signals (Geroliminis et al., 2013). The initial vehicle accumulation is set to be $n_{ij,0}=[600,1300,300,2400]$ so that the accumulation remains approximately at an equilibrium at the beginning of the simulation.

Each scenario is run $25$ times since randomness is inherent in RL and performance may vary across simulation iterations. Each iteration lasts for $75$ episodes, with the first $50$ under no disruption so that the RL agent can be properly trained and the following $25$ with either static or incremental disruption, and outputs the performance results from NC, MPC, RL baseline, and RL proposed, respectively. The number of episodes with disruption $25$ is deliberately set to be the same as the number of simulation iterations to improve the clarity of the results from the scenarios with disruption uncertainties, as illustrated by the zigzag lines for the simulation scenarios in Fig. 6. A list of multipliers $\epsilon=[\epsilon_{1},\epsilon_{2},\ldots,\epsilon_{24},\epsilon_{25}]$ following a normal distribution $\mathcal{N}\sim(1,0.15)$ is randomly generated to introduce uncertainties by calculating the cross product with the list of disruption magnitudes. The multiplier list will be shuffled by 1 for each simulation run, for example, the list would be $\epsilon=[\epsilon_{2},\epsilon_{3},\ldots,\epsilon_{25},\epsilon_{1}]$ for the second iteration. Through reshuffling, the disruption magnitudes from different episodes will experience exactly the same list of uncertainty multipliers, but with different sequences throughout the simulation iterations. The evaluation will be detailed in Section 5.2 together with distribution skewness as the quantitative indicator.

The most important hyperparameters for both the baseline and the proposed RL algorithms are the same and summarized in Table. 3 below. Note that the minimal learning rates and noise scale are not set to be an extremely small value as common RL algorithms do, we aim to get a trade-off between optimality and adaptiveness when the algorithm is experiencing disruptions, which can be demonstrated by the results for both good performance and being antifragile in Section 6. The exponent coefficient for the damping term $r_{dam}(t)$ to counter the potential oscillations actions in Eq. 10 is set to be $6$.

Although the reward in the RL algorithm is defined based on trip completion, together with a damping term and a redundancy term, to better showcase antifragility and the associated performance loss represented by the shaded area in Fig. 1 to coincide the relationship between resilience and antifragility, the main performance indicator being evaluated here is the Total Time Spent (TTS), which is calculated by adding up the number of vehicles within the network at each second of the simulation. Another major benefit of choosing TTS over completion is that two scenarios with the same trip completion at the end of the simulation may exhibit distinct TTS and the one with the lower value should be regarded to have demonstrated a superior performance overall.

Since urban road networks are always subject to capacity constraints, and as Sun et al. (2024) has recently proven the fragile nature of urban road networks in terms of progressive fragility, fully antifragile traffic control strategies may be impossible to achieve and will inevitably fall into being fragile when the magnitude of disruptions is large enough. Therefore, this paper aims to demonstrate that the proposed perimeter control algorithm is less fragile than the state-of-the-art baseline algorithms, and we normalize all the other perimeter control algorithms over the RL baseline method to study the relative antifragility. To quantify the progressive antifragility of different algorithms, we calculate the skewness of distribution based on the samples from the last 25 incremental episodes, with $\mu$ and $\sigma$ denoting the mean and the standard deviation:

When the performance distribution skewness of a certain system with or without a control algorithm is $0$, it means that the system itself or the applied algorithm makes the system neither fragile nor antifragile. On the other side, negative skewness indicates the distribution has a longer or fatter right tail and thus a higher degree of concavity in the performance function, which showcases antifragility, whereas a positive skewness signifies fragility. Therefore, other than demonstrating a superior performance regarding a lower TTS, the proposed method should also showcase a smaller skewness than the baseline method.

Systems with the property of proto-antifragility can learn from past adverse events and anticipate possible ongoing disruptions of similar magnitudes to enhance future performance. To validate such property, particularly for the RL-based algorithms, after training the agent with the base demand profile for 50 episodes, we apply disruptions with static magnitude for another 25 episodes.

Progressive antifragility describes a system that exhibits a nonlinear response to growing disruptions. In terms of TTS, since an increase in traffic density or vehicle accumulation within a network will inevitably reduce average vehicle speed or trip completion rate as per speed-density MFD (Geroliminis and Daganzo, 2008), and thus increase TTS, a concave response between TTS and vehicle accumulation can indicate a progressively antifragile system, and vice versa. Therefore, instead of 25 episodes of a static demand or supply disruption, a linearly increasing magnitude of demand or supply disruption is simulated, and skewness is calculated for each applied algorithm as the metric of (anti-)fragility.