Active Reinforcement Learning for Robust Building Control

Our focus is on robust RL for building energy consumption, which represent $73\%$ of electricity usage and $40\%$ of greenhouse gases in the US (U.S. Department of Energy—EIA 2020). Buildings are generating increasingly large amounts of sensory information that can be used to increase energy efficiency, such as temperature, airflow, humidity, occupancy, light, and energy usage (Hayat et al. 2019).

There are several ways buildings can be automatically controlled: heating, ventilation, and air conditioning (HVAC) units, energy storage systems, plug-in electric vehicles, photovoltaic power sources, and lights (Gong et al. 2022). We will focus on HVAC as it represents roughly a third of total building energy consumption (Wemhoff and Frank 2010). Traditionally, HVAC setpoint control has been approached through model-predictive control (MPC, Kou et al. 2021) or a heuristic rule-based-controller (RBC, Mathews et al. 2001).¹¹1”setpoints” are the numbers a human might input into their thermostat to tell the HVAC systems their desired temperature. ”Setpoint control” is the problem of automatically determining these setpoints to optimize some objective. MPC is generally not scalable to high dimensional input or output spaces compared to RL, and heuristics are inflexible.

Recently, RL-based HVAC setpoint control has grown in popularity(Das et al. 2022). Rizvi and Pertzborn (2022) used Q-learning to control HVAC setpoints in the presence of unseen disturbances. Kurte et al. (2020) demonstrated how RL and meta-RL could train an RL HVAC agent that quickly adapts to different buildings. Xu et al. (2020) explored how to use transfer learning to train an RL HVAC controller that outperforms a RBC baseline across a variety of simulated buildings and climates. Figure 1 illustrates the flow of information in our RL HVAC control setup.

As climate change continues, we will experience more droughts, heat waves, rising temperatures, and cold snaps (Masson-Delmotte et al. 2021). These extreme weather events are likely underrepresented in the training data, but are essential to account for in reliable and safe building control. For example, consider an RL controller trained on the dry weather of the fictional country of Desertland. If the climate of Desertland changes to have increased humidity, the RL controller may react by raising temperature during a heat wave, which is energy-inefficient, unacceptable for occupant comfort, and may even threaten occupant health.

To our knowledge, the problem of training RL HVAC controllers that are robust to changing weather is understudied, and current work focuses on detecting climate change and retraining RL controllers as their performance drops over time (Naug, Quinones-Grueiro, and Biswas 2022; Deng, Zhang, and Qi 2022), or training RL controllers that can be transferred to buildings in other climates(Xu et al. 2020; Lissa, Schukat, and Barrett 2020). These works focus on RL training pipelines that are robust to long-term changes in climate, but will still underperform during short-term extreme weather events that are rare in their training distribution.

The process of automatically selecting areas of the state space to explore is known as active learning, or as optimal experiment design. Active learning has mostly been explored for supervised learning. For example, Cohn, Atlas, and Ladner (1994); Faria et al. (2022) find regions of uncertainty in the data distribution through misclassification rates and output entropy. Makili, Sánchez, and Dormido-Canto (2012) uses conformal prediction to quantify the similarity of new data points to their dataset. EVOP (Lynch 2003) uses the sequential simplex method in order to identify experiment configurations that can maximize information gain. Bouneffouf (2016) use random exploration to identify new, promising data samples. Some of these concepts are already in use in many RL algorithms; for example, the RL algorithm we use in this paper (PPO Schulman et al. 2017), is incentivized to explore new data samples via random exploration and increasing the output action distribution entropy.

Our setting of Unsupervised Environment Design (UED, Dennis et al. 2020) is related, but different from active learning, in that we are not directly selecting training data points to sample, but selecting parameters of the environment that generates training data points. This is a more helpful problem setting in RL, as RL agents perform well with on-policy data that is collected as the RL agent explores the environment. UED is the problem of selecting new environment parameters that maximize the RL agent’s generalization across diverse environments.²²2Environments that can change behavior according to configuration parameters are often referred to as Procedural Content Generation environments (Risi and Togelius 2020)Parker-Holder et al. (2022); Dennis et al. (2020) find adversarial but feasible environment configurations with high regret; Jiang, Grefenstette, and Rocktäschel (2021) re-samples previously seen environments based on their 1-step TD error. These algorithms focus on training an RL agent that performs well across a distribution of similar tasks. SAMPLR (Jiang et al. 2022), a recent method, introduces the concept of curriculum induced covariate shift (CICS) and addresses it by launching several child simulations at each step to explore other step trajectories, but this approach does not scale computationally, especially when in an area such as building control, initializing state of the art building physics simulations comprises a large proportion of the overall computation.

We present a novel gradient-based algorithm for UED in building control called ActivePLR that leverages agent uncertainty. To the best of our knowledge, this is the first time neural network uncertainty has been incorporated into the problem of UED; most current works focus on some form of regret; this is also the first time environment configuration variables have been directly optimized under gradient ascent rather than through some evolutionary process (Parker-Holder et al. 2022), resampling procedure (Jiang, Grefenstette, and Rocktäschel 2021), or training a separate teacher network to select new environments (Dennis et al. 2020). To the best of our knowledge, we are also the first to focus on training RL HVAC agents that are robust to short-term extreme weather events rather than long-term climate change.

We demonstrate how ActivePLR trains RL HVAC controllers that are (1) more performant overall, (2) robust to extreme weather conditions, and (3) more robust to the Sim2Real transfer than the current state-of-the-art in UED.

RL is a framework for finding the optimal policy in an environment. Environments are formalized by a Markov Decision Process (MDP), which consists of a tuple $(S,A,T,R)$. State and Action spaces ($S$ and $A$) consist of tuples of some fixed length indexed per timestep $t$; $T\colon S\times A\to S$ is a transition function; and $R\colon S\times A\times S\to\mathbb{R}$ is a reward function. Agents choose actions according to a probability distribution $p_{\pi_{\theta}}$, determined by a policy $\pi_{\theta}$ with parameters $\theta$ to optimize the RL objective $J(\theta)$, defined $J(\theta)=\mathds{E}_{\pi}\left[\sum_{s_{t},a_{t}\sim p_{\pi}}[r(s_{t},a_{t})]\right]$. (See Sutton and Barto 2018.)

We use the PPO (Schulman et al. 2017) RL algorithm due to its performance and previous use in building control. Many works (Chen, Cai, and Bergés 2019; Zhang et al. 2021) have used PPO for HVAC control. Although we focus on PPO in this paper, our algorithm should easily extend to any actor-critic RL algorithm.

To estimate the uncertainty of our RL agent, we use Monte Carlo Dropout (Gal and Ghahramani 2016), in which nodes in the neural network are set to zero (“dropped out”) at random. This is used at inference time to generate multiple predictions for an individual input from different variations of the same model. The variance in these predictions is then used as a measure of the model’s uncertainty. We use Monte Carlo Dropout as opposed to other methods of estimating neural network uncertainty such as bootstrapped ensembles (Lakshminarayanan, Pritzel, and Blundell 2017) or Bayesian neural networks (Wang et al. 2020) because Monte Carlo Dropout is simpler and cheaper to train than bootstrapped ensembles and Bayesian neural networks while still providing good quantifications of uncertainty.

Formally, suppose we have a neural network $f_{\theta}\coloneqq\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ mapping $n$-dimensional input vectors to $m$-dimensional output vectors. $f$ is parameterized by a list of $l$ weight matrices $\theta\coloneqq{W_{i}|}_{i=1}^{l}$, where $W_{i}\in\mathbb{R}^{N_{i-1}\times N_{i}}$ denotes the weight matrix for layer $i$ of the neural network and $N_{i}$ denotes the number of neurons in that layer. We denote the dropout operation by $d_{p}\colon\mathbb{R}^{k}\rightarrow\mathbb{R}^{k}$, where $d_{p}(\theta)$ zeros-out each column of $W_{i}$ in $\theta$ with probability $p$. We define the uncertainty $L$ for input $x\in\mathbb{R}^{n}$ as $L(x,\theta)=\operatorname{Var}(f_{d_{p}(\theta)}(x))$, estimated by

Essentially, we conduct $C$ independent stochastic forward passes through the model with dropout at inference time, and use the sample variance of the outputs as our uncertainty metric. To estimate the uncertainty of our RL agent, we use the uncertainty of the critic network similar to other works (An et al. 2021; Wu et al. 2021).

One key assumption we make in the design of this algorithm is that, at the beginning of each training episode, we interact with an RL environment that can change its dynamics in response to some configuration parameters; for example, in this work we focus on a building simulation environment that can change its simulated weather patterns in response to weather configuration variables that we provide at the beginning of each training episode.³³3These environments are also known as Procedural Content Generation (PCG (Risi and Togelius 2020)) environments. Prioritized Level Replay (PLR, Jiang, Grefenstette, and Rocktäschel 2021) is a state-of-the-art framework for selectively sampling training levels in environments with procedurally generated content. Levels with higher value loss are prioritized, inducing an emergent curriculum of increasingly difficult levels.⁴⁴4In our case, a ”level” is just an environment configuration $\phi$. We use the term ”level” in describing PLR to be consistent with the original paper (Jiang, Grefenstette, and Rocktäschel 2021). For each episode, PLR samples $d\sim P_{D}$, to decide whether to sample a new level from the training distribution $\Lambda_{\text{train}}$ or pick one from the replay buffer $\Lambda_{\text{seen}}$. Jiang, Grefenstette, and Rocktäschel (2021) parameterized $P_{D}$ as a Bernoulli distribution with probability $p=\frac{|\Lambda_{\text{seen}}|}{|\Lambda_{\text{train}}|}$. Since we consider the setting where $\phi$ is continuous, $|\Lambda_{\text{train}}|$ is infinite, so we set the denominator as a hyperparameter $N_{PLR}$, so $p=\frac{|\Lambda_{\text{seen}}|}{N_{PLR}}$.

The probability of each level in the replay buffer being sampled is determined by the value loss, and how stale that estimate of the value loss is. If we do not sample a level from the replay buffer, we sample a new one from the training distribution and add it to the buffer. In this paper, vanilla PLR samples new levels to add to the buffer uniformly at random from the set of all possible training environments. Recently, Jiang et al. (2021) proposed Robust PLR, in which only training on the PLR-selected levels, and stopping gradient updates from the randomly selected levels, generally performs better; we will refer to this variant as RPLR.

We present ActivePLR: a novel addition to PLR that samples new levels to add to the replay buffer through an uncertainty-based optimization procedure instead of at random.⁵⁵5We also tried to use RPLR with our uncertainty-based approach, but we found it performs worse than ActivePLR In order to generate new environments at the frontier of the agent’s uncertainty, we will backpropagate gradients from the uncertainty back to the state variable and do gradient ascent.⁶⁶6We make the assumption that at least some of the environment configuration variables are continuous This will allow us to find the state at which the agent is most uncertain, and generate a new environment that allows the agent to interact with the world at that state. Formally, assume we have a environment $E$ with some parameters $\phi\in\mathbb{R}^{k}$ that are part of the agent’s initial state space $S$. That is, the state $s_{0}\in\mathbb{R}^{n}$ can be divided into $\phi$ and $\overline{s_{0}}$, where $\overline{s_{0}}\in\mathbb{R}^{n-k}$ is defined such that $s_{0}=[\phi,\overline{s_{0}}]$ (and $[]$ is the concatenation operator). We can define an objective function that tries to maximize the uncertainty of the RL agent:

where $L$ is our uncertainty estimate. We can then update $\phi$:

We use this optimization procedure to identify novel training environments to add to PLR’s environment replay buffer. We call this procedure ActivePLR, as it is an active learning method that seeks to identify what data would be most useful for the RL agent.⁷⁷7Pseudocode for ActivePLR can be found in Algorithm 1, and an illustration can be found in Figure 3. We can also use this optimization procedure to generate all the training environments instead of using a replay buffer.⁸⁸8This is equivalent to ActivePLR with $P_{D}$ assigning $100\%$ probability to $d=0$ We denote this case as ActiveRL.

Trying to identify parameters $\phi$ by maximizing uncertainty can lead to unrealistic parameters that are outside of the test distribution and not useful for learning. Thus, we integrate both hard constraints and soft constraints on $\phi$ generation in ActivePLR. The hard constraints are useful when trying to train an RL agent that generalizes over a certain region of the $\phi$ space, and the soft constraints are useful when trying to train an RL agent that emphasizes performance near a particular $\phi_{0}$, which is still robust to different values of $\phi$. We will refer to the latter setting as the $\phi_{0}$-neighborhood setting for brevity.

The hard constraints constrain the search space within some lower and upper limits specified by the user for $\phi$ using the extragradient (Korpelevich 1976) method. Suppose we have a lower bound constraint $\phi>b$ for some $b\in\mathbb{R}^{k}$ and an upper bound constraint $\phi<a$ for some $a\in\mathbb{R}^{k}$. Then we can use Lagrangian optimization methods from the Cooper library (Gallego-Posada and Ramirez 2022) to search for a $\phi$ with high uncertainty within the bounds of $a$ and $b$, helping to avoid unrealistic values of $\phi$.⁹⁹9A more detailed treatment of the constrained optimization problem can be found in Appendix A. Throughout this paper, we will use the implementation of ExtragradientAdam from Gallego-Posada and Ramirez (2022), which adjusts the learning rate $\eta$ for each parameter according to Adam (Kingma and Ba 2014).

In the $\phi_{0}$-neighborhood setting, the hard constraints are not enough – there is no guarantee that states near $\phi_{0}$ will be sampled. Thus we introduce a soft constraint to $O$ to minimize the Euclidean distance from $\phi$ to $\phi_{0}$.

where $\gamma$ emphasizes the soft constraint.

We use a modified version of the Sinergym (Jiménez-Raboso et al. 2021) OpenAI Gym (Brockman et al. 2016) environment to simulate buildings in different weather conditions. The environment uses the EnergyPlus (Crawley et al. 2001) simulation engine to model the dynamics of building systems. We use the ”5Zone” building provided with Sinergym: a $463.6m^{2}$ single-story building with a DX cooling coil and gas heating coils that is divided into 5 zones (1 indoor and 4 outdoor). Actions in Sinergym are continuous, two-dimensional vectors, where the agent can control the heating and cooling setpoints for the HVAC systems. We use a reward function that rewards high occupant comfort and low energy use. To quantify occupant comfort, we use the Fanger Percentage of People Dissatisfied (PPD, Fanger 1967). Energy use is the total HVAC electricity demand rate in Watts (W). The reward can be formulated as:

where $\rho$ controls how much to weight comfort against energy use, $P_{t}$ is the electricity demand rate, $\lambda_{E}$ and $\lambda_{P}$ are scaling factors to account for varying units, $\mathds{1}_{(occupancy_{t}>0)}$ ensures there is no penalty for uncomfortable conditions when there are no occupants, and $\mathds{1}_{PPD_{t}>20}$ ensures the agent is not penalized if the PPD is below the ASHRAE guidelines’ comfort threshold of $20\%$ (ANSI and ASHRAE 2017). We use $\lambda_{E}=0.0001$, $\lambda_{P}=0.1$, $\rho=0.5$. The state is a continuous, 20 dimensional vector that includes 5 outdoor weather variables: outdoor air temperature, outdoor relative humidity, wind speed, wind direction, solar irradiance, and 15 other variables: indoor air temperature, indoor relative humidity, clothing value, thermal comfort, current HVAC setpoints, total HVAC electricity demand rate, occupancy count, and date (year, month, day, hour).

To simulate outdoor weather, Sinergym takes as input a file with hourly measurements of each outdoor weather variable. Originally, Sinergym added noise to outdoor temperature through an Ornstein-Uhlenbeck (OU, Doob 1942) process, to help prevent overfitting the agent to a static weather pattern. We modified Sinergym so it could add this noise to the other outdoor weather variables as well. An OU process has three parameters: $\sigma$, $\mu$, and $\tau$. $\sigma$ controls the variance of the added noise, $\mu$ is the average value of the noise, and $\tau$ determines how quickly the noise reverts to the mean.

We can obtain reasonable values for $\sigma$ and $\tau$ for each weather variable from the original input weather file, so we have 5 remaining parameters to customize Sinergym: the $\mu$ offset parameters for each weather variable.¹⁰¹⁰10See Appendix B for details Thus the $\phi$ for Sinergym that we vary to attempt to train a robust RL agent, is the 5 dimensional vector $<\mu_{1},\mu_{2},\mu_{3},\mu_{4},\mu_{5}>$. These essentially change the average outdoor temperature, relative humidity, wind speed, wind direction, and solar irradiance over the course of the simulation. Varying the environment configuration $\phi\in\mathbb{R}^{5}$ enables us to collect training data from diverse outdoor weather conditions.

All experiments used 24 Intel Xeon E5-2670 CPUs.¹¹¹¹11Code is available at https://github.com/Demosthen/ActiveRL

Our basic RL baseline is a PPO agent composed of a neural network with two hidden layers, each with 256 neurons, using dropout and ReLU activations, that is trained on $\phi_{0}$.

The most common method of training agents that generalize across diverse environments is domain randomization (DR), where $\phi$ is selected uniformly at random. This method is often used so agents can transfer from simulation to the real world (Chen et al. 2021; Vuong et al. 2019; Tobin et al. 2017). In the buildings domain, Jang et al. (2021) used DR to train an RL energy pricing agent to be robust to the Sim2Real transfer. In our setting: we have some lower bounds $a\in\mathbb{R}^{5}$ and upper bounds $b\in\mathbb{R}^{5}$ for each variable, described in Table 1 in the appendix¹²¹²12The appendix can be found. We sample $\phi\sim U(a,b)$, where $U$ is the uniform distribution.

Jiang et al. (2022) propose Sampled Matched PLR (SAMPLR) to generalize across diverse environments while combating the problem of curriculum-induced covariate shift (CICS), in which the distribution of training environments ($\phi\sim P$) generated through UED may become too different from the distribution of test environments ($\phi_{test}\sim\bar{P}$). Unfortunately we were not able to use SAMPLR as a baseline: SAMPLR resets the simulator at every timestep to collect fictitious trajectories; thus, SAMPLR is infeasible when a reset is expensive compared to a timestep, which is true for Sinergym and many other simulators.¹³¹³13Sinergym takes $\sim 3$ seconds to reset to a new weather pattern. Each of our experiments involved training for 3M timesteps, where episodes were reset every 8760 timesteps. Implementing SAMPLR would have increased the time spent resetting Sinergym by 8760x, for a total of 9M seconds, or 104 days.

We use a rule-based controller (RBC) based on Sinergym’s RBC, and a random controller as baselines. The random controller outputs a random cooling setpoint $a[0]\sim Uniform(22.5,30.0)$ and a heating setpoint $a[1]\sim Uniform(15,22.5)$.¹⁴¹⁴14The specific values are taken from Sinergym. Sinergym’s RBC sets the desired temperature range ($26$-$29\text{\textdegree}C$) higher in the summer and lower in the winter ($20$-$23.5\text{\textdegree}C$) to minimize energy consumption. We added a rule that if no occupants are in the building, the RBC sets setpoints with a wide enough range that the HVAC system is turned off. We added this new occupancy-based rule for the sake of a fair comparison because we included occupancy information in the reward.

Our last RL baseline, RPLR, is described in Section 2.

Hyperparameters are in Table 2 in the appendix.

In order to evaluate how robust agents trained by each algorithm are to extreme weather events, we evaluate each agent in a suite of 5 different extreme weather environments parameterized by 5 different $\phi$ weather configurations, as well as $\phi_{0}$ for a total of 6 environments. The agent is trained on automatically generated environments according to each UED algorithm. It is then evaluated in the following 6 environments: $\phi_{0}$ simulates realistic weather based on recordings from Arizona, USA, $\phi_{1}$ simulates an extremely hot and dry drought, $\phi_{2}$ simulates a wet and windy storm, $\phi_{3}$ simulates a humid heatwave, $\phi_{4}$ simulates a cold snap, and $\phi_{5}$ simulates erratic weather. Our hypothesis is that using uncertainty to identify new environments to collect data from will allow us to train RL agents that are more robust to extreme weather conditions.

In order to test whether or not the RL policies trained in simulation can be extended to the real world, we evaluated each RL algorithm in each of the 6 environments by running the EnergyPlus simulator at a higher fidelity than the agents were trained on, thus simulating the ”Sim2Real” jump with a more realistic simulator. The agents were trained on a simulator operating at a granularity of $\Delta t=1\text{hour}$ per timestep, and we evaluate on a granularity of $\Delta t=0.25\text{hours}$ per timestep. During evaluation, each of the RL agents’ actions are simply repeated four times so that it still takes an action every hour. Our hypothesis is that by increasing the state space supported by the training distribution, ActivePLR will help the agent be robust to compounding errors caused by the Sim2Real jump.

In order to evaluate how robust agents trained by each algorithm are to extreme weather, we evaluate the agent in a suite of 6 different environments, parameterized by 5 different $\phi$ extreme weather configurations and $\phi_{0}$. Figure 4 shows the overall performance of each algorithm. See Figure 5 in the appendix for performance in specific environments.

Surprisingly, DR and RPLR did not have significant improvements over the vanilla RL algorithm. By the end of training, DR and RPLR achieved $9\%$ higher reward than the RBC on the base environment $\phi_{0}$ after 3M timesteps of training. However, the vanilla RL policy had a $8\%$ improvement over the DR and RPLR policies with $\phi_{0}$. Over the 5 extreme weather environments, DR did about as well as the RBC. The unexpected lack of performance gain may mean the environments generated with DR were too unrealistic to learn how to perform in extreme weather conditions.

Over the extreme weather conditions, RPLR beat DR and the RBC, but performed worse than the vanilla RL policy. This indicates that as RPLR uses DR to sample new environments, it may still suffer from generating unrealistic environments. However, its weighted environment resampling procedure helps it generalize better than naive DR, even though it is resampling from unrealistic environments. We found that training the HVAC controller with ActivePLR resulted in agents that performed better in both the extreme environments and the base environment.

Generally, ActiveRL and ActivePLR performed similarly. In three out of the five extreme environments: (1) the hot drought, (2) the cold and windy, and (3) the cold snap environment, ActivePLR performed significantly better than all other baselines and performed competitively in the remaining two environments. In the base environment, ActivePLR provides a $9\%$ improvement over vanilla RL, and a $24\%$ improvement over RBC. Over all 6 environments, it provides a $3\%$ improvement over vanilla RL. We also found that over the 6 environments, ActivePLR and ActiveRL provided a $15\%$ relative decrease in days with ASHRAE thermal comfort violations compared to vanilla RL (which was the best baseline in terms of thermal comfort), resulting in significantly more comfortable occupants even during extreme weather conditions.¹⁵¹⁵15ASHRAE defines uncomfortable thermal conditions as at least $20\%$ of occupants are predicted to be uncomfortable (PPD $>20\%$) The fact that ActivePLR significantly outperforms all baselines indicates there is considerable value in seeking out realistic new training environments that maximize an agent’s uncertainty rather than choosing environments at random or merely replaying old ones.

One flaw in this work and UED algorithms in general is that a simulator is required to train the model in different environments. Thus, it is important to ask whether or not the RL policies trained in simulation can be extended to the real world. In order to approximate the Sim2Real gap, we conducted evaluated each RL algorithm in each of the 6 environments by running the EnergyPlus simulator at a higher fidelity than the agents were trained on, so that these test environments (1) had slightly different dynamics from the original training simulation, which should give rise to similar distribution shift issues as the Sim2Real gap, and (2) had dynamics that were as close to those of the real world as possible because the test simulation was run with higher fidelity and should therefore be more accurate to real dynamics than the training simulation. An illustration of the performances of each algorithm on each of the 6 handcrafted environments from Section 4 are shown in Figure 4.

When the agents are transferred from the simulation to our surrogate for the real world, we see there is a significant performance drop across all data-driven algorithms. Vanilla RL achieves a reward that is $8.5\%$ lower on average across the 6 handcrafted environments when evaluated on the higher fidelity simulation. DR and RPLR have smaller relative drops of about $7\%$. However, since the $7\%$ is relative to the performance of DR and RPLR in the original low fidelity simulation, vanilla RL still performs better in terms of absolute reward over the six environments. Random and RBC have very small or no relative performance degradation, which is to be expected because they are not data-driven models. They still perform the worst in terms of absolute reward. ActiveRL and ActivePLR, however, achieve both smaller relative drops in performance and higher absolute reward across all the different extreme weather scenarios. ActiveRL has only a $6.1\%$ relative drop in reward while ActivePLR has only a $3.1\%$ relative drop. These are promising results that indicate that these algorithms would still perform well if deployed in the real world after being trained in simulation. Furthermore, these algorithms result in agents that are more robust to the Sim2Real transfer than other methods.

We found that ActivePLR trains agents that are more robust to the Sim2Real transfer than ActiveRL, which is surprising since ActivePLR was not significantly different from ActiveRL in the extreme weather experiment. There might be some attractive local optimum in the HVAC control task in the low fidelity simulation that both ActivePLR and ActiveRL fall into that is not present in the high fidelity simulation, resulting insimilar performance in the experiments from Section 4 but better Sim2Real transfer. In addition, the recorded value loss that RPLR and ActivePLR use is likely a less noisy signal of environment curriculum value than the uncertainty over the value estimate that ActiveRL uses. The value loss is obtained by actually collecting data while the value uncertainty is estimated using only the model weights through Monte Carlo Dropout.