Price-responsive control using deep reinforcement learning for heating systems: Simulation and living lab experiment

Contents lists available at ScienceDirect Energy journal homepage: www.elsevier.com/locate/energy Price-responsive control using deep reinforcement learning for heating systems: Simulation and living lab experimentI Reza Mokhtari a,∗, Mina Montazeri b, Hanmin Cai b, Philipp Heer b, Rongling Li a aTechnical University of Denmark, Department of Civil and Mechanical Engineering, Kgs. Lyngby, Denmark bUrban Energy Systems Laboratory, Swiss Federal Laboratories for Materials Science and Technology, Dübendorf, Switzerland A R T I C L E I N F O Keywords: Deep reinforcement learning Building energy management Price-responsive control Smart buildings Demand side flexibility A B S T R A C T While buildings are one of the main sources of global carbon emissions, they also present significant opportunities for decarbonization. Building energy systems are typically controlled using pre-defined schedules and rule-based strategies, which usually lack adaptability to dynamic conditions such as changing weather states, occupancy and energy prices. In this study, we develop a price-responsive Deep Reinforcement Learning (DRL) controller to autonomously control the indoor temperature of a building. The designed controller aims to minimize heating costs while keeping the indoor temperature within the acceptable comfort bounds, responding to dynamic energy prices. To improve learning efficiency, this study introduces TemporallyWeighted State Difference (TSD), a method to compress future price data into a single state observation for the controller, enabling future-aware decisions while reducing training complexity. The controller is developed and trained in a simulation environment using a physically consistent neural network model. It is then implemented in a real-world residential unit in the NEST building in Dübendorf, Switzerland, and evaluated during the winter of 2024 to 2025 using the dynamic electricity prices of the Swiss market. Experimental results indicate that incorporating future price information leads to more informed decisions by the controller and improves the performance of DRL by 11.5%. Experimental results show a 79% reduction in heating costs compared to a rulebased controller, at a cost of exceeding the thermal comfort range by 0.3 ◦C on average. The study demonstrates the potential of price-responsive DRL in reducing energy costs and providing demand-side flexibility to the energy grid. 1. Introduction In recent years, buildings have received significant attention for decarbonization, as they are among the main contributors to climate change with a great potential for mitigation (i.e., 50%–90%) [1]. This potential can be tapped by using energy-efficient Heating, Ventilation, and Air Conditioning (HVAC) systems, retrofitting building elements for reduced heat losses, and utilizing advanced control systems [2]. Control systems are the arms of an energy-efficient building, responsible for managing energy resources to meet certain demands (e.g., heating and cooling) within the building. These demands are often flexible in time and quantity, which leaves a huge potential for controllers to act as a tool for providing energy efficiency [3]. Most buildings currently rely on simple pre-defined schedules and setpoints, so-called RuleBased Controllers (RBC), for control. Although simple to implement, they cannot adapt to dynamic environments (i.e., weather, price, occupancy, etc.) and operate efficiently [4]. There has been significant IThis article is part of a Special issue entitled: ‘ICDIES2025’ published in Energy. ∗Corresponding author. E-mail address: [email protected] (R. Mokhtari). interest in developing advanced controllers for buildings, including Model Predictive Control (MPC), Reinforcement Learning (RL), and fuzzy-based controllers [5,6]. To ensure the occupant comfort condition is maintained, these studies monitor indoor conditions by sensors, user feedback, etc., while conducting control for demand response [7]. According to the latest Energy Performance of Buildings Directive (EPBD) [8], ‘‘A zero-emission building shall, where economically and technically feasible, offer the capacity to react to external signals and adapt its energy use, generation or storage’’. This highlights the current importance of signal-responsive buildings and demand response in addressing climate change. The advanced controllers in buildings must provide demand response services to the grid and mitigate demand and supply mismatches [9]. However, an external signal is required for communication between the grid operator and the responsive building’s controller. Dynamic energy price can be a proper external signal for this purpose, as it is often an indicator of supply availability and https://doi.org/10.1016/j.energy.2025.138517 Received 30 June 2025; Received in revised form 3 September 2025; Accepted 14 September 2025 Energy 337 (2025) 138517 Available online 19 September 2025 0360-5442/© 2025 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). R. Mokhtari et al. carbon intensity of the grid [10]. Hence, price-responsive controllers can unleash the hidden potential of buildings and strategically shift energy consumption to low-cost periods, reducing energy costs for end users and helping to balance demand and supply. Among the available advanced controllers, Deep Reinforcement Learning (DRL) has received significant attention recently, especially for HVAC control [11]. This method integrates deep neural networks and enables the agent (i.e., HVAC system) to learn optimal control policies through interaction with the environment and receiving rewards based on the performance of each policy. Unlike MPC which requires an explicit physical model of the building to operate, DRL benefits from being model-free in operation. DRL operates in a model-free manner, relying solely on a simulator for interaction during the training phase. Although the performance of DRL highly depends on the training stage and is not guaranteed to be optimal [12], studies show the advantage of DRL over MPC in being more efficient and scalable [13]. Several studies have applied DRL to building HVAC control, aiming to reduce energy costs while maintaining thermal comfort. Typically, these studies define multi-objective reward functions that balance energy consumption against temperature deviation penalties [14], and design the state space to include weather conditions, occupancy patterns, and indoor thermal states [15]. Many works have introduced electricity price signals into the DRL for improved demand response capacity and to facilitate the integration of renewable energy sources in buildings [16]. For instance, Biemann et al. [17] used a recurrent neural network (RNN) for cooling control in data centers and incorporated real-time electricity price into the state space, reporting cost savings compared to ordinary DRL and rule-based controllers (RBC). Xie et al. [18] proposed a multi-agent DRL setup for grid-interactive buildings, using dynamic prices in both the state and reward functions to flatten the aggregated load curve and reduce peak demand. Li et al. [19] implemented a time-of-use (TOU) price signal in the reward function for a cooling system with thermal storage, where the reward dynamically adjusted its weighting based on price levels. The result was a 9.17% cost reduction. Du et al. [20] employed the Deep Deterministic Policy Gradient (DDPG) agent algorithm to control multi-zone heating systems, incorporating current retail prices to enable pre-heating during low-price periods. Similarly, Liu et al. [21] developed a multi-step DRL prediction framework, using real-time price as an external state input, and reported improved cost-efficiency over traditional control methods. While these studies show that price-responsive DRL can improve cost savings, they share two main limitations. First, most of them rely solely on simulation without experimental test and validation [22–24], making it difficult to assess their real-world applicability. Second, even in studies that include electricity prices into the controller state or reward function, the agent typically only has access to current or historical prices. In contrast, in many real-world electricity markets (e.g., day-ahead markets), future prices within the same day are known in advance. Neglecting this information limits the agent’s ability to make the optimum decision. This study addresses these gaps by introducing a novel Temporally-Weighted State Difference (TSD) approach that encodes the future price trends into a single state, enabling the agent to make informed decisions without increasing the controller complexity. To evaluate its effectiveness, we first test the method in a simulation and compare it with other DRLs. We then conduct a realworld experiment in a residential unit of the NEST building at Empa, using early-2025 electricity prices [25]. The proposed DRL controller is tested under fixed, random, and dynamic prices and compared with a default rule-based control strategy. The paper is organized into nine sections. In Section 2 we introduce the fundamentals of DRL as the theoretical foundation of this study. Section 3 is problem formulation, and the main problem and the process of conducting the study are described. Afterwards, in Section 4, the building thermal model, being the simulation environment as the testbed, is explained. Section 5 is about the design of the thermal controller and introducing the TSD method. In Section 6, the case study to conduct the simulations and experiments is introduced. Eventually, Section 7 is dedicated to presenting the results of training, simulations and experiments. The main points are discussed in Section 8, and the paper is concluded with the key findings and lessons learned in Section 9. 2. Preliminary This section presents the fundamental governing equations for the DRL framework and the agent. These equations form the theoretical foundation of the paper for conducting the simulations and experiments. 2.1. Deep reinforcement learning In reinforcement learning, at each timestep, an agent interacts with an environment in a loop. It first observes a state, representing the current situation, takes an action with a control variable, and receives feedback or reward based on the outcome of that action. This loop is iteratively used to allow the controller to find the best decisions in different situations. Deep reinforcement learning is a subset of reinforcement learning relying on deep neural networks to handle highdimensional state spaces. DRL is shown to be a promising approach to handle problems with the Markov property, i.e., the future state depends only on the current state and actions and not the previous states or actions [26]. DRL works well in contexts with predictable patterns such as daily prices, weather trends, etc. In indoor thermostat, specifically, the thermal inertia of the building helps the controller to take cost-saving decisions such as preheating in cheap hours. The algorithm examines the current state given the observation vector 𝑠𝑡∈𝑆 at each timestep 𝑡, and takes the action 𝑎𝑡∈𝐴. 𝑆 and 𝐴 are the state and action spaces, respectively. Since the control variable in this problem is continuous, we consider a Deep Deterministic Policy Gradient (DDPG) for the DRL, which is proven to be a powerful algorithm for handling continuous state and action spaces [27]. DDPG combines ideas from DPG and Deep Q-learning, as it uses two separate neural networks for an actor and a critic. The critic network is responsible for estimating the value of the actions 𝑄(𝑠, 𝑎|𝜃𝑄), and the actor network is used to learn deterministic policies for mapping states to actions 𝜇(𝑠|𝜃𝜇). Critic and actor networks are neural networks that require training, and the size of the network should be selected based on the state space size. DDPG agent is trained with the goal of finding the best actions 𝑎𝑡 at each state 𝑠𝑡 leading to maximum cumulative expected reward E[∑∞ 𝑘=0 𝛾𝑘𝑟𝑡+𝑘], where 𝛾 is the discount factor. The discount factor is used to adjust how the DRL should value immediate rewards over future rewards. DDPG uses Bellman equation to determine the value of the action-state pair [28]: 𝑄𝜇(𝑠𝑡, 𝑎𝑡) = E[𝑟(𝑠𝑡, 𝑎𝑡) + 𝛾𝑄𝜇(𝑠𝑡+1, 𝑎𝑡+1)],(1) where 𝜇 is the set of policies that connects the desirable action 𝐴 to state space 𝑆. DDPG algorithm utilizes a replay buffer for storing experience and reusing it for more stable learning. To ensure sufficient exploration, DDPG uses the Ornstein–Uhlenbeck noise process (𝑡) which adds exploration noise for continuous actions. This noise helps exploration in a temporally correlated way, which is useful in this case. The reward function is formulated to calculate the reward of each action that the controller takes and use it as a reference for the evaluation of each action. Through time, the controller moves towards actions leading to positive rewards and tries to avoid actions with negative rewards. Therefore, it is crucial to formulate the reward function carefully. Energy 337 (2025) 138517 2 R. Mokhtari et al. Fig. 1. The workflow for implementation of the controller in the test building. The blue symbol in the boxes indicates that the module is currently being trained, while the green symbol means that the module is already trained and fixed. 3. Problem formulation The objective is to control the building’s indoor temperature with a DRL to achieve heat cost savings while maintaining thermal comfort. This requires initial training of the controller in a simulator, followed by its deployment in the test building. The steps are illustrated in Fig. 1. Step 1 involves training the Physically Consistent Neural Network (PCNN) model, which serves as the building thermal model and simulation environment for DRL. This training utilizes historical data, including valve opening, weather conditions, room temperature, and time variables. Further details on the PCNN are provided in Section 4. In step 2, the trained PCNN is used as the testbed for training the DRL controller. DRL explores actions in different states through an iterative learning process. When the training is converged, the best iteration of DRL is selected for the control. In step 3, the DRL is implemented for a real-world test. The electricity price is passed to the controller, and the future prices are converted to the 𝑠𝑡 within the controller. The DRL communicates with the building energy system through MQTT, a widely used standard for IoT messaging [29]. The MQTT broker receives data from different units and passes it to the right assets at each time. The DRL receives the observation states together with the electricity price, and sends back the valve control signal to the MQTT broker. The broker then sends the signal to the heating valve of the specified room. The monitoring apparatus for the MQTT is set to 5 s, so that the controller gets updated measurements and sends control signals every 5 s. 4. Building thermal model The training of DRL is usually associated with the exploration of extreme actions, which can lead to thermal comfort violation in the building. Therefore, a model of the building is generally utilized to train the DRL on generated episodes and then to test the obtained policy on a real case study. Here, we use a physically consistent neural network model as a simulation testbed for training DRL [30]. PCNNs use a linear model for physical knowledge of the building and a nonlinear datadriven neural network for modeling the nonlinearities and dynamics of the building. The advantage of PCNNs over other existing methods is that they do not require physical models and use solely data to fit the model, but still stick to physical principles. The inputs of the PCNN model are ambient temperature, solar irradiation, neighboring room temperature, valve opening, and time features such as day of week, month, and time of day. The model then predicts the room temperature using the trained networks. For further details of PCNNs, please refer to [30]. To train the PCNN model, historical data for the target room from November 2024 to March 2025, with a 15 min interval, are used. 5. Thermal controller 5.1. Input states State space selection is a critical step in designing the DRL. The states should cover the essential variables influencing the control behavior. Weather data is considered one of the most influential parameters affecting the thermal behavior of buildings. More specifically, ambient temperature 𝑇𝑎𝑚𝑏 𝑡 and solar irradiation 𝐺𝑠𝑜𝑙𝑎𝑟 𝑡 are considered in the state vector. They are collected in real-time by the local weather station on the building. To give the controller feedback on the indoor conditions, the indoor temperature of the building 𝑇𝑧𝑜𝑛𝑒 𝑡 is considered as another state. DRL can learn time-dependent patterns of variables through training. For example, it can learn that solar irradiation would be maximum at noon and it would take action accordingly. Also, other hidden time-dependent behaviors, such as internal gain, can also be captured by the controller. As another input state, the price per consumption of heat 𝑢𝑡 is considered in the states. This enables the controller to consider the price of consuming heat, while taking actions. Energy 337 (2025) 138517 3 R. Mokhtari et al. Fig. 2. A randomly generated price profile with two probable future price trends. According to Fig. 2, if the controller is only aware of 𝑢𝑡, then it would take action solely based on the value of the real-time price. In this case, if the controller perceives 𝑢𝑡 as an expensive hour and decides to shut the heating system, and price trend 1 occurs, then it might be urged to turn on the heating due to comfort violation in the future, i.e., more expensive hours. This leads to suboptimal decisions. Hence, having knowledge about future prices would help the controller in making more informed decisions, leading to enhanced performance. For example, if price trend 2 is valid, then the controller might decide to turn off heating at 𝑡, and use the heating in the later cheaper hours. Considering a day-ahead market, prices are known for each hour of the next day. Therefore, these hourly prices can be directly incorporated into the controller, and would eliminate the need for a forecast model for price. 5.2. Reward function The DRL in this application is considered to primarily decrease the heating costs while keeping the indoor temperature within a thermally comfortable range. Hereby, the reward value (𝑟𝑡) at each time (𝑡) is calculated by: 𝑟𝑡=𝑟𝑇 𝐶 +𝛽.𝑟𝐻𝐶 (2) 𝑟𝑇 𝐶 =⎧ ⎪ ⎨ ⎪ ⎩ 𝑇𝑧𝑜𝑛𝑒 𝑡−𝑇𝑙𝑜𝑤 if 𝑇𝑧𝑜𝑛𝑒 𝑡< 𝑇 𝑙𝑜𝑤 𝑇𝑢𝑝 −𝑇𝑧𝑜𝑛𝑒 𝑡if 𝑇𝑧𝑜𝑛𝑒 𝑡> 𝑇 𝑢𝑝 0if 𝑇ℎ𝑖𝑔ℎ 𝑡≥𝑇𝑧𝑜𝑛𝑒 𝑡≥𝑇𝑙𝑜𝑤 (3) 𝑟𝐻𝐶 =𝑢𝑡.𝑃𝑡(4) In this equation, the total reward is calculated as the weighted sum of two reward terms of 𝑟𝑇 𝐶 and 𝑟𝐻𝐶 , which are reward values associated with thermal comfort and heating cost, respectively. Therefore, the positive actions are usually a trade-off between comfort and energy costs. The weighting factor 𝛽 determines how one reward type should be weighted over the other one. According to Eq. (2), a high 𝛽 value emphasizes the controller on reducing heat costs rather than improving thermal comfort, and vice versa. Therefore, the designed controller highly depends on 𝛽. 𝛽 is a user-specific parameter and can vary among different users and buildings. It is usually derived by trying random values and observing the controller’s behavior. We used the 𝛽 value of 0.1 in a similar way. Eq. (3) shows the thermal comfort reward function. The function returns a zero reward value for being within the pre-specified lower bound (𝑇𝑙𝑜𝑤) and upper bound (𝑇𝑢𝑝), and penalizes the zone temperatures out of the bounds. Eventually, Eq. (4) calculates the heating cost, by simply multiplying the heat price (𝑢𝑡) to the heat consumption (𝑃𝑡) at each timestep. 5.3. Temporally-weighted state difference Although future price data can enhance the performance of the controller, it will increase the state space size and controller complexity. This, in turn, can lead to longer training time and convergence issues. For example, including price data for the next 6 h would require adding six additional variables to the state. To resolve this shortcoming, we proposed a method called Temporal-weighted State Difference. TSD takes the weighted average of the difference of future states with the current state using a weighting function 𝛼. The weighting function is usually temporally descending, meaning that the future state changes are valued less than the immediate state changes. The concept is closely related to the discounted return calculation in DRL. The future state change 𝑠𝑇 𝑡 at each time 𝑡, considering a horizon of 𝑁 can be calculated as: 𝑠𝑇 𝑡= 𝑁 ∑ 𝑖=1 𝛼𝑖(𝛥𝑠𝑡+𝑖−𝑠𝑡)(5) where 𝑠𝑡 is the state value at each timestep. We apply the TSD method on the price state to include future price information into the controller. Accordingly, the TSD method reflects the upcoming price trajectories at each timestep and includes this information in the controller input states. A horizon of 6 h is considered for being long enough for the controller to respond in time. For the weighting function, a linear descending weighting function is considered, where recent prices have a higher impact than future price changes. The weights are calculated using: 𝛼𝑖=2(𝑁+1−𝑖) 𝑁(𝑁+ 1) (6) To illustrate the concept, TSD is applied to a randomly generated price profile 𝑢 for 24 h, and the future price change 𝑢𝑇 is calculated accordingly in Fig. 3. Fig. 3 shows the calculated 𝑢𝑇 for a horizon of 6 h, compared to 𝑢. 𝑢𝑇 shows the upcoming price at each hour of the day for the next 6 h. For example, by looking at 𝑢 between the hours 0 and 6, it can be realized that although the price is reducing in the beginning, the overall trend is increasing. 𝑢𝑡 at time 0 confirms this by a positive value. However, since the weighting factor is linearly decreasing, if a significant price change appears, then the 𝑢𝑡 will be heavily influenced by it. All in all, the figure shows that 𝑢𝑡 can complement the information carried by 𝑢, and can allow the controller to see the bigger picture with only a single state. 6. Case study Our experimental work is conducted within the ‘‘Urban Mining and Recycling (UMAR)’’ unit, a residential component of the NEST platform located in Dübendorf, Switzerland [25]. This vertically integrated district, operational since 2016, offers comprehensive sensor and actuator data such as temperature, heat pump power, and solar irradiance. UMAR is equipped with a high level of thermal insulation and large Energy 337 (2025) 138517 4 R. Mokhtari et al. Fig. 3. Hourly electricity price 𝑢 and calculated 𝑢𝑇 with a horizon of 6 h. Fig. 4. (a) Side view of UMAR unit in the NEST building. (b) Floor plan of the unit and the target room. windows, enabling a large solar gain. The unit is built mostly from lightweight recycled composites and reusable modules, and the heating and cooling are delivered by light aluminum ceiling panels, leading to low thermal inertia and making it highly reactive to heat inputs. The UMAR unit has two bedrooms, bathrooms, and a living room. The building and the UMAR unit are shown in Fig. 4. Each room is equipped with low-temperature heating systems, and for the experiment, one of the bedrooms is selected to control (i.e., R272). The room is heated by heating-cooling ceiling panels made of aluminum with copper pipes inside. The control variable for the heating is the valve position, which can be set to either 0 or 1. This method approximates the required heating power and ensures the system does not activate for very low values. The target room is occupied by a single occupant, and no information is available for the occupancy schedule. The temperature of the neighbor zone, i.e., the living room, is controlled by the default rule-based controller. Since the DRL is designed to have continuous actions for valve opening, but the heating valve accepts only boolean values, the duty cycle approach is taken to approximate the continuous valve position values. The method divides an hour into quarters, then uses the remaining time in each quarter to calculate a duty cycle for valve opening. 7. Results In this section, the settings for the training, simulations, and experiments are first explained. Afterwards, the corresponding findings are provided and discussed. 7.1. Setting Choosing the right learning parameters (so-called hyperparameters) is the most critical step of designing the DRL. These parameters should be selected based on the problem complexity and state space size. They are usually derived using a trial-and-error method in a simulation environment. The considered hyperparameters for the DRL training problem are shown in Table 1. Policy network size and Q-layer size show the architectural dimensions of the neural networks used for the policy and the value functions, respectively. The Policy network learning rate and Q-layer learning rate indicate how fast the agents learn to update their action strategy and evaluate actions. In the DDPG algorithm, the replay buffer stores experiences of state, action, reward, etc. The replay buffer capacity indicates the maximum number of transitions that the replay buffer can hold, and the initial value is the minimum number of experiences to collect before training begins. The Ornstein–Uhlenbeck noise process has a rate of convergence towards the average (𝜃) and a randomness scale (𝜎). 𝛾 is the discount factor, a parameter determining the importance of future rewards, which is a value between 0 and 1. Eventually, 𝛽 is the cost scale, introduced in Eq. (3), and determines the value of reward function terms, i.e., heating cost over thermal comfort. To analyze the influence of including future price in the controller states, three DRLs are considered and trained with different states, shown in Table 2. 𝐷𝑅𝐿_𝑢_𝑢𝑇 is the main controller that includes all the states described. 𝐷𝑅𝐿_𝑢 does not include the future price and only includes 𝑢𝑡 information from the price and takes action based on the current price. 𝐷𝑅𝐿_𝑏𝑎𝑠𝑖𝑐 is a controller that ignores both future and current price. The controllers are trained using the historical data logged by the local computer. The 𝑆𝑖𝑛(𝜋ℎ∕12+7𝜋∕6) and 𝐶𝑜𝑠(𝜋ℎ∕12+7𝜋∕6) provide the cyclical temporal patterns, where ℎ is the hour of the day. After analyzing the simulation results, one of the DRL models will be considered to conduct the experiment. The experiments take place, using a remote computer connected to the MQTT system in NEST. To ensure the connection stability, a toggle signal is set between the actuator and the broker every 5 s. In case of disconnection, the control is changed to the default rule-based controller. Energy 337 (2025) 138517 5 R. Mokhtari et al. Table 1 Hyperparameter values of the DRL. Parameter Value Policy network size 128 × 256 × 256 × 32 Policy learning rate 2 × 10−4 Q-layer size 128 × 256 × 256 × 32 Q-layer learning rate 1 × 10−4 Replay buffer capacity 5 × 104 Replay buffer initial capacity 5 × 103 𝜃 (convergence to mean) 0.2 𝜎 (randomness scale) 0.2 𝛾 (discount factor) 0.99 𝛽 (cost scale) 0.1053 Table 2 Input states of developed controllers. States 𝐷𝑅𝐿_𝑏𝑎𝑠𝑖𝑐 𝐷𝑅𝐿_𝑢 𝐷𝑅𝐿_𝑢_𝑢𝑇 𝑇𝑎𝑚𝑏 𝑡✓ ✓ ✓ 𝐺𝑠𝑜𝑙𝑎𝑟 𝑡✓ ✓ ✓ 𝑇𝑧𝑜𝑛𝑒 𝑡✓ ✓ ✓ 𝑆𝑖𝑛(𝜋ℎ∕12 + 7𝜋∕6) ✓ ✓ ✓ 𝐶𝑜𝑠(𝜋ℎ∕12 + 7𝜋∕6) ✓ ✓ ✓ 𝑢𝑡✓ ✓ 𝑢𝑇 𝑡✓ The experiments are designed to test the controller under three scenarios of without considering price, with random price, and with dynamic electricity price. For not considering the price, the price is set to an average value of 25 𝑐𝑡.𝐶𝐻𝐹 ∕kWh. For the random price case, a randomly generated price signal using Pseudo Random Binary Sequence (PRBS) [31]. The PRBS generates a deterministic price signal with white-noise properties, with random values and durations, suitable to test the controller under extreme dynamic conditions. Eventually, the electricity price data from the historical data is used to test the controller under more realistic conditions. The electricity price is taken from the tariffs charged by the local utility company in Dübendorf in 2024. The performance of the DRL controller is then compared with the default RBC. Since it is not possible to reconduct the experiment in a controlled environment with the same weather conditions, the RBC was implemented under weather conditions that are most similar to the experiment period. The DRL experiment lasted 66 h, therefore, same period was selected for RBC. Accordingly, February 8–10, 2025, is selected, where the RBC is taking the control task. The RBC is a simple bang–bang control, keeping the zone temperature within 21 ◦C–25 ◦C. The same price profile that was used in the experiment is used in this period to compare the achieved heating costs. The boxplots in Fig. 5 show the weather conditions in both periods. 7.2. Training Here, the result of training the controller is described. Since no ending criteria were considered, the training continued for 1800 iterations, as the maximum iteration number reached. The best cumulative reward of each iteration is analyzed, and the episode with the highest reward is selected. The building’s default controller, i.e., a simple rule-based controller, is used as the baseline for evaluating the performance of the DRL. It is important to note that in the training, the historical electricity price was not considered. The electricity price data has a recurring similar profile, including morning and evening peaks. Training the controller using this dataset would generate time-dependent policies to prevent heating in the morning and evenings. The aim of this study is to design a controller that can respond to any price profile, and not just a single profile, so random price profiles were used to train the controller. The training results can be found in Fig. 6. Fig. 6 shows the highest reward occurring in iteration 60. However, the trained agent performed poorly under unseen events, revealing that the agent is not trained enough and was still exploring random policies. Therefore, the agent in iteration with the second highest reward, i.e., iteration 1320, is selected. 7.3. Simulation The trained DRLs are simulated over a period of 1 week. The timeseries plot of three controllers in a single day are shown in Fig. 7. The temperature graph associated with 𝐷𝑅_𝑏𝑎𝑠𝑖𝑐 never falls out of the comfort bound, due to the more frequent valve opening throughout the day. Accordingly, the DRL opens the valves starting from 21:00 until 09:00, since the controller knows the temperature drop usually occurs at this period. From interaction with the environment and the training dataset, the controller has learned that heat demand is usually required between 20:00 and 08:00, therefore, it has learned the policy to open the valves within this period instead of waiting for the indoor temperature to exceed the lower temperature bound. This behavior might lead to higher thermal comfort, but since it neglects the price profile, the savings might not be much. The 𝐷𝑅𝐿_𝑢 controller has a different behavior, as it opens the valve in three periods. Comparing the valve opening times with the price profile shows that the controller tries to avoid expensive hours and opens the valve when it is relatively cheap. The last controller i.e. 𝐷𝑅𝐿_𝑢_𝑢𝑇 has quite similar behavior to 𝐷𝑅𝐿_𝑢 in most hours, with a difference that it does not open the valve in 00:00–03:00 period. Unlike 𝐷𝑅𝐿_𝑢, this controller understands that although price is relatively cheaper than the peak hours, there is no need to heat up the building since (1) the indoor temperature has not reached the lower bound yet, and (2) there is lower price approaching in a couple of hours. Although the indoor temperature ends up exceeding the lower thermal bound at 07:00, the controller has higher saving potential. To conduct a valid evaluation of the performance of the controllers, a one-month simulation is conducted and the performance of controllers are evaluated every 4 days. Results are shown in Fig. 8. The boxplots show the calculated daily heat cost for the trained controllers, together with the RBC for comparison, for one month. By looking at the medians of the boxplots, RBC stands higher than DRL controllers, clearly indicating higher daily heat costs. 𝐷𝑅𝐿_𝑏𝑎𝑠𝑖𝑐 performs better than RBC and has reduced the cost on average by 66.7%. This saving has been achieved by lowering the total heat demand and tracking the lower comfort bounds. 𝐷𝑅𝐿_𝑢 has a 44% lower heat cost on average than 𝐷𝑅𝐿_𝑏𝑎𝑠𝑖𝑐 since it receives current prices and can act accordingly. Eventually, 𝐷𝑅𝐿_𝑢_𝑢𝑇 has the lowest heat cost among the other controllers. According to the overall results, 𝐷𝑅𝐿_𝑢_𝑢𝑇 has an 11.5% lower heat cost than 𝐷𝑅𝐿_𝑢 as a result of including future price in states. To ensure that 𝐷𝑅𝐿_𝑢_𝑢𝑇 is truly performing better than 𝐷𝑅𝐿_𝑢, a t-test analysis for the two independent datasets is conducted, checking whether the means of the two groups are significantly different. The result showed a 𝑝-value of 2.7 × 10−16, indicating that 𝐷𝑅𝐿_𝑢_𝑢𝑇 is performing statistically better. Therefore, 𝐷𝑅𝐿_𝑢_𝑢𝑇 controller would be used in the experiment. 7.4. Real-world experiment From the simulations, it was seen that 𝐷𝑅𝐿_𝑢_𝑢𝑇 can provide higher savings. Here, we use 𝐷𝑅𝐿_𝑢_𝑢𝑇 to conduct the experiments, and we call it 𝐷𝑅𝐿 from now on. To conduct a comprehensive experiment, the controller is tested under different price conditions in different scenarios. 7.5. Scenario A: Without considering price First, the controller is tested without considering price in a heating period. This would allow us to evaluate the controller’s reaction to other states. The findings related to not considering price are illustrated in Fig. 9. According to the bottom figure, ambient temperature Energy 337 (2025) 138517 6 R. Mokhtari et al. Fig. 5. Comparison of weather conditions in two periods for RBC and DRL operation. The left two boxplots show the ambient temperature distribution, and the right two boxplots show solar irradiation distributions throughout the periods. Fig. 6. DRL training results (𝐷𝑅𝐿_𝑢_𝑢𝑇), showing the cumulative reward at each iteration. RBC is used as a baseline controller for comparison. ranges between 2 ◦C and 10 ◦C and the solar radiation reaches 500 W∕m2, corresponding to a mild winter period. The price is fixed at 25 𝑐𝑡.𝐶𝐻𝐹 ∕kWh throughout the experiment. The indoor temperature starts from higher than the upper bound, and the controller remains off until March 14, 7:00 where a window opening causes a sudden temperature drop and the controller opens the valve until the indoor temperature surpasses the lower band. It can be seen that in the whole experiment period, the controller opens the valves mainly when the indoor temperature reaches the lower bound. This means that when no price change is expected, the controller finds no reason to heat the building, unless comfort is in danger. This is the optimal behavior, leading to the lowest heat cost, while maintaining indoor thermal comfort. Another study of using a price-ignorant DRL in the test building showed the same behavior [32]. 7.6. Scenario B: Random price To test the controller’s reaction to a highly dynamic price, first, a randomly generated PRBS signal is used as the price signal. The controller is expected to control the heating by taking into account the current and future prices. The findings can be seen in Fig. 10. According to Fig. 10, the controller starts with heating the building hours before the price peaks, and delays the heat demand until the next cheap hour, i.e. Feb 27th, 01:40. The temperature pumps again until 06:30, where the indoor temperature falls slightly below the lower bound. However, despite being cheap, the controller is opening the valve. One possible explanation for this behavior is that the controller expects a sunrise and heat gain from the sun, deciding to sacrifice some comfort. There is only a tiny heat input occurring at 07:20, right before the price increases. This leads to an indoor temperature rise, right on the lower bound. The heat price profile and valve opening curves together indicate that the heating is mostly occurring right before prices start to peak. However, on Feb 27th, 22:30, the heating continues in the expensive hour as well, which is due to the unexpected window opening, leading to temperature drop and unexpected behavior. However, in this period, since the window is open, the heating is forcefully turned off to save energy. 7.7. Scenario C: Electricity price To test the controller in a more realistic condition, the electricity price from the historical data is used as the price signal. The findings can be seen in Fig. 11. According to the bottom graph that shows the weather conditions during the experiment, the ambient temperature ranges from −3 ◦C to 15 ◦C, and the solar irradiation is the highest on the 3rd day due to a clear sky. The top graph shows that the indoor temperature is held around the lower bound most of the time, allowing for a low heat cost while maintaining comfort. In some hours, however, a small deviation from the lower bound comfort can be recognized. Only on the last day did the indoor temperature exceed the upper comfort bound, i.e., due to high solar radiation and ambient temperature. In the middle graph, the dynamic electricity prices and the heating system state are shown. The electricity price shows a typical recurring pattern with expensive morning (06:00–09:00) and evening hours (18:00–21:00), followed by cheap afternoon and night. Comparing the heating system function with the price profile shows that the controller tries to avoid expensive hours. Only on the last day (i.e., March 3rd) at 08:00 the heating system is occasionally turning on during the peak hour. This occurs because the ambient temperature is the lowest value at this hour, reaching −2.5 ◦C, and the controller has learned to prioritize comfort over cost in this situation. Since the controller does not observe weather predictions, it does not expect a sudden increase in temperature and solar irradiation. During cheap hours, however, the controller does not necessarily decide to turn the heating system on. For example, at 16:00 on March 2, the price is relatively low, but the heating system is not turning on. This behavior is the result of the training process, when the controller has learned that Energy 337 (2025) 138517 7 R. Mokhtari et al. Fig. 7. The simulation result time series of three controllers in Jan 6, 2025. Fig. 8. Heat cost saving of trained control algorithms and baseline RBC in simulation during a month (December 13th 2023–January 13th 2024.). maximizing heating during cheap hours before expensive hours does not necessarily lead to a positive reward, and may lead to unnecessary heating. However, this does not limit the demand response and loadshifting potential of the controller. Between 12:00–18:00 March 2, where the price is constant, the controller observes an increasing price trend and decides to turn on heating partially to store heat enough to postpone the heating demand right to the next cheap hour (i.e., 21:00 March 2). The indoor temperature reaches the lower comfort bound at 21:00, which is the next cheapest hour. Such control actions showcase the load-shifting ability of the controller. Eventually, since the controller receives indoor temperature as a state, it decides to turn off the heating from 10:00 on March 3 to the end of the day due to the high indoor temperature, no matter what other states are. 7.8. Comparison with RBC To evaluate the controller behavior, it is compared with an RBC. The results for the RBC, given the dynamic electricity price are shown in Fig. 12. The RBC clearly does not respond to the price signal, and keeps the indoor temperature between 21 ◦C and 25 ◦C. This leads to long heating periods but also cools more slowly. To assess the comfort violation, it is defined as the product of the deviation of indoor temperature beyond comfort bounds (K) and its duration (h). The comparison of the, e.g. total heating cost and comfort violation by both controllers are depicted in Fig. 13. DRL could reduce the heating costs by 79% compared to the RBC. However, this comes with the cost of 7 Kh comfort violation, i.e., exceeding the comfort bounds by 0.32 ◦C on average. The trade-off between heating cost and thermal comfort is highly dependent on 𝛽 (reward weight) in the design of the DRL. Meteorological variables (i.e., ambient temperature and solar irradiation) have the highest impact on the heating demand and the heating cost of the buildings. The correlation between these states is plotted in Fig. 14. The heating cost of RBC is higher in both figures than the DRL controller. The gap is higher in lower ambient temperature and solar irradiation. This denotes that for the same weather situation, DRL managed to reduce heat costs compared to the DRL. 8. Discussion This study tested a DRL in thermal control of a test-bed building and showed promising results. The problem of controlling an indoor thermostat is analogous to pole control, which is a classic DRL problem in which a DRL agent tries to keep the pole upright within a safe angle range by moving to the left or right; except that the building’s temperature is more stable due to thermal inertia. Therefore, DRL is a suitable method to control the indoor thermostat. When trained correctly, the agent learns that by preheating during hours before prices get expensive and reducing heating during expensive periods, it achieves significant cost reductions while maintaining comfort. Additionally, the underlying reward function is relatively simple with two Energy 337 (2025) 138517 8 R. Mokhtari et al. Fig. 9. Experiment results of deployed DRL for four consecutive days without considering price. Top: indoor temperature, middle: price signal, and the valve position, bottom: ambient temperature and solar irradiation, captured by the local weather station. Fig. 10. Experiment results of deployed DRL for three consecutive days with random price. terms of comfort and heat cost. However, DRL might not be the best option for problems with a volatile and unpredictable environment, which has too many state variables. Although DRL has been shown to be effective in controlling indoor temperature, it comes with some limitations and challenges. DRL requires an environment to interact and learn from, which is normally a detailed building model, hindering its scalability and wide application. This can, however, be solved by transfer learning, an active area of research where one trained controller can be used in another system with a short fine-tuning period [32]. Another issue is with the training process, which is usually heavy in computation. In systems with a high number of state variables, the total state situations to explore by the agent grows rapidly, and the training might undergo unexplored situations, leading to suboptimal control. Furthermore, there is no guarantee of the control performance of the trained DRL. This is because the decisions DRL makes are the result of learned policies through trialand-error, and there is no guarantee that the controller is controlling optimally or sub-optimally. The only way to answer this is to compare it with other controllers. Therefore, in practice, it is recommended to use Rule-guided RL, where the RL or DRL is constrained with an RBC to make sure extreme actions do not occur. We chose a DDPG agent for the DRL controller, which is a widely used algorithm for continuous action spaces. While DDPG favors being model-free and stable, and is extensively studied, it is also known to be sample inefficient, highly sensitive to hyperparameters, and to pose exploration challenges. Newer extensions of DDPG are TD3 and SoftActor–Critic algorithms, which tend to resolve previous issues. For the communication, we used MQTT protocol to communicate between the sensors and the controller. While MQTT is widely used in test facilities and living labs, modern buildings usually have a Building Management System (BMS), which does not support two-way communication, hindering the application of advanced controllers. The DRL was trained and applied to a single heating unit and thermostat. The current approach could be extended by using multiagent reinforcement learning, where each thermostat is an individual Energy 337 (2025) 138517 9