Reinforcement Learning: Foundations, Algorithms, and Energy Applications

Reinforcement learning is a framework concerned with learning what is the 'best' policy to take actions through interactions with environments. It offers a powerful framework for sequential optimization problems suited to the high-dimensional environment that is common in energy transition planning. Unlike supervised learning in which labeled examples of the optimal decision are required, agents in RL learn with trial-and-error through reward signals that indicate the quality of the decision. This allows agents to iteratively improve their policy or strategy to maximize the cumulative long-term reward. The present paradigm can easily integrate the temporal, stochastic as well as high dimensional nature of energy system optimization where today's decisions regarding capacity expansion affect tomorrow's costs, emissions, reliability, etc. over multi-decade horizons. This part discusses basic concepts of RL (Markov Decision Processes and Bellman equations), algorithm families (value based, policy gradient, evolutionary and hybrid), and uses in energy systems in order to establish a theoretical backbone of the multi-agent RL optimization framework that this dissertation proposes in Chapter 3.

Markov Decision Processes and Bellman Optimality

Reinforcement learning problems are modeled as Markov Decision Processes (MDPs), denoted as a tuple $(\mathcal{S},\mathcal{A},P,R,\gamma )$:

- $\mathcal{S}$ refers to the state space of the system, which consists of all possible configurations of the system. - $\mathcal{A}$ refers to action space, which denotes the available action or decision at any states. - $P(s'|s,a)$ is **stochastic transition probability** (likelihood of reaching state $s'$ from state $s$ after action $a$). The probability of reaching state $s'$ from state $s$ after performing action $a$ is a form of a stochastic model. The stochastic transition probability $P(s'|s,a)$ models all the uncertainty that one may face in the future. Examples include uncertainty regarding demand growth, technology costs, potential changes in public policy, climatic variations, etc. The stochastic characteristic captures the inherent uncertainties of energy system dynamics, giving MDPs detachment from deterministic optimization problems. - $R(s,a,s')$ is reward function (immediate payoff for transition) - $\gamma \in [0,1]$ is discount factor (present value of future rewards). The discount factor is represented by $\gamma$ and its value lies within the range of 0 to 1.

The Markov property states that the distribution of the next state, $s_{t+1}$, is independent of the history of previous states and actions: $P(s_{t+1}|s_t,a_t,s_{t-1},\dots ,s_0)=P(s_{t+1}|s_t,a_t)$. The simplifying nature of the memoryless assumption does not come without a cost. Hence the state which is used should embody all information necessary for the decision to be optimal.

A policy $\pi :\mathcal{S}\to \mathcal{A}$ (deterministic) or $\pi (a|s)$ (stochastic) maps states to actions. Finding a near-optimal policy ${\pi }^{\ast }$ for maximizing the expected cumulative discounted reward (i.e. return):

$$J(\pi )={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t,s_{t+1})\right]$$

$V^{\pi }(s)$ value function gives expected return starting at state $s$ under policy $\pi$:

$$V^{\pi }(s)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t,s_{t+1})\mid s_0=s\right]$$

The action-value function, also known as the Q-function, denoted as $Q^{\pi }(s,a)$, computes the expected return starting at state $s$, taking action $a$, then following policy $\pi$:

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t,s_{t+1})\mid s_0=s,a_0=a\right]$$

The Bellman equations refer to recursive relationships for value functions. For $V^{\pi }$:

$$V^{\pi }(s)=\sum _a\pi (a|s)\sum _{s^{\prime }}P(s^{\prime }|s,a)\left[R(s,a,s^{\prime })+\gamma V^{\pi }(s^{\prime })\right]$$

For $Q^{\pi }$:

$$Q^{\pi }(s,a)=\sum _{s^{\prime }}P(s^{\prime }|s,a)\left[R(s,a,s^{\prime })+\gamma \sum _{a^{\prime }}\pi (a^{\prime }|s^{\prime })Q^{\pi }(s^{\prime },a^{\prime })\right]$$

The Bellman optimality equation is satisfied by the optimal value function $V^{\ast }(s)={\max }_{\pi }{V}^{\pi }(s)$:

$$V^{\ast }(s)=\underset{a}{\max }\sum _{s^{\prime }}P(s^{\prime }|s,a)\left[R(s,a,s^{\prime })+\gamma V^{\ast }(s^{\prime })\right]$$

And optimal Q-function:

$$Q^{\ast }(s,a)=\sum _{s^{\prime }}P(s^{\prime }|s,a)\left[R(s,a,s^{\prime })+\gamma \underset{a^{\prime }}{\max }{Q}^{\ast }(s^{\prime },a^{\prime })\right]$$

The best policy is ${\pi }^{\ast }(s)=\arg {\max }_a{Q}^{\ast }(s,a)$. All RL algorithms use these equations to approximate their optimal value function or policy through updates.

@sutton2018reinforcement provide comprehensive treatment of RL fundamentals. For the application of energy models, the MDP formulation captures sequential capacity expansion naturally. Here, the definition of components is mentioned. The state $s_t$ represents the current capacity mix, demand, costs, emissions. Action $a_t$ represents the capacity additions. Further, the transition $P(s_{t+1}|s_t,a_t)$ reflects capacity accumulation, cost evolution, demand growth. The reward $R_t$ aggregates cost minimization, emission reduction and reliability maintenance. The discount factor $\gamma$ balances short-term and long-term goals. If $\gamma$ is around zero, it means that immediate rewards are a priority. Otherwise, if $\gamma$ is close to 1, it means that the distant future is equally weighted as the present.

Value-Based Methods: Q-Learning and Deep Q-Networks

Value-based methods learn optimal value functions that use greedy action selection to derive optimal policies. The foundational algorithm for reinforcement learning is Q-learning, a Q-value updating procedure based on the Bellman equation:

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[R_{t+1}+\gamma \underset{a^{\prime }}{\max }Q(s_{t+1},a^{\prime })-Q(s_t,a_t)\right]$$

where $\alpha$ is learning rate. Q-learning is an off-policy algorithm that learns a near-optimal policy while executing some exploratory policy like an epsilon-greedy policy. Under tabular representation (finite $\mathcal{S},\mathcal{A}$) with sufficient exploration (an assumption), learning rates must decrease [watkins1992qlearning] and convergence to $Q^{\ast }$ is guaranteed.