Multi-Agent Reinforcement Learning Framework

This section presents the MARL architecture for optimizing energy transition in Turkey. The system utilizes Multi-Agent Proximal Policy Optimization which has been used for benchmarking against a total of 12 more algorithms. The main algorithms include value-based (DQN, A3C), policy gradient (SAC, TD3), evolutionary (GA, PSO, DE), fuzzy (Fuzzy Q-Learning, Neuro-Fuzzy), multi-agent (MARL), and meta-learning (MAML, Progressive NN) approaches. Section 3.5.1 describes the MAPPO algorithm and actor-critic architecture; Section 3.5.2 describes multi-agent coordination with a total of 7 regional agents; Section 3.5.3 specifies benchmark algorithms; Section 3.5.4 documents hyperparameter configurations.

Multi-Agent Proximal Policy Optimization (MAPPO): Primary Algorithm

MAPPO is special because it tells us how to fix a policy that has just made a big mistake. MAPPO [schulman2017proximal] achieves a balance between sample efficiency (by enabling multiple epochs per batch through importance sampling), training stability (through a clipped objective that guards against destructive updates) as well as implementation simplicity (with only a few hyperparameters, and being robust to misspecification). This balance motivates many researchers and practitioners to adopt MAPPO as their primary algorithm.

Policy gradient foundation: The parameterized policy $\pi (a|s;\theta )$ is optimized to maximize expected return:

$$J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\gamma }^{t}R(s_t,a_t)\right]$$

The policy gradient theorem [sutton2000policy] offers a way to calculate the optimal policies:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log \pi (a_t|s_t;\theta ){\hat{A}}_t\right]$$

${\hat{A}}_t=Q(s_t,a_t)-V(s_t)$ refers to the advantage. The advantage estimates generated using a critic network $V(s;\varphi )$ reduce variance while maintaining unbiasedness.

Clipped surrogate objective: Updates on policy gradients are like standard policy gradients but for large $\alpha$ it breaks. MAPPO clips probability ratio:

$$r_t(\theta )=\frac{\pi (a_t|s_t;\theta )}{\pi (a_t|s_t;{\theta }_{old})}$$

Clipped objective:

$$L^{\mathbf{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \left(r_t(\theta ){\hat{A}}_t,\mathbf{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right]$$

Where epsilon equals 0.2, which is a typical value. Clipping restricts $r_t$ to the range of [0.8, 1.2], limiting policy changes to 20% per update. If the advantage function is positive (${\hat{A}}_t>0$), then the objective increases the probability $\pi (a_t|s_t)$ but with 1.2× clipping. If the advantage function is negative, then the objective decreases this probability but with 0.8× clipping. This conservative updating maintains training stability.

Actor-critic architecture: MAPPO uses two neural networks, actor $\pi (a|s;\theta )$ that returns action distribution and critic $V(s;\varphi )$ that estimates state value. The two networks share common layers that extract features, like convolutional or fully connected processing state. Then their paths diverge to separate heads. Whose parameter count is 40% less than if they were independent networks.

Actor network (policy): A 446-D state is run through the actor network (policy):

1. The input normalization process is represented by the formula $s_{norm}=(s-{\mu }_s)/{\sigma }_s$.
2. The first fully connected layer consists of 512 neurons and ReLU activation.
3. A layer with 512 neurons activated through ReLU.
4. The average head has 119 neurons (one for each action dimension) that are not activated (unbounded).
5. The log-std head with 119 neurons has a starting value of -0.5 (std ≈ 0.6), learned during training.
6. Generating Action: Sample an action from a normal distribution: $a\sim \mathcal{N}({\mu }_{\theta }(s),\exp (2\ast log-st{d}_{\theta }(s)))$.
7. The maximum action is multiplied by the effect of squashing function: $a_{bounded}=a_{\max }\cdot \sigma (a)$ where $\sigma$ is sigmoid.

The Gaussian policy allows for continuous action spaces which will allow capacity additions of 0-15 GW/year. The standard deviation is a learnable parameter of the Gaussian policy, which allows one to balance exploration (high std in early training) and exploitation (low std in late training).

Critic network (value function): The critic network or value function receives a 446-dimensional state as input similar to the actor:

1. Shared feature extraction (layers 1-2 identical to actor).
2. Head value: 1 neuron no activation (no upper bound value estimate).
3. The output from the 3rd step is: $V(s;\varphi )\in \mathbb{R}$, estimated cumulative discounted reward from state $s$.

Critic training minimizes mean squared error:

$$L^V(\varphi )={\mathbb{E}}_t\left[(V(s_t;\varphi )-V_t^{\mathbf{target}}{)}^2\right]$$

The Monte Carlo return, which is the sum of the observed rewards $V_t^{target}={\sum }_{k=0}^{T-t}{\gamma }^k{r}_{t+k}$ or Generalized Advantage Estimation (GAE, λ-return; Schulman et al., 2016) for lower variance.

Generalized Advantage Estimation (GAE): There are three different techniques that are used to estimate the advantage function, they include the following:

- Monte Carlo: Estimate unbiased, high variance (${\hat{A}}_t={\sum }_{k=0}^{T-t}{\gamma }^k{r}_{t+k}-V(s_t)$) - TD(0): Estimate biased, low variance (${\hat{A}}_t=r_t+\gamma V(s_{t+1})-V(s_t)$) - Generalized Advantage Estimation (GAE): The results comes about through combinations of those two equations above. GAE interpolates via λ-weighting:

$${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)(TDerror){\hat{A}}_t^{\mathbf{GAE}(\gamma ,\lambda )}=\sum _{k=0}^{T-t}(\gamma \lambda {)}^k{\delta }_{t+k}$$