System Organisation and Design Principles

The architecture of the hybrid AI system is built on four design principles which guide the choice of components, specification of interfaces and integration strategy. The governing principles-modularity, scalability, interpretability, and reproducibility of the developed system will ensure that this system will address the identified research gaps (Chapter 2, Section 2.6) scientifically meaningful and practically useful.

Summary of core architecture

[Figure]

*System architecture diagram: Three interconnected subsystems (Forecasting, RL Optimization, Integration) exchanging information through well-defined interfaces. Forecasting subsystem generates probabilistic capacity projections, RL Optimization formulates capacity expansion as MDP, Integration subsystem mediates bidirectional information exchange enabling iterative co-evolution.*

The communication between the three subsystems, as depicted in Figure is through a set of interfaces. Upon initialization (or extrapolation), the forecasting subsystem fed with 2000-2024 historic data generates a first-level forecast of capacity of the horizon 2025-2050 taken as ${\hat{K}}_t^{(0)}$; supply subsystem with forecast demand ${\hat{q}}_t^{(0)}$ as ${\hat{D}}_t^{(i)}$ from which the emergence of first-level market price ${\hat{p}}_t^{(0)}$ is computed. The forecast outputs are converted to RL outputs through a forward pass application. The RL environment is initialized by mapping the (mean capacity projections ${\mu }_t$, uncertainty distributions ${\sigma }_t$, and the transition probabilities $P(s_{t+1}|s_t,a_t)$) previously obtained to the RL environment state space and transition dynamics. The forecast mean ${\mu }_t$ is taken as the initial state value in particular. The uncertainty bands, ${\mu }_t-2{\sigma }_t,{\mu }_t+2{\sigma }_t$ set the feasible state range that can restrict the reinforcement learning exploration. During the RL optimization phase, the multi-agent RL system explores the action space with a focus on the 26 years (2025-2050) long horizon to derive near-optimal capacity expansion trajectories Kt* that maximises cumulative rewards. The forecast model is improved by online learning and the set of RL-discovered trajectories Kt* t=1:26 is treated as an extension of the dataset for forecast training. The forecast model is retrained on a combined data (historical + RL trajectories), so it gives us a better prediction of the RL-optimal pathways. The process will repeat the steps 2-4 until the forecast-RL alignment is reached or ||K^((k+1))_(t) − K*^((k+1))_(t)||2 < ϵ, where ϵ = 1 GW. Generally comes together within four iterations (in a range of 3-5, worst-case 7). Capacity projections were transferred to RL with approximately 4732 values for 26 technologies × 7 regions × 26 years, along with variance matrices and state-action-next state triplets for the uncertainty distributions and transition probabilities respectively. RL – Forecasting instrumental tool optimal capacity trajectories –4,732 values; action sequences 119 dimensions × 26 years –3,094 values; reward signals –26 values per trajectory. The integration results in 90.2% reduction in forecast-policy gap (final achievement) from 8.2 GW to 0.8 GW at final milestone.

[Figure]

*Forecast-RL integration diagram: Bidirectional information flow between forecasting subsystem and RL optimization subsystem. Forward direction: forecasts initialize RL environment. Backward direction: RL-optimal trajectories refine forecast training. Iterative co-evolution converges in 3-5 iterations, reducing forecast-policy gap from 8 GW to 1 GW.*

Figure illustrates the bidirectional link facilitating iterative improvability between the forecasting subsystem and the optimization subsystem.

[Figure]

*Methodology flowchart: Complete workflow from data acquisition through forecasting, RL optimization, integration, and validation. Process includes data preprocessing, model training, policy optimization, iterative refinement, and statistical validation.*

The full methodology processes are in Figure

The Forecasting Subsystem produces probabilistic capacity projections for 26 technologies in 7 regions over 26 years (2025-2050) using historical data (capacity, generation, costs, demands) and exogenous variables (like GDP, population, technology learning curves). Physics-informed neural networks (HNSM, ENSM) utilize specialized knowledge (S-curve diffusion characterization, energy balance validation, capacity factor bounds) to ensure that the results remain physically plausible. Monte Carlo dropout generating 1000 samples of each prediction along with uncertainty for mean, variance, 5th percentile, and 95th percentile. The initial state distributions feed the RL environment as well as the transition probabilities.

The capacity expansion issue is represented as a Markov Decision Process by the RL optimization subsystem. The state space comprises 446 dimensions in its formulation (which includes capacities, costs, demands, emissions, year indices and forecast projections), while the action space is a 119-dimensional continuous one (which includes capacity additions by technology-region pairs plus storage investments).

Formal Definition of MDP:

The energy transition optimization issue is formalized by the MDP (Markov Decision Process) defined by the tuple:

$$M=(S,A,P,R,\gamma )$$

where:

- **$S$ (State Space):** The state space is continuous, defined by the agent in all of its dimensions:

| Component | Dimensions | Description | | --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Technology capacities | 182 | $C_t^{tech,region}\in {\mathbb{R}}^{26\ast 7}$ for 26 technologies across 7 regions | | Forecast projections | 182 | ${\hat{K}}_t\in {\mathbb{R}}^{26\ast 7}$ from forecasting subsystem (ENSM) | | Exogenous variables | 34 | ${\mathbf{x}}_t\in {\mathbb{R}}^{34}$: GDP (7 regions), population (7), technology learning rates (20) | | Cost parameters | 26 | ${\mathbf{c}}_t\in {\mathbb{R}}^{26}$: technology-specific CAPEX, OPEX, fuel costs | | Policy indicators | 10 | ${\mathbf{p}}_t\in {\mathbb{R}}^{10}$: carbon price, renewable targets, subsidies, etc. | | Demand forecasts | 7 | $D_t^{region}\in {\mathbb{R}}^7$: regional electricity demand projections | | System metrics | 2 | $R_t,U_t\in {\mathbb{R}}^2$: reliability index, utilization factor | | Temporal indices | 2 | $t,year\in {\mathbb{R}}^2$: timestep (0-26), calendar year (2024–2050) | | Emission levels | 1 | $E_t\in \mathbb{R}$: cumulative CO₂ emissions | | **Total** | **446** | State vector:${\mathbf{s}}_t=[C_t^{tech,region},{\hat{K}}_t,{\mathbf{x}}_t,{\mathbf{c}}_t,{\mathbf{p}}_t,D_t^{region},R_t,U_t,t,year,E_t]\in {\mathbb{R}}^{446}$ |

: State Space Dimension Breakdown (446 dimensions total) {#tbl:state-space-breakdown} - **$A$ (Action Space):** The action space $A$, consists of continuous values, has dimension 119, and lives in R119, which captures both storage investments GWh/year and capacity additions GW/year. The action space is structured hierarchically across six decision levels:

| Level | Dimensions | Description | Hierarchical Structure | | ------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | Level 1: Technology investments | 91 | $\Delta C_t^{tech,region}\in {\mathbb{R}}^{91}$: capacity additions by technology-region pairs (26 technologies × 7 regions - overlaps) | Sector allocation → Technology choice | | Level 2: Storage investments | 28 | $\Delta S_t^{region}\in {\mathbb{R}}^{28}$: storage capacity by region-type (7 regions × 4 storage types: lithium-ion, pumped hydro, hydrogen, thermal) | Regional adjustments → Storage type | | **Total** | **119** | Action vector:${\mathbf{a}}_t=[\Delta C_t^{tech,region},\Delta S_t^{region}]\in {\mathbb{R}}^{119}$ | Hierarchical: technology → region → capacity → timing |

: Action Space Dimension Breakdown (119 dimensions total) {#tbl:action-space-breakdown}

The structure allows: Level 1 (sector allocation) that is selection of energy source categories (renewable, storage, nuclear, fossil); Level 2 (technology-level) where specific technology selection (solar PV, onshore wind etc.); and Level 3 (regional adjustment) where allocation to 7 regions based on quality of resources, geographical and transmission limitations. - **$P$ (Transition Probability Function):** Stochastic transition function $P:S\ast A\ast S\to [0,1]$ modeling the evolution of states:

$$P(s_{t+1}|s_t,a_t)=P(C_{t+1},D_{t+1},E_{t+1},\dots |C_t,D_t,E_t,a_t)$$

The transition dynamics are stochastic because a forecasting subsystem is uncertain in its forecasts. Specifically: