Turkey's energy transition is formalized as a Markov Decision Process in the RL environment by defining the representation of the states, action spaces, transition dynamics, and reward structure. We balance realism—which captures the essential system dynamics—with tractability for computational efficiency, allowing training for over 5000 episodes. This part will detail the state space that is 446-dimensional, the continuous action space that is 119-dimensional, the physics-based transition dynamics and the methods employed to enforce the constraints.
*State and action space visualization: Breakdown of 446-dimensional state space (Global State: 26D, Tech-Region State: 182D, Regional State: 112D, Exogenous State: 126D) and 119-dimensional action space (Technology Investments: 91D, Storage Investments: 28D). Pie charts show proportional distribution of dimensions across categories, enabling intuitive understanding of state-action space complexity in the RL environment.*
State Space Representation (446 Dimensions)
Everything we need to know about the system to make near-optimal capacity expansion decisions at time $t$ (year 2024+t, t=0 corresponding to 2025) is encoded in the state space. A hierarchical partitioning of the state vector :
Formal State Vector Definition:
The state vector is formally defined as:
where:
- ****: The solar capacity vector, which represents the utility PV, commercial PV, residential PV, and CSP, across 7 regions (28 dimensions). - ****: The wind capacity vector refers to the vector of wind capacity (onshore, offshore, distributed) across the 7 regions (18 dimensions, with offshore limited to the 4 coastal regions). - ****: The hydro capacity vector which includes run-of-river, reservoir, and pumped storage across 7 regions which has 19 dimensions. - ****: Nuclear capacity scalar (1 dimension, national total). - ****: The vector contains capacity information for geothermal energy across 7 regions (7 dimensions). - ****: The vector representing biomass capacity across 7 regions has 7 dimensions. - ****: The fossil capacity vector, that is, gas combined cycle gas turbine (CCGT), gas open cycle gas turbine (OCGT), coal lignite, coal hard, oil across 7 regions (21 dimensions). - ****: The regional electric demand (TWh/year) vector across 7 regions (7 dimensions). - ****: The carbon price, measured at € per ton CO₂, is a numeric scalar (1 dimension). - ****: Technology cost vector (CAPEX, €/kW) for 26 technologies (26 dimensions). - ****: Cumulative CO₂ emissions scalar (MtCO₂, 1 dimension). - ****: Policy parameter vector made up of renewable targets, subsidies, carbon tax and others in 10-dimensions. - ****: The forecast subsystem computes a matrix of forecasts (26 technologies × 7 regions, 182 dimensions) from forecasting subsystem. - ****: Timestep index (0-25, 1 dimension). - ****: Calendar year (2025-2050, 1 dimension). - ****: The System Reliability Index is a real number (0-1, 1 dimension). - ****: The system utilization index is represented by (capacity factor, 0-1, 1 dimension). - ****: The vector of exogenous variables (GDP, population, technology learning rates, etc., 34 dimensions).
State Space Properties:
- **Type:** The system contains continuously state-space – 446 i.e. , and all components are real-valued. - **Normalization:** Normalization of all state components to $[0, 1]$ or $[-1, 1]$ range is carried out to ensure that different scales will not affect the neural network training. - **Temporal Dependence:** The temporal dependence defines how the components of the state evolve through time. These can be specified as transition dynamics that should take into account forecast uncertainty and stochastic perturbations.
Global state (26 dimensions): Country totals (26 dimensions): The high-level context of nations:
- Total installed capacity (1): (GW). - Renewable share (1): Share of renewables is the summed capacity of the respective renewable technology (t) divided by that of all technology (t): . - Cumulative cost (1): Cost at time t: (billion $). - Cumulative emissions (1): Total emissions:$\sum_{\tau=0}^{t} Emissions_\tau$(MtCO₂). - Year index (1):$t \in [0, 25]$(timestep within 26-year horizon). - Time-to-target (1): In year t, the target time is$25 - t$(years remaining until 2050). - 2050 target progress indicators (4): Indicators measuring progress towards reaching the goal set for the year 2050: (i) Share of Renewables Target (90%), (ii) Emission Reduction Target (87%), (iii) Total Capacity Target (218 GW), (iv) Percentage Progress. - Demand met indicator (1): An indicator of demand satisfaction should be$Generation_t / Demand_t$(reliability metric, should be ≥1). - National capacity factor (1): The national capacity factor is measured as$Generation_t / (Capacity_t * 8760/1000)$where Generation is in TWh and Capacity is in GW. - Technology diversity index (1): The technology diversity index is calculated as follows: Shannon entropy$-\sum_i p_i \log p_i$where$p_i = K_i / \sum K$. - Financial health (1): Cost per GW installed, indicating capital efficiency. - Emissions intensity (1): The measurement for this indicator is emissions intensity measured as emissions per TWh generated, expressed in kg CO₂/MWh. - Reserve margin (1): The reserve margin is equal to (Capacity - Peak Demand) / Peak Demand. It is a measure of adequacy of the grid. - Storage-to-renewable ratio (1): The term "storage-to-renewable ratio" refers to a flexible indicator: Storage capacity / Renewable capacity. - Nuclear status (1): Binary indicator whether Akkuyu was commissioned or not. - Fossil phase-out progress (1): The share of fossil capacity that is retired. - Policy regime (5): The policy regime with one-hot encoding has five alternatives such as baseline, intermediate, accelerated, high-subsidy scenario, low-subsidy scenario.
Technology-region state (182 dimensions): The installed capacity across 26 technologies and 7 regions:
- 26 technologies: The IEA has classified 26 technologies that are currently effective, which include: (i) solar PV; (ii) onshore wind; (iii) offshore wind; (iv) hydro (run-of-river); (v) hydro (reservoir); (vi) pumped hydro storage; (vii) geothermal; (viii) biomass; (ix) natural gas (CCGT); (x) natural gas (OCGT); (xi) coal (lignite); (xii) coal (hard coal); (xiii) oil; (xiv) nuclear; (xv) battery storage (4-hour); (xvi) battery storage (8-hour); (xvii) hydrogen storage; (xviii) compressed air storage; (xix) thermal storage; (xx) demand response capacity; (xxi) rooftop solar; (xxii) concentrated solar power (CSP); (xxiii) waste-to-energy; (xxiv) tidal; (xxv) wave; (xxvi) small modular reactors (SMR). - 7 regions: Turkey, being a geographically diverse and rich country, offers 7 regions: Marmara, Aegean, Mediterranean, Central Anatolia, Black Sea, Eastern Anatolia, Southeast Anatolia. - Not all combinations of technology and region appear valid (for instance, offshore wind being limited to the regions of Marmara, Aegean, Mediterranean, Black Sea which are coastal regions. This leaves us 4 regions and the remaining regions we assume zero). We finally arrive at 182 dimensions which will have a non-zero value. Hence after the invalid combinations, we will have 182 non-zero dimensions.
Regional state (112 dimensions): Variables based on 7 regions of country and 16 variables (7 regions × 16 variables):
- Regional demand (7): Regional Demand – Power Demand (TWh/year) in Each Region. - Peak demand (7): Peak Demand – Maximum demand (GW) during any hour for determining adequacy. - Renewable penetration (7): Renewable capacity divided by total capacity in region. - Transmission capacity (7): Capacity to exchange energy with neighboring regions or countries (GW). - Storage capacity (7): The total energy storage in the region, measured in GWh. - Capacity factor by technology group (28): The capacity factor for each technology group. This metric refers to the average CF for solar (7), wind (7), hydro (7), thermal (7) in each region, and reflects the quality of the resource and operational efficiency. - Regional cost index (7): The regional cost index measures the relative costs in a region. The index shows 1.0 for the national average. More than 1.0 represents a more expensive area and less than 1.0 shows a cheaper area. - Land availability (7): Renewable development square kilometres of land remaining for development. - Grid strength (7): Grid strength measures how strong is the transmission infrastructure of the country (1-10 scale, 10 = strongest). - Population (7): Millions of people live in regional areas. This represents the total demand population. - Industrial share (7): Portion of demand coming from industry against residential and commercial. - Urbanization rate (7): The demand patterns are affected by the population percentage (urban population percentage). - Renewable resource quality (7): The composite index (0-1) of solar irradiance, wind speed, and hydro potential of renewable resources quality. - Curtailment rate (7): Grid limitations will restrict renewable energy generation. As such this is known as curtailment (percentage of potential renewable generation wasted due to grid constraints).
Exogenous state (126 dimensions): Exogenous state refers to external factors beyond an agent's control affecting it:
- Technology costs (26): The cost of technology Capital costs per kW (€/kW) for each technology, which decrease over time along learning curves. - Fixed O&M costs (26): Annual O&M (€/kW-year) cost for fixed O&M. - Variable O&M costs (26): Variable maintenance costs: Generation-dependent maintenance (€/MWh). - Capacity factors (26): These are technology specific average capacity factors (updated annually). - Fuel costs (8): The prices of natural gas, coal, oil, biomass, uranium, hydrogen, synthetic fuel, carbon price (€/tonne CO₂). - Demand growth rate (1): Annual electricity demand is expected to grow at 2.5-4.5%. - GDP growth rate (1): The growth rate of the economy that creates demand and capacity for investment. - Policy parameters (12): The parameters of the precise government policy specs are the renewable subsidies (solar, wind, hydro, geothermal), the carbon tax rate, fossil retirement schedule (coal, gas, oil year-specific), nuclear policy (Sinop approved), grid investment budget, and storage mandate. - Forecast uncertainty (20): The standard deviations of forecasted capacities for the top 20 technologies will help take robust decisions. - External events (5): Outside events: Binary flags for extraordinary events (Demand shock like COVID, cost breakthrough, policy U-turn, resource find, grid going down).
This hierarchical structure enables attention mechanisms (Chapter 2, Section 2.3) focusing on relevant state components: global state for strategic decisions (should we push for more renewables?), technology-region state for allocation decisions (where to invest in solar?), regional state for constraint decisions (is transmission congested?), and exogenous state for adaptation decisions (have costs come down more than we thought?).
Action Space Design (119 Dimensions)
A design with action area of 119 dimensions.
At any time$t$, the additions and retirements across regions form the continuous action vector$\mathbf{a}_t \in \mathbb{R}^{119}Fwhere 119 is the dimensionality of the action vector.
The table provide the data be it Technology category, specific technology etc parameters which is mentioned in the above table.