Physics-Informed Neural Forecasting System

Neural forecasts that take physics into consideration is a topic of discussion due to its importance. The hybrid system gives scenario-based generation forecasts and responsiveness estimates. The Hybrid Neural S-curve Model (HNSM) and the Enhanced Neural S-curve Model (ENSM) are two model variants based on LSTM/GRU temporal modeling, convolutions and sigmoid/Gompertz diffusion curves and physics constraints (energy balance and lower/upper capacity factor bounds), to fill the gaps in forecasting identified in Chapter 2, Section 2.6.

The Limitations of Pure Data-Driven vs Pure Physics-Based Forecasting

Models that simply recognize patterns in the data like LSTM and XGboost fail to extrapolate further out of the distribution on which they were trained. It is necessary to develop additional modifications and stress logic for the successful conduct of predictions during a systems break (change in policy, cost break out). Unless made explicit, these models may yield physically unreasonable outcomes with a negative capacity or capacity factors greater than one.

Modeling methods that strictly utilize physics knowledge (such as system dynamics models embedding the Bass diffusion model, learning curves, etc.) ensure plausible forecasts through knowledge embeddedness from the domain of application. However, they require heavy calibration of many parameters (market size m, innovation coefficient p, imitation coefficient q in the Bass model). These parameters are often hardly available for new technologies. Additionally, when the actual path of adoption follows other routes, the physicists' models having fixed functional forms (e.g., logistic S-curve) could be misspecified.

The hybrid physics-informed neural networks incorporate the advantages of neural networks and physics-informed constraints. The former provides the flexibility of function approximation while learning from data. The latter ensures the plausibility of the models while improving the extrapolation capability of the model. Raissi et al. were the first to apply this methodology for fluid dynamics in 2019, which was subsequently used by Chakraborty et al. for energy in 2021. To date, however, there is no effort made with this methodology for capacity forecasting. This particular study attempts to fill this gap.

HNSM stands for Hybrid Neural S-Curve Model Architecture

HNSM employs customized loss functions and output transformations to incorporate sigmoid S-curve dynamics into LSTMs. The architecture consists of four components.

Component 1: Temporal encoder (LSTM layers): Two layers of encoding are applied to the previous capacity sequences by bidirectional LSTM:

$${\overrightarrow{\mathbf{h}}}_t={\mathbf{LSTM}}_{fwd}({\mathbf{x}}_t,{\overrightarrow{\mathbf{h}}}_{t-1}){\overleftarrow{\mathbf{h}}}_t={\mathbf{LSTM}}_{bwd}({\mathbf{x}}_t,{\overleftarrow{\mathbf{h}}}_{t+1}){\mathbf{h}}_t=[{\overrightarrow{\mathbf{h}}}_t;{\overleftarrow{\mathbf{h}}}_t]$$

At time step t, ${\mathbf{x}}_t$ is input (capacity, costs, exogenous variables). ${\overrightarrow{\mathbf{h}}}_t$ and ${\overleftarrow{\mathbf{h}}}_t$ are forward/backward hidden states (256 dimensions), concatenated to 512-dimensional ${\mathbf{h}}_t$. Training uses forward and backward passes to enhance error signals that create better representations.

Component 2: S-curve output transformation: The S-curve output transformation is the second component. One or more fully connected layers receive the hidden states, giving rise to three sigmoid parameters per technology-region:

$$K_{max}=\mathbf{Softplus}({\mathbf{W}}_K{\mathbf{h}}_T+{\mathbf{b}}_K)(carryingcapacity)r=\mathbf{Softplus}({\mathbf{W}}_r{\mathbf{h}}_T+{\mathbf{b}}_r)(growthrate)t_0={\mathbf{W}}_{t_0}{\mathbf{h}}_T+{\mathbf{b}}_{t_0}(inflectionpoint)$$

The softplus activation function, given by the formula log(1 + e^x), strongly suggests that the capacity Kmax cannot be negative. This is because the softplus function does not yield negative values for any real number. The adoption process will incur a growth rate of $r>0$ due to positive output. Expected capacity resembles sigma:

$${\hat{K}}_t=\frac{K_{max}}{1+\exp (-r(t-t_0))}$$

Forecasts exhibit an S-curve functional form, whereby the initial slowdown (with the growth of installed base happening during the experimentation phase), then exponential growth (mechanism that is word-of-mouth diffusion and economies of scale), saturation and so on. Following the phase, is counterbalanced by market limit and resources. Calibration through backpropagation based on data eliminates the need for specification.

[Figure]

*Hybrid Neural-Physical Forecasting System Architecture: HNSM and ENSM model architectures integrating LSTM temporal encoding with S-curve diffusion dynamics and physics constraints (energy balance, capacity factor bounds).*

Component 3: Physics constraints (soft penalties): The loss function now includes penalty terms for keeping the soft physics more plausible:

$${\mathcal{L}}_{physics}={\lambda }_1{\mathcal{L}}_{energy}+{\lambda }_2{\mathcal{L}}_{CF}+{\lambda }_3{\mathcal{L}}_{mono}$$

where:

- **Energy balance:** The energy balance limitation is dependent on the following ratio:

$$G_t=\sum _{i=1}^{26}\sum _{r=1}^7{K}_{i,r,t}\cdot C{F}_{i,r,t}\cdot \frac{8760}{1000}$$

In this equation, $G_t$ indicates total generation (TWh) at time $t$, while $K_{i,r,t}$ represents installed capacity (GW) for technology $i$ in region $r$. In the same manner, CFi,r,t is a dimensionless capacity factor. Moreover, $8760/1000 = 8.76$ converts GW×hours TWh. The loss term imposes penalties for violations:

$${\mathcal{L}}_{energy}=\sum _t{\left|{\hat{G}}_t-\sum _{i=1}^{26}\sum _{r=1}^7{\hat{K}}_{i,r,t}\cdot {\widehat{CF}}_{i,r,t}\cdot 8.76\right|}^2$$

- **Capacity factor bounds:** The capacity factor bounds as given below penalizes the negative of capacity factor and a capacity factor greater than one. ${\mathcal{L}}_{CF}={\sum }_t[\max (0,C{F}_t-1{)}^2+\max (0,-C{F}_t{)}^2]$ penalizes $CF > 1$ (impossible) or $CF < 0$ (nonsensical). - **Monotonicity (absent retirements):** The penalty function ${\mathcal{L}}_{mono}={\sum }_t\max (0,{\hat{K}}_t-{\hat{K}}_{t+1}{)}^2$ ensures that no further retirements happen.