Artificial intelligence and machine learning models are becoming lucid, increasingly sophisticated and deployed in energy systems. Thus, ensuring explainability and interpretability is becoming critical. Black-box models can have a high prediction accuracy but pose obstacles to adoption in policy-critical settings requiring understanding, risk assessment, and compliance from stakeholders. In this section, I review explainable AI (XAI) methods and how they are being applied to systems in the energy domain. I also position the SHAP-based analysis presented in this dissertation within the broader XAI literature.
Foundations of Explainable AI
Explainable artificial intelligence incorporates processes that help humans understand the decisions taken by the models and how they predict and behave. The field has transitioned from post-hoc methods (interpreting models after training) to inherently interpretable models (transparent by design). @doshivelez2017towards developed a taxonomy that differentiates interpretability (the ability to explain a model's behaviour by using concepts that a human can understand) and explainability (the ability to explain a particular prediction). When it comes to energy systems, both interpretability and explainability are essential to policymaking. Specifically, the interpretation helps policymakers understand a more general normative model behavior while they can see the specific capacity expansion decision with explainability.
Post-hoc explanation methods analyze trained models without changing their architecture. SHAP (SHapley Additive exPlanations), developed by [lundberg2017unified], has a unified framework that quantifies contribution of each feature of prediction of a data point based on the Shapley values of cooperative game theory. SHAP satisfies four desirable properties. First, efficiency means that the sum of feature contributions equals the difference between the model's prediction and the prediction among the training data. Second, symmetry states that features with equal marginal contributions to a prediction receive equal SHAP values. Third, dummy states that features with zero marginal contribution to prediction receive SHAP value of 0. Finally, additivity means the SHAP values for an ensemble model decompose into the contributions of each component model. SHAP helps identify the drivers of specific capacity expansion decisions such as technology costs, policy parameters, and resource availability as part of energy foresighting and optimisation as a tool to design clear policy.
LIME (Local Interpretable Model-agnostic Explanations), introduced by Ribeiro et al. [ribeiro2016why], is a method that approximates any complex model locally by an interpretable linear model. LIME creates explanations by making slight changes to input features around specific instances. Next, we train simple models (for example, Linear regression, decision trees) on perturbed samples to extract feature importance. LIME provides an easy-to-understand explanation of the output of a black box model but it suffers from instability (explanations vary across runs). In energy systems, demand forecasting models [antonopoulos2020energy] and renewable generation forecasts [wang2021energy] have been interpreted with LIME.
Attention mechanisms in neural networks show some interpretability allowing one to see the input feature which is focused upon to make predictions. Attention weights determine the importance of features, allowing us to visualize the attention patterns of the model. Transformer architectures that rely on self-attention have been used for energy forecasting [wu2020energy]. Further, it delivers interpretable feature importance without post hoc analysis. Nonetheless, attention weights may not be directly related to causal importance, requiring careful interpretation.
Integrated Gradients, proposed by Sundararajan et al. (2017), assigns a prediction to a feature by integrating the gradient of the features on a path from a baseline to the input. This technique meets implementation invariance because it assign the same attribution to functionally equivalent models. Integrated Gradients has been effective in interpreting deep learning models for electricity price forecasting conducted by Zhang et al. (2022).
XAI Applications in Energy Systems
Forecasting applications: XAI methods have been applied to forecasting applications, specifically energy demand modelling which highlights that temperature, day-of-week and historical consumption has the largest bearing [antonopoulos2020energy]. SHAP analysis illustrates that weather variables like wind speed and solar irradiance and temporal features such as hour-of-day and season are the main drivers for forecasting renewable generation [wang2021energy]. Information can be utilized for model validation (ensuring that models depend on physically meaningful features) and feature engineering (determining which inputs matter more).
Optimization applications: Utilization of XAI for optimization is less explored. Many optimization analyses claim optimal solutions but do not provide rationale for technology and location selection. The SHAP analysis performed on the RL policies in this dissertation has been able to reveal the decision drivers for the capacity expansion. The costs of wind, carbon price, transmission capacity have greater variance of 24%, 19%, and 14% respectively making them a greater driver of capacity expansion. The designed the policies will be transparent: if renewables deployment is response of carbon pricing, carbon prices will be shaped if policymakers do not like.
Policy applications: XAI enables counterfactual and sensitivity analysis to support evidence-based policy. What-if scenario counterfactual. Parameter importance ranking sensitivity analysis. Analysis using SHAP indicates that the impact of the expansion of the transmission network along the Southeast→Marmara corridor for the energy transition of Turkey is stronger than that of the solar cost reductions. Therefore, investment in infrastructure should be prioritised over a subsidy policy on technology. This information informs recommendations for policy (Chapter 4, Section 4.3).
Challenges and Limitations
In the application of energy systems, XAI methods face challenges:
1. High-dimensional feature spaces: Energy models account for more than 100 features like costs, policy, weather, weather and economics. SHAP computation takes exponential time in the number of features, and approximation methods like Kernel SHAP and Tree SHAP are commonly used to speed things up but at the cost of accuracy.
2. Temporal dependencies: Energy systems show considerable temporal dependencies, with capacity accumulation and cost evolution. Standard XAI approaches presume independent features, potentially influencing temporal interaction effects.
3. Multi-objective trade-offs: Energy optimization involves trade-offs between a number objectives (cost, emissions, reliability). XAI algorithms usually explain predictions when there's a single objective. They need to be extended for multi-objective scenario.
4. Causal vs. correlational explanations: XAI techniques acknowledge that their identification of statistical associations should not be inferred as causality. Causal insights are essential for policy design. For example, knowing that renewable deployment correlates with carbon price is different from knowing that carbon pricing *causes* deployment.
5. Model-agnostic vs. model-specific: Model-agnostic ways like SHAP and LIME can work with any type of model. But they may not be as precise as methods designed specifically for the model type. An example of model-specific methods would be focus weights for Transformers. Similarly, tree-based importance for Random Forests is another example.
Future Directions
In 2023 and 2024, researchers have introduced several new tools that involve XAI (explainable artificial intelligence). Some causal XAI tools have taken SHAP and combined it with causal inference methods [pearl2023causality]. This will help to distinguish between causal and correlational relationships. Next, there has been a temporal XAI that extends SHAP for time series models (TimeSHAP, Bento et al., 2023). Moreover, there have also been multi-objective XAI tools that help explain Pareto-optimal solutions (ParetoSHAP, Chen et al., 2024). Finally, researchers have been looking into adversarial XAI which tests the robustness of these explanations against adversarial perturbations (AdversarialSHAP, Li et al., 2024).
This dissertation contributes to the XAI literature in four primary ways. First, it applies SHAP to explain multi-agent reinforcement learning (RL) policies, which is a novel application domain. Second, it quantifies decision variance across the 446-dimensional state space of an RL policy. Third, it enables counterfactual policy analysis through the use of SHAP. Finally, it provides contrasting explanations that demonstrate the value of XAI to evidence-based energy policy design. To avoid any confusion, I use the expressions "446-dimensional" for state space and "119-dimensional" for action space throughout this dissertation.