TCAV-Based Operator Support System for Human Reliability Enhancement in Safety-Critical Environments

This study introduces a concept-level explanation method for AI-based nuclear power plant (NPP) diagnostic systems using Testing with Concept Activation Vectors (TCAV). Current explainable AI (XAI) methods, such as Layer-wise Relevance Propagation (LRP), provide sensor-level attributions that identify which input variables are important for the model's prediction. However, these attributions lack behavioral context-they indicate the importance of a sensor without describing how the sensor behaves. This limitation makes it difficult for operators to interpret and verify the AI's reasoning during complex diagnostic situations. To address this limitation, we propose a TCAV-based operator support system that provides concept-level explanations. The proposed method defines behavioral concepts (e.g., "very high," "low," "oscillating") that correspond to recognizable sensor signal patterns in NPP operations. These concepts are combined with PageRankweighted centrality scoring to rank the most influential and structurally important diagnostic patterns. We applied the proposed method to the IAEA iPWR simulator dataset, focusing on two representative scenarios: feedwater pump failure (AB_01) and loss of coolant accident (EM_01). The experimental results demonstrate that LRP and TCAV identify fundamentally different aspects of model behavior, with only 5−10 % overlap between the two methods. While LRP tends to highlight directly affected sensors (e.g., pump speed during a pump failure), TCAV identifies system-level behavioral patterns (e.g., steam generator inlet temperature being very high) that provide more informative reasoning for operators. The proposed approach offers a practical framework for generating AI explanations that operators can directly verify against their domain knowledge and current plant observations.

Keywords: Explainable AI, Test with Concept Activation Vector, Nuclear Power Plant, Operator Suppoprt, Conceptlevel Explanation.