Beyond Component Failures: A System-Theoretic Hazard Identification for AIEnabled Systems

The deployment of Artificial Intelligence (AI) systems into safety-critical systems offers the potential to improve safety, but also introduces novel hazards. Traditional hazard identification methods, focused on component-level failures and chain of failure events, are often insufficient for these complex AI-enabled systems where unsafe component interactions are a primary concern for accidents. Recent research advocates conducting hazard identification for AI-enabled systems with a system-theoretic approach to address unsafe component interactions of system components and traditional component failures. This study adopts a system-theoretic perspective, employing system-theoretic process analysis (STPA) to investigate operational safety challenges in a liquid hydrogen (L H₂) bunkering system that incorporates an AI system for predicting the violation of the lower flammability limit (LFL) of hydrogen. Through our analysis, we identify the hazards and loss scenarios of two different system designs for such an AI-enabled safety-critical system. Furthermore, we derive benefits, limitations and propose enhancements to the STPA methodology itself to better address the characteristics of AI-enabled systems. Our findings contribute to advancing hazard identification methods and risk assessment frameworks, promoting the safe integration of AI systems in safety-critical systems.

Keywords: STPA, AI-enabled systems, safety-critical systems, hazard identification, risk assessment, liquid hydrogen bunkering.