Why compliance is not enough: Evaluating AI Systems in Safety-Critical Contexts

In 2024 the European Union began implementing the Artificial Intelligence Act (EU AI Act) - the first regulation of its type. This has generated interest and concern across industrial and academic forums. The EU AI Act follows a risk classification approach based on the impact that different Artificial Intelligence (AI) systems can have on health, safety, and fundamental rights. However, this conception of risk primarily reflects societal and human rights concerns, being distinct from the typical risk classification approaches in safety critical contexts, such as the oil and gas industry, which focus on preventing major accidents and controlling associated risks.
This discrepancy on the interpretation of risk can result in mismatches in assessment outcomes. For instance, an AI system that is considered "low risk" under the EU AI Act could be "high risk" when assessed through a major accident prevention framework. In this work we explore this inconsistency and analyse its impact for safety critical industries.
We illustrate the issue presenting examples of AI systems that would need to be further investigated, evaluated, and qualified for deployment, even though they would not entail further assessment according to the EU AI Act requirements. We discuss the implications of these findings for technology qualification processes and human-AI interaction evaluations, highlighting the relevance of other pre-existing regulation for safety critical industries. Additionally, we consider how safety critical sectors might benefit from specific requirements and upcoming standards associated with high-risk AI systems under the EU AI Act. We argue that these requirements and guidelines could be valuable beyond the EU AI Act classification paradigm when integrated into pre-existing industry safety frameworks. We highlight the relevance of nuanced perspectives as well as context aware approaches to AI risk assessment, safeguarding that the new regulatory focus does not overlook recognised safety good practices.

Keywords: Artificial intelligence, safety risk, safety critical systems, human-AI interaction, technology qualification, EU AI Act.