Autonomous Systems and the Erosion of Adaptation: Challenges for Human-AI Collaboration

The rapid development of Adaptive Artificial Intelligence (AAI) has revived ambitions to reduce or remove human involvement from safety-critical systems. Yet, by embedding only partial or simplified models of human adaptive behavior, current engineering approaches risk undermining the very adaptive capacity they aim to achieve. This paper examines how autonomous systems draw on either top-down formalized adaptation models or bottom-up data-driven learning from experts and shows why both approaches struggle to represent real-world operational decision-making. Human adaptation is grounded on expectations, mental simulation, and experimentation, features that dominant models fail to fully capture. Data-driven systems, meanwhile, cannot distinguish normal from adapted behavior unless designers explicitly identify and encode the cues, evaluative criteria, and deviation strategies underlying adaptive decision-making. As increasing autonomy removes humans from practice, opportunities to observe adaptation diminish. The paper argues that maintaining adaptive capacity requires grounding autonomous systems in naturalistic decision-making models and applying ergonomic principles that support human adaptive contributions in collaborative human-AI systems.

Keywords: Autonomous Systems, Human-AI Collaboration, Adaptation, Decision-making.