On Benchmarking Modified Metrics in Security Risk Assessment

Risk assessment often requires balancing accuracy and usability. While quantitative metrics promise objectivity, they are typically costly to apply, leading practitioners to rely on qualitative, semi-quantitative, or simplified quantitative metrics. However, this entails modifications to quantitative metrics that may introduce discrepancies and reduce validity. This paper aims to contribute to the assessment of the quality of such metrics by conceptualizing metric modification as a structured design space defined by key degrees of freedom, including scaling, dimensionality reduction, and scenario clustering. Rather than proposing another metric, the paper focuses on the conditions under which modified metrics can be considered methodologically defensible with respect to a quantitative reference. Criteria for qualifying deviations from benchmark metrics are discussed, covering objectivity, reliability, scale resolution, contextual fit, and challenges related to aleatoric and epistemic uncertainty. Based on these criteria, the paper derives explicit assumptions and alignment strategies that allow systematic identification, explanation, and limitation of distortions introduced by metric simplification. The approach deliberately abstracts from adaptive attacker behavior; changes in attacker capabilities, tools, or strategies are treated as defining distinct scenarios rather than as dynamic modifications within a fixed scenario. Using physical security as an illustrative domain, the paper examines the assumptions required for metric alignment and contributes guidance for the practical use of simplified metrics aligned with quantitative risk assessment. The results provide actionable guidance for metric designers and practitioners on when simplified metrics are suitable for decision support and where their use becomes methodologically unjustifiable.

Keywords: Modified Metrics, Risk Analysis, Uncertainty, Benchmarks.