Single-valued Risk Bound for Normal Classifiers

Machine Learning has entered almost every aspect of our lives, particularly in software systems. Their performance in several tasks has reached human-level and is therefore interesting to use in safety-critical systems. However, the black-box character of most advanced models hinders the use of many classical assessment techniques. This makes it more difficult to acquire the crisp failure rates needed in techniques like fault trees, reliability block diagrams or partially FMCEA. In this contribution, we aim to provide a bridge from testing to an estimation of an upper bound. This estimated delimiter of the model's risk can then be used in established techniques and tools to obtain early estimations. That is particularly useful in early design stages to foster a feedback loop about planned system designs. Our method is suited for any classifier with a normal distributed score. The estimations are established with the use of confidence bounds on the parameters. More specifically, we are going to propose a function, which has the confidence levels as variables, that serves as an upper bound estimation. In a follow-up step, the minimum of this function can be used as an upper bound estimation, which is given by a single number. On top of that, the minimizing arguments can be seen as an optimal choice of the confidence levels. In a simulation, we are going to prove the validity of this attempt. Additionally, we are going to demonstrate its application with the example of breast cancer detection.

Keywords: machine learning, risk assessment, confidence selection, upper bound estimation, reliability, normal distribution.