Cross-Validation vs. Information Criteria: Weibull Model Selection under Incomplete and Uncertain Failure Data

Weibull analysis is a central tool in reliability engineering, to model system failures and support maintenance planning. Failure data are rare and expensive to obtain, practically always limited, censored, and the number or type of underlying failure mechanisms uncertain. Consequently, it can be unclear which Weibull model accurately describes the data. Traditionally, candidate Weibull models are fitted to all available data, with information criteria (ICs) such as Akaike's information criterion (AIC) or Bayesian information criterion (BIC) used to identify the best model by balancing fit and model complexity. However, fitting to the entire dataset without assessing predictive performance on unseen data can lead to suboptimal model selection. Cross-validation (CV) offers an alternative by evaluating a model's predictive accuracy, which can improve model selection and failure forecasts. Although prior work indicates that CV can outperform ICs in some modeling tasks, the circumstances in which it yields superior performance for single and multiple Weibull models across different censoring regimes remain insufficiently understood. To address this gap, synthetic data are generated from single and multiple two- and three-parameter Weibull models to assess model selection across realistic sparsity levels. CV exhibits the lowest predictive errors in small-sample, high-censoring scenarios. ICs yield comparable predictive performance only for larger, less censored samples. Performance in identifying the true model depends on the Weibull model type, with BIC performing best for parsimonious two-parameter models, CV for three-parameter models, and AICc for competing or mixture models. The methods are then applied to long-term failure data from two systems at the European Organization for Nuclear Research (CERN) to validate the findings, revealing additional effects not fully modeled by the synthetic setup. The findings can be translated to improved reliability modeling and cost savings in scenarios involving incomplete and uncertain failure data, but should be refined using additional real-world examples.

Keywords: Weibull analysis, model selection, cross-validation, information criteria, AICc, BIC, incomplete data, uncertain data, predictive accuracy, true underlying model.