Predictive diagnostics for logistic models.

Abstract

Novel methodology is implemented to assess the predictive power of covariate information associated with sequential binary events. Logistic models are first fitted on the basis of a subset of the observations and then evaluated sequentially on the rest. The probabilistic forecasts are compared to the outcomes via a scoring function, but as most validation samples are small, the usual reference distribution for the test statistics is inadequate. However, bootstrap-based distributions can easily be constructed. The first example pertains to the evaluation of screening tests for major depression. It illustrates that goodness-of-fit and predictive assessments lead to the selection of very different models. The second example deals with the prediction of a major event in the natural history of HIV-induced disease. It shows that this type of analysis can reveal features missed by other approaches.