PMC:4289553 / 2313-5050 JSONTXT

{"project":"2_test","denotations":[{"id":"25532820-16271675-11953994","span":{"begin":445,"end":446},"obj":"16271675"},{"id":"25532820-9819841-11953994","span":{"begin":445,"end":446},"obj":"9819841"},{"id":"25532820-24950066-11953994","span":{"begin":445,"end":446},"obj":"24950066"},{"id":"25532820-8970487-11953995","span":{"begin":1918,"end":1920},"obj":"8970487"},{"id":"25532820-11470385-11953995","span":{"begin":1918,"end":1920},"obj":"11470385"},{"id":"25532820-10790680-11953995","span":{"begin":1918,"end":1920},"obj":"10790680"},{"id":"25532820-19952321-11953996","span":{"begin":2132,"end":2134},"obj":"19952321"}],"text":"Background\nPrediction of binary outcomes is important in medical research. The interest in the development, validation, and clinical application of clinical prediction models is increasing [1]. Most prediction models are based on logistic regression analysis (LR), but other, more modern techniques, may also be used. Support vector machines (SVM), neural nets (NN) and random forest (RF) have received increasing attention in medical research [2–6], since these hold the promise of better capturing non-linearities and interactions in medical data. The increased flexibility of modern techniques implies that larger sample sizes may be required for reliable estimation. Little is known, however, about the sample size that is needed to generate a prediction model with a modern modelling technique that outperforms more traditional, regression-based modelling techniques in medical data.\nUsually, only a relatively limited number of subjects is available for developing prediction models. In 1995, a comparative study on the performance of various prediction models for medical outcomes concluded that the ultimate limitation seemed due to the availability of the information in data. This study used the term “data barrier” [7].\nSome researchers aimed to develop a “power law” that can be used to determine the relation between sample size and the discriminatory ability of prediction models in terms of accuracy [8–10]. These studies clarified how a larger sample size leads to a better accuracy. The studies revealed that a satisfactory level of accuracy (the accuracy at infinite sample size +/− 0.01) can be achieved by sample sizes varying from 300 to 16,000 records, depending on the modelling technique and the data structure. The relation between sample size and accuracy was reflected in learning curves. Similarly, the number of events per variable (EPV) has been studied in relation to model performance [11–15].\nIn the current study, we aimed to define learning curves to reflect the performance of a model in terms of discriminatory ability, which is a key aspect of the performance of prediction models in medicine [16]. We assumed that the discriminatory ability of a model is a monotonically increasing function of the sample size, converging to a maximum at the infinite sample size. We hypothesized that modern, more flexible techniques are more “data hungry” [17] than more conventional modelling techniques, such as regression analysis. The concept of data hungriness refers to the sample size needed for a modelling technique to generate a prediction model with a good predictive accuracy. For fair comparison, we generated reference models with each of the modelling techniques considered in our simulation study."}