PMC:4289553 / 10419-18055 JSONTXT

{"target":"http://pubannotation.org/docs/sourcedb/PMC/sourceid/4289553","sourcedb":"PMC","sourceid":"4289553","source_url":"https://www.ncbi.nlm.nih.gov/pmc/4289553","text":"Results\n\nHNSCC cohort\nThe best performance in terms of mean validated AUC-values was achieved when the full development set was used (n = 12,820, number of events = 6013, event rate 46.9%) and by the models generated with the same modelling technique as the reference model, except when the reference model was generated with NN, in which case the RF model had the best performance (AUC 0.810, Table 2).\nTable 2 AUCmax per reference model, HNSCC cohort\nReference model\nLR CART SV NN RF\nLR 0.797 0.745 0.803 0.802 0.880\nCART 0.730 0.748 0.749 0.728 0.822\nSVM 0.787 0.740 0.814 0.802 0.898\nNN 0.785 0.744 0.800 0.804 0.869\nRF 0.784 0.747 0.810 0.810 0.929\nBold numbers are for model performance when the underlying model was specified according to the modelling technique considered.\nThe level that could be reached (AUCmax) depended foremost on the reference model used to generate the 0/1 outcomes. All models performed best when the reference model RF was used. For all reference models, except the CART reference model, the CART model performed worst (Table 2).The data hungriness of the various modelling techniques is reflected by the first part of the learning curves with \u003c100 events per variable (Figure 2). As expected, all models converged monotonically to AUCmax. For each of the reference models, the LR model showed the most rapid increase to a stable mean validated AUC-value, while the RF model needed the largest number of events per variable to reach a stable mean validated AUC-value (Figure 2).We calculated the relative performance of a model by setting the performance of the model resulting from the modelling technique that generated the reference model at 100%. Figure 3 shows the relative performance of the models for each reference model.\nFigure 2 Validated AUC-values vs. events per variable, HNSCC cohort.\nFigure 3 Relative validated AUC-values vs. events per variable, HNSCC cohort.\nFor all reference models, the optimism of the models decreased with an increasing number of events per variable. For all reference models, except when the reference model was CART, the modelling technique LR needed the smallest number of events per variable to reach an optimism \u003c0.01 (55 to 127 events per variable).\nWhen CART was the reference model, the modelling technique CART needed the smallest number of events per variable to reach an optimism \u003c0.01 (62 events per variable). The modelling techniques NN and RF and, to a lesser extent, SVM needed the most events per variable to generate models with an optimism \u003c0.01.The modelling technique RF needed 850 events per variable when the reference model RF was used, but for the other reference models the optimism of the RF model remained \u003e =0.01, despite the large number of events per variable (Figure 4).\nFigure 4 Optimism vs. events per variable, HNSCC cohort.\n\nTBI cohort\nFor the TBI artificial cohort, with a development set consisting of 8655 subjects and 1930 events (event rate 22.3%), the CART models performed poorly, irrespective of the reference model (Table 3). The models generated with the same modelling technique as the reference model showed the best performance, except when the reference model was generated with CART, in which case the LR model had the best performance (AUC 0.712, Table 3). All models, except the CART model, showed the lowest AUC when the reference model CART was used (Table 3).The NN model needed the largest number of events per variable to reach AUCmax. For each of the reference models, the LR model showed the most rapid increase to a stable AUC (Figure 5).Again, we calculated the relative performance of a model by setting the performance of the model resulting from the modelling technique that generated the reference model at 100%. Figure 6 shows the relative performance of the models for each reference model.For all models, optimism decreased with an increasing number of events per variable. The LR model needed 18–23 events per variable to reach an optimism \u003c0.01, whereas the optimism of the RF model remained high, except for the reference model RF, in which case optimism was \u003c0.01 at 163 events per variable (Figure 7).\nTable 3 AUCmax per reference model, TBI cohort\nReference model\nLR CART SVM NN RF\nLR 0.806 0.712 0.743 0.762 0.817\nCART 0.710 0.702 0.676 0.652 0.684\nSVM 0.754 0.677 0.765 .0765 0.838\nNN 0.800 0.701 0.746 0.802 0.828\nRF 0.744 0.685 0.750 0.776 0.988\nBold numbers are for model performance when the underlying model was specified according to the modelling technique considered.\nFigure 5 Validated AUC-values vs. number of events per variable, TBI cohort.\nFigure 6 Relative validated AUC-values vs. events per variable, TBI cohort.\nFigure 7 Optimism vs. events per variable, TBI cohort.\n\nCHIP cohort\nFor the CHIP artificial cohort, with a development set consisting of 9543 subjects and 729 events (event rate 7.64%), the findings were largely similar to the results of the HNSCC cohort. The best performance was achieved by the same modelling technique that generated the reference model (Table 4). The modelling technique CART generated models with a poor performance, irrespective of the reference models. The modelling technique SVM also generated models with a poor performance, irrespective of the reference models, except when the RF model was used as reference model (AUC 0.871, Table 4). All models performed poorly when the reference models CART and SVM were used. All models, except the CART model, performed well when the reference model RF was used (AUC \u003e 0.8, Table 4).Considering the learning curves (Figure 8), the CART models performed poorly. For each of the reference models, the LR model showed a rapid increase to a stable mean validated AUC-value, in contrast to the NN model which needed far more events to reach a stable mean validated AUC-value. The CART model showed a decreasing mean validated AUC-value despite increasing number of events, except when the reference model CART was used (Figure 8).Figure 9 shows the relative performance of the models for each reference model.For the reference models LR, SVM and NN, the modelling technique LR required 14 to 28 events per variable to reach an optimism \u003c0.01 and CART required 11 to 17 events per variable. Despite an increasing number of events per variable, the modelling techniques SVM, NN and RF generated models with optimism \u003e0.01 for all reference models. For the reference models CART and RF, none of the modelling techniques was able to generate a model with optimism \u003c0.01 (Figure 10).\nTable 4 AUCmax per reference model, CHIP cohort\nReference model\nLR CART SVM NN RF\nLR 0.786 0.572 0.607 0.782 0.903\nCART 0.562 0.578 0.580 0.500 0.666\nSVM 0.584 0.560 0.615 0.616 0.871\nNN 0.758 0.564 0.589 0.791 0.856\nRF 0.728 0.579 0.594 0.755 0.916\nBold numbers are for model performance when the underlying model was specified according to the modelling technique considered.\nFigure 8 Validated AUC-values vs. events per variable, CHIP cohort.\nFigure 9 Relative validated AUC-values vs. events per variable, CHIP cohort.\nFigure 10 Optimism vs. events per variable, CHIP cohort.\n\nSensitivity analysis CHIP cohort\nWhen we increased the event rate in the CHIP cohort from 7.6% to 50% (“CHIP5050 cohort”), the behaviour of the learning curves became largely similar to the behaviour of the curves generated for the HNSCC cohort (Additional file 3, Figures 11, 12 and 13).\nFigure 11 Validated AUC-values vs. events per variable, CHIP5050 cohort.\nFigure 12 Relative validated AUC-values vs. events per variable, CHIP5050 cohort.\nFigure 13 Optimism vs. events per variable, CHIP5050 cohort.","divisions":[{"label":"title","span":{"begin":0,"end":7}},{"label":"sec","span":{"begin":9,"end":2842}},{"label":"title","span":{"begin":9,"end":21}},{"label":"p","span":{"begin":22,"end":787}},{"label":"table-wrap","span":{"begin":404,"end":787}},{"label":"label","span":{"begin":404,"end":411}},{"label":"caption","span":{"begin":412,"end":453}},{"label":"p","span":{"begin":412,"end":453}},{"label":"table","span":{"begin":454,"end":659}},{"label":"tr","span":{"begin":454,"end":469}},{"label":"th","span":{"begin":454,"end":469}},{"label":"tr","span":{"begin":470,"end":486}},{"label":"th","span":{"begin":470,"end":472}},{"label":"th","span":{"begin":473,"end":477}},{"label":"th","span":{"begin":478,"end":480}},{"label":"th","span":{"begin":481,"end":483}},{"label":"th","span":{"begin":484,"end":486}},{"label":"tr","span":{"begin":487,"end":520}},{"label":"td","span":{"begin":487,"end":490}},{"label":"td","span":{"begin":490,"end":496}},{"label":"td","span":{"begin":497,"end":502}},{"label":"td","span":{"begin":503,"end":508}},{"label":"td","span":{"begin":509,"end":514}},{"label":"td","span":{"begin":515,"end":520}},{"label":"tr","span":{"begin":521,"end":556}},{"label":"td","span":{"begin":521,"end":526}},{"label":"td","span":{"begin":526,"end":531}},{"label":"td","span":{"begin":532,"end":538}},{"label":"td","span":{"begin":539,"end":544}},{"label":"td","span":{"begin":545,"end":550}},{"label":"td","span":{"begin":551,"end":556}},{"label":"tr","span":{"begin":557,"end":591}},{"label":"td","span":{"begin":557,"end":561}},{"label":"td","span":{"begin":561,"end":566}},{"label":"td","span":{"begin":567,"end":572}},{"label":"td","span":{"begin":573,"end":579}},{"label":"td","span":{"begin":580,"end":585}},{"label":"td","span":{"begin":586,"end":591}},{"label":"tr","span":{"begin":592,"end":625}},{"label":"td","span":{"begin":592,"end":595}},{"label":"td","span":{"begin":595,"end":600}},{"label":"td","span":{"begin":601,"end":606}},{"label":"td","span":{"begin":607,"end":612}},{"label":"td","span":{"begin":613,"end":619}},{"label":"td","span":{"begin":620,"end":625}},{"label":"tr","span":{"begin":626,"end":659}},{"label":"td","span":{"begin":626,"end":629}},{"label":"td","span":{"begin":629,"end":634}},{"label":"td","span":{"begin":635,"end":640}},{"label":"td","span":{"begin":641,"end":646}},{"label":"td","span":{"begin":647,"end":652}},{"label":"td","span":{"begin":653,"end":659}},{"label":"table-wrap-foot","span":{"begin":660,"end":787}},{"label":"p","span":{"begin":660,"end":787}},{"label":"p","span":{"begin":788,"end":1919}},{"label":"figure","span":{"begin":1771,"end":1840}},{"label":"label","span":{"begin":1771,"end":1779}},{"label":"caption","span":{"begin":1780,"end":1840}},{"label":"p","span":{"begin":1780,"end":1840}},{"label":"figure","span":{"begin":1841,"end":1919}},{"label":"label","span":{"begin":1841,"end":1849}},{"label":"caption","span":{"begin":1850,"end":1919}},{"label":"p","span":{"begin":1850,"end":1919}},{"label":"p","span":{"begin":1920,"end":2237}},{"label":"p","span":{"begin":2238,"end":2842}},{"label":"figure","span":{"begin":2785,"end":2842}},{"label":"label","span":{"begin":2785,"end":2793}},{"label":"caption","span":{"begin":2794,"end":2842}},{"label":"p","span":{"begin":2794,"end":2842}},{"label":"sec","span":{"begin":2844,"end":4752}},{"label":"title","span":{"begin":2844,"end":2854}},{"label":"p","span":{"begin":2855,"end":4752}},{"label":"table-wrap","span":{"begin":4159,"end":4541}},{"label":"label","span":{"begin":4159,"end":4166}},{"label":"caption","span":{"begin":4167,"end":4206}},{"label":"p","span":{"begin":4167,"end":4206}},{"label":"table","span":{"begin":4207,"end":4413}},{"label":"tr","span":{"begin":4207,"end":4222}},{"label":"th","span":{"begin":4207,"end":4222}},{"label":"tr","span":{"begin":4223,"end":4240}},{"label":"th","span":{"begin":4223,"end":4225}},{"label":"th","span":{"begin":4226,"end":4230}},{"label":"th","span":{"begin":4231,"end":4234}},{"label":"th","span":{"begin":4235,"end":4237}},{"label":"th","span":{"begin":4238,"end":4240}},{"label":"tr","span":{"begin":4241,"end":4274}},{"label":"td","span":{"begin":4241,"end":4243}},{"label":"td","span":{"begin":4244,"end":4250}},{"label":"td","span":{"begin":4251,"end":4256}},{"label":"td","span":{"begin":4257,"end":4262}},{"label":"td","span":{"begin":4263,"end":4268}},{"label":"td","span":{"begin":4269,"end":4274}},{"label":"tr","span":{"begin":4275,"end":4310}},{"label":"td","span":{"begin":4275,"end":4279}},{"label":"td","span":{"begin":4280,"end":4285}},{"label":"td","span":{"begin":4286,"end":4292}},{"label":"td","span":{"begin":4293,"end":4298}},{"label":"td","span":{"begin":4299,"end":4304}},{"label":"td","span":{"begin":4305,"end":4310}},{"label":"tr","span":{"begin":4311,"end":4345}},{"label":"td","span":{"begin":4311,"end":4314}},{"label":"td","span":{"begin":4315,"end":4320}},{"label":"td","span":{"begin":4321,"end":4326}},{"label":"td","span":{"begin":4327,"end":4333}},{"label":"td","span":{"begin":4334,"end":4339}},{"label":"td","span":{"begin":4340,"end":4345}},{"label":"tr","span":{"begin":4346,"end":4379}},{"label":"td","span":{"begin":4346,"end":4348}},{"label":"td","span":{"begin":4349,"end":4354}},{"label":"td","span":{"begin":4355,"end":4360}},{"label":"td","span":{"begin":4361,"end":4366}},{"label":"td","span":{"begin":4367,"end":4373}},{"label":"td","span":{"begin":4374,"end":4379}},{"label":"tr","span":{"begin":4380,"end":4413}},{"label":"td","span":{"begin":4380,"end":4382}},{"label":"td","span":{"begin":4383,"end":4388}},{"label":"td","span":{"begin":4389,"end":4394}},{"label":"td","span":{"begin":4395,"end":4400}},{"label":"td","span":{"begin":4401,"end":4406}},{"label":"td","span":{"begin":4407,"end":4413}},{"label":"table-wrap-foot","span":{"begin":4414,"end":4541}},{"label":"p","span":{"begin":4414,"end":4541}},{"label":"figure","span":{"begin":4542,"end":4619}},{"label":"label","span":{"begin":4542,"end":4550}},{"label":"caption","span":{"begin":4551,"end":4619}},{"label":"p","span":{"begin":4551,"end":4619}},{"label":"figure","span":{"begin":4620,"end":4696}},{"label":"label","span":{"begin":4620,"end":4628}},{"label":"caption","span":{"begin":4629,"end":4696}},{"label":"p","span":{"begin":4629,"end":4696}},{"label":"figure","span":{"begin":4697,"end":4752}},{"label":"label","span":{"begin":4697,"end":4705}},{"label":"caption","span":{"begin":4706,"end":4752}},{"label":"p","span":{"begin":4706,"end":4752}},{"label":"sec","span":{"begin":4754,"end":7127}},{"label":"title","span":{"begin":4754,"end":4765}},{"label":"p","span":{"begin":4766,"end":7127}},{"label":"table-wrap","span":{"begin":6540,"end":6922}},{"label":"label","span":{"begin":6540,"end":6547}},{"label":"caption","span":{"begin":6548,"end":6588}},{"label":"p","span":{"begin":6548,"end":6588}},{"label":"table","span":{"begin":6589,"end":6794}},{"label":"tr","span":{"begin":6589,"end":6604}},{"label":"th","span":{"begin":6589,"end":6604}},{"label":"tr","span":{"begin":6605,"end":6622}},{"label":"th","span":{"begin":6605,"end":6607}},{"label":"th","span":{"begin":6608,"end":6612}},{"label":"th","span":{"begin":6613,"end":6616}},{"label":"th","span":{"begin":6617,"end":6619}},{"label":"th","span":{"begin":6620,"end":6622}},{"label":"tr","span":{"begin":6623,"end":6655}},{"label":"td","span":{"begin":6623,"end":6625}},{"label":"td","span":{"begin":6626,"end":6631}},{"label":"td","span":{"begin":6632,"end":6637}},{"label":"td","span":{"begin":6638,"end":6643}},{"label":"td","span":{"begin":6644,"end":6649}},{"label":"td","span":{"begin":6650,"end":6655}},{"label":"tr","span":{"begin":6656,"end":6691}},{"label":"td","span":{"begin":6656,"end":6660}},{"label":"td","span":{"begin":6661,"end":6666}},{"label":"td","span":{"begin":6667,"end":6673}},{"label":"td","span":{"begin":6674,"end":6679}},{"label":"td","span":{"begin":6680,"end":6685}},{"label":"td","span":{"begin":6686,"end":6691}},{"label":"tr","span":{"begin":6692,"end":6726}},{"label":"td","span":{"begin":6692,"end":6695}},{"label":"td","span":{"begin":6696,"end":6701}},{"label":"td","span":{"begin":6702,"end":6707}},{"label":"td","span":{"begin":6708,"end":6714}},{"label":"td","span":{"begin":6715,"end":6720}},{"label":"td","span":{"begin":6721,"end":6726}},{"label":"tr","span":{"begin":6727,"end":6760}},{"label":"td","span":{"begin":6727,"end":6729}},{"label":"td","span":{"begin":6730,"end":6735}},{"label":"td","span":{"begin":6736,"end":6741}},{"label":"td","span":{"begin":6742,"end":6747}},{"label":"td","span":{"begin":6748,"end":6754}},{"label":"td","span":{"begin":6755,"end":6760}},{"label":"tr","span":{"begin":6761,"end":6794}},{"label":"td","span":{"begin":6761,"end":6763}},{"label":"td","span":{"begin":6764,"end":6769}},{"label":"td","span":{"begin":6770,"end":6775}},{"label":"td","span":{"begin":6776,"end":6781}},{"label":"td","span":{"begin":6782,"end":6787}},{"label":"td","span":{"begin":6788,"end":6794}},{"label":"table-wrap-foot","span":{"begin":6795,"end":6922}},{"label":"p","span":{"begin":6795,"end":6922}},{"label":"figure","span":{"begin":6923,"end":6991}},{"label":"label","span":{"begin":6923,"end":6931}},{"label":"caption","span":{"begin":6932,"end":6991}},{"label":"p","span":{"begin":6932,"end":6991}},{"label":"figure","span":{"begin":6992,"end":7069}},{"label":"label","span":{"begin":6992,"end":7000}},{"label":"caption","span":{"begin":7001,"end":7069}},{"label":"p","span":{"begin":7001,"end":7069}},{"label":"figure","span":{"begin":7070,"end":7127}},{"label":"label","span":{"begin":7070,"end":7079}},{"label":"caption","span":{"begin":7080,"end":7127}},{"label":"p","span":{"begin":7080,"end":7127}},{"label":"title","span":{"begin":7129,"end":7161}},{"label":"figure","span":{"begin":7418,"end":7491}},{"label":"label","span":{"begin":7418,"end":7427}},{"label":"caption","span":{"begin":7428,"end":7491}},{"label":"p","span":{"begin":7428,"end":7491}},{"label":"figure","span":{"begin":7492,"end":7574}},{"label":"label","span":{"begin":7492,"end":7501}},{"label":"caption","span":{"begin":7502,"end":7574}},{"label":"p","span":{"begin":7502,"end":7574}},{"label":"label","span":{"begin":7575,"end":7584}}],"tracks":[]}