Model development and validation
The prediction models based on (i) clinical features (C model), (ii) radiological features (R model), and (iii) the combination of clinical features and radiological features (CR model) were developed. ROC analyses for the primary and validation cohort are shown in Table 4 and Fig. 3. The CR model yielded a maximum AUC of 0.986 (95% CI 0.966~1.000) in the primary cohort with the highest accuracy and specificity, which was 0.936 (95% CI 0.866~1.000) in the validation cohort. The AUC for the C model was 0.952 (95% CI 0.988~0.915) and 0.967 (95% CI 0.919~1.000) in the primary and validation cohorts, respectively. For the R model, the AUC of the two cohorts was 0.969 (95% CI 0.940~0.997) and 0.809 (95% CI 0.669~0.948), respectively.
Table 4 Performance of the individualized prediction models
Primary cohort (n = 98) Validation cohort (n = 38)
Models AUC 95% CI Accuracy Specificity Sensitivity AUC 95% CI Accuracy Specificity Sensitivity
C model 0.952 0.915~0.988 0.888 0.894 0.882 0.967 0.919~1.000 0.868 0.859 0.842
R model 0.969 0.940~0.997 0.929 0.851 1.000 0.809 0.669~0.948 0.684 0.368 1.000
CR model 0.986 0.966~1.000 0.959 0.957 0.961 0.936 0.866~1.000 0.763 0.789 0.737
C, R, and CR indicate the predicted model based on clinical features, radiological features, and the combination of clinical features and clinical radiological features, respectively. CI confidence interval
Fig. 3 ROC of the three models in primary and validation cohort curves. Comparison of receiver operating characteristic (ROC) curves among the radiological mode (R model), clinical model (C model), and the combination of clinical and radiological model (CR model) for the diagnosis of COVID-19 in the primary (a) and validation (b) cohorts
To determine the clinical usefulness of the diagnostic model, we developed the decision curve (Fig. 4), which showed better performances for the CR model compared with that for the C model and the R model. Across the majority of the range of reasonable threshold probabilities, the decision curve analysis showed that the CR model had a higher overall benefit than the C model and R model.
Fig. 4 Decision curve analysis for each model in the primary dataset. The y-axis measures the net benefit, which is calculated by summing the benefits (true-positive findings) and subtracting the harms (false-positive findings), weighting the latter by a factor related to the relative harm of undetected metastasis compared with the harm of unnecessary treatment. The decision curve shows that if the threshold probability is over 10%, the application of the combination of clinical and radiological model (CR model) to diagnose COVID-19 adds more benefit than the clinical model (C model) and radiological model (R model)
The nomogram (Fig. 5) was developed by the CR model in the primary cohort, with the factors of the total number of mixed GGO in peripheral area (TN_Mixed_GGO_IP), tree-in-bud, offending vessel augmentation in lesions (OVAIL), respiration, heart ratio, temperature, white blood cell count, cough, fatigue and lymphocyte count category incorporated. The total points were calculated by summing the points identified on the “points” scale for each factor. By comparing the “total points” scale and the “probability” scale, the individual probability of COVID-19 infection could be obtained.
Fig. 5 Nomogram of the CR model in the primary cohort. TN_Mixed_GGO_IP represented the total number of mixed GGO in peripheral area. AVAIL represented offending vessel segmentation in lesions. N was a negative result, and P was a positive result. Norm represented normal. Note that in probability scale, 0 = non-COVID-19, 1 = COVID-19