Multivariate logistic regression analysis
Five published prediction rules were validated (Tables 2 and 3): Diamond & Forrester [22], Pryor et al. [23], Morise et al. 1994 [24], Morise et al. 1997 [17] and Shaw et al. [25]. The Diamond & Forrester prediction rule includes age, sex and type of chest pain, all of which were significant predictors of obstructive CAD in our dataset and with an area under the ROC curve (AUC) of 0.798. Including CTCS increased the AUC to 0.890, which was a statistically significant improvement (p < 0.001). In the expanded model age and sex were no longer significant predictors (Table 2).
Table 2 Comparison of multivariate logistic regression models
Variables Model 1: Diamond & Forrester 1979 Model 2: Pryor et al. 1993 Model 3: Morise et al. 1994
No calcium score Calcium score No calcium score Calcium score No calcium score Calcium score
OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI
ln CTCSa (CT calcium score)  1.97 1.60, 2.41  1.93 1.55, 2.41  1.87 1.52, 2.29
Age  1.05 1.02, 1.08 0.98 0.95, 1.02 1.02 0.95, 1.09 0.93 0.86, 1.01  1.05 1.01, 1.08 0.98 0.95, 1.02
Male sex  2.97 1.57, 5.61 1.64 0.79, 3.42 0.19 0.00, 12.20 0.02 0.00, 1.91  3.48 1.77, 6.83 2.05 0.94, 4.45
Typical chest pain  6.61 3.67, 11.89  5.34 2.70, 10.56  5.44 2.90, 10.18  5.22 2.52, 10.81  5.44 2.94, 10.05  4.91 2.43, 9.91
Smoking 0.08 0.00, 9.40 0.06 0.00, 11.40
Dyslipidaemia 10.90 0.19, 610.99 2.43 0.03, 195.86  3.03 1.63, 5.61 1.94 0.96, 3.94
Diabetes  2.81 1.07, 7.38 2.14 0.71, 6.42  2.61 1.02, 6.73 2.22 0.74, 6.65
Age–smoking 1.04 0.97, 1.13 1.03 0.95, 1.12
Age–dyslipidaemia 0.98 0.92, 1.05 1.00 0.93, 1.07
Sex–smoking 1.77 0.36, 8.84 3.82 0.66, 22.09
Age–sex 1.05 0.98, 1.12 1.07 1.00, 1.15
Oestrogen
Hypertension
Family history
Dyslipidaemia–family history
Obesity
BMI
 
AUCb 0.798 0.742, 0.854 0.890 0.851, 0.930 0.838 0.789, 0.887 0.901 0.863, 0.938 0.831 0.780, 0.881 0.899 0.861, 0.937
LR testc  p < 0.001  p < 0.001  p < 0.001
Odds ratios (ORs) in bold typeface are statistically significant
aNatural logarithm of CTCS + 1
bArea under the receiver operating characteristic curve
cLikelihood ratio test comparing model without CTCS and model including CTCS
Table 3 Comparison of multivariate logistic regression models
Variables Model 4: Morise et al. 1997 Model 5: Shaw et al. 1998
No calcium score Calcium score No calcium score Calcium score
OR 95% CI OR 95% CI OR 95% CI OR 95% CI
ln CTCSa (CT calcium score)  1.87 1.50, 2.31  1.86 1.51, 2.30
Intercept
Age  1.05 1.01, 1.09 0.99 0.95, 1.03  1.05 1.02, 1.08 0.98 0.95, 1.02
Male sex  3.15 1.29, 7.70 1.37 0.48, 3.95  3.42 1.74, 6.74 2.05 0.94, 4.45
Typical pain  5.56 2.94, 10.51  4.82 2.36, 9.86  5.50 2.96, 10.21  4.91 2.44, 9.90
Smoking 1.53 0.72, 3.24 1.03 0.44, 2.39 1.63 0.80, 3.30 1.04 0.47, 2.27
Dyslipidaemia  3.20 1.29, 7.95 1.80 0.62, 5.22  3.04 1.63, 5.66 1.95 0.96, 3.94
Diabetes 2.66 0.98, 7.26 2.01 0.61, 6.61  2.85 1.10, 7.39 2.24 0.74, 6.75
Age–smoking
Age–dyslipidaemia
Sex–smoking
Age–sex
Oestrogen 0.78 0.33, 1.86 0.53 0.19, 1.48
Hypertension 1.83 0.93, 3.60 1.31 0.62, 2.79
Family History 2.02 0.77, 5.29 1.14 0.38, 3.39
Dyslipidaemia–family history 0.74 0.21, 2.60 1.15 0.28, 4.82
Obesity 0.88 0.31, 2.48 0.65 0.20, 2.12
BMI 0.99 0.88, 1.13 1.05 0.91, 1.22
 
AUCb 0.840 0.792, 0.889 0.898 0.859, 0.936 0.833 0.783, 0.883 0.899 0.861, 0.937
LR testc  p < 0.001  p < 0.001
Odds ratios (ORs) in bold typeface are statistically significant
aNatural logarithm of CTCS + 1
bArea under the receiver operating characteristic curve
cLikelihood ratio test comparing model without CTCS and model including CTCS
Pryor et al. analysed age, sex, type of chest pain, smoking, dyslipidaemia, diabetes and the interaction between age and smoking, age and dyslipidaemia, sex and smoking, and age and sex, of which type of chest pain and the presence of diabetes were significant predictors. This model resulted in an AUC of 0.838. After including CTCS, the AUC increased to 0.901 which was a statistically significant improvement (p < 0.001). In the expanded model diabetes was no longer a significant predictor (Table 2).
Morise et al. (1994) included diabetes and dyslipidaemia in addition to the variables used by Diamond & Forrester and resulted in an AUC of 0.831. All variables were significant predictors of the presence of obstructive CAD. After including CTCS, type of chest pain was the only variable that remained significant. After inclusion of CTCS the AUC increased to 0.899, which was a statistically significant improvement (p < 0.001) (Table 2).
Morise et al. (1997) assessed age, sex, type of chest pain, smoking, dyslipidaemia, diabetes, oestrogen status, hypertension, family history, obesity, BMI and the interaction between dyslipidaemia and family history. This model resulted in an AUC of 0.840. Age, sex, type of chest pain and dyslipidaemia were significant predictors. After including CTCS, the AUC increased to 0.898, which was a significant model improvement (p < 0.001). Age, sex and dyslipidaemia were no longer significant predictors after the addition of CTCS (Table 3).
Shaw et al. considered age, sex, typical chest pain, smoking, dyslipidaemia and diabetes and resulted in an AUC of 0.833. After including CTCS, only type of chest pain remained a significant predictor and the AUC increased to 0.899 which was a statistically significant improvement (p < 0.001) (Table 3).
Reclassification tables for the Diamond & Forrester model and the Pryor model are presented in Tables 4 and 5. The addition of CTCS to Diamond & Forrester resulted in reclassification of 47.2% of patients of whom 73.3% were correctly reclassified. The reclassification calibration statistic (RCS) indicated a strong lack of fit for the Diamond & Forrester model (p < 0.00001) which decreased substantially when CTCS was added to the model (p < 0.01). The NRI (net reclassification improvement) was 33.6% (p < 0.0001) and the IDI (integrated discrimination improvement) was also statistically significant (18.8%, p < 0.001) indicating improvement in the classification of cases and non-cases in probability categories and improvement in discrimination between cases and non-cases.
Table 4 Probability of coronary artery disease: reclassification table after addition of CTCS to the Diamond and Forrester (model 1)
Probability category based on model 1 Probability category based on model 1 + CTCS Total, n (%)
<30% ≥30–50% ≥50–70% ≥70%
<30%
 N (%) 51 (67.1) 19 (25.0%) 4 (5.2) 2 (2.6) 76 (29.9)
 Observed probability, % 11.8 36.8 50.0 100.0 22.4
≥30–50%
 N (%) 25 (39.7) 13 (20.6) 19 (30.2) 6 (9.5) 63 (28.4)
 Observed probability, % 0.0 23.1 68.4 50.0 30.2
≥50–70%
 N (%) 4 (11.1) 5 (13.9) 6 (16.7) 21 (58.3) 36 (14.2)
 Observed probability, % 25 80.0 50.0 66.7 61.1
≥70%
 N (%) 5 (6.3) 2 (2.5) 8 (10.1) 64 (81.0) 79 (31.1)
 Observed probability, % 0.0 50.0 37.5 95.3 82.3
 
Total
 N (%) 85 (33.5) 39 (15.4) 37 (14.6) 93 (36.6) 254 (100.0)
 Observed probability, % 8.2 38.5 56.8 86.0 48.4
CTCS computed tomography coronary calcium score
Table 5 Probability of coronary artery disease: reclassification table after addition of CTCS to the model published by Pryor et al. (model 2)
Probability category based on model 2 Probability category based on model 2 + CTCS Total, n (%)
<30% ≥30–50% ≥50–70% ≥70%
<30%
 N (%) 76 (85.4) 10 (11.2) 2 (2.3) 1 (1.1) 89 (35.0)
 Observed probability, % 9.2 40.0 100.0 100.0 15.7
≥30–50%
 N (%) 10 (27.0) 10 (27.0) 12 (32.4) 5 (13.5) 37 (14.6)
 Observed probability, % 10.0 20.0 50.0 100.0 37.8
≥50–70%
 N (%) 7 (13.5) 6 (11.5) 15 (28.9) 24 (46.2) 52 (20.5)
 Observed probability, % 0.0 50.0 73.3 66.7 57.7
≥70%
 N (%) 3 (4.0) 3 (4.0) 9 (11.8) 61 (80.3) 76 (29.9)
 Observed probability, % 33.3 0.0 66.7 95.0 85.5
 
Total
 N (%) 96 (37.8) 29 (11.4) 38 (15.0) 91 (35.8) 254 (100.0)
 Observed probability, % 9.4 31.0 65.8 87.9 48.4
CTCS computed tomography coronary calcium score
For the model by Pryor et al., 36.2% of the patients were reclassified, of whom 54.3% were correctly classified. The RCS indicated a lack of fit (p = 0.01), which decreased when CTCS was added to the model (p = 0.03). The NRI was 24.0% (p < 0.0001) and the IDI was 14.8% (p < 0.001).
The reclassification measures for all models are presented in Table 6.
Table 6 Reclassification measures obtained by adding CTCS to the existing prediction models
Model Reclassification percentages % Correcta Chi-squared excluding CTCSb  p value Chi-squared including CTCSc  p value NRI, %  p value for NRI Reclassification improvement (cases)d Reclassification improvement (non-cases)d IDI, %  p value for IDI
Overall Reclassified from ≥30–50% Reclassified from ≥50–70%
1 47.2 79.4 83.3 73.3 25.01 <0.00001 10.91 <0.01 33.6 <0.0001 <0.0001 0.23 18.8 <0.001
2 36.2 73.0 71.2 54.3 6.57 0.01 4.70 0.03 24.0 0.001 <0.001 0.31 14.8 <0.001
3 38.2 77.5 75.5 63.9 4.14 0.04 3.31 0.07 21.8 0.005 <0.001 0.58 14.9 <0.001
4 34.3 74.4 74.0 83.9 3.82 0.05 1.58 0.21 24.9 <0.001 <0.001 0.29 13.3 <0.001
5 37.8 77.5 75.9 69.8 3.65 0.06 5.14 0.02 22.6 0.003 <0.001 0.58 14.4 <0.001
CTCS computed tomography coronary calcium score, NRI net reclassification improvement, IDI integrated discrimination improvement
aIf the predicted probability of obstructive CAD of the model including CTCS was closer to the observed probability of CAD compared with the prediction of the model without CTCS, the reclassification was considered to be correct
bReclassification calibration (Hosmer–Lemeshow) statistic for model without CTCS, using cells from the reclassification table with at least 20 observations
cReclassification calibration statistic for models including CTCS, using cells from the reclassification table with at least 20 observations
dThe reclassification improvement is defined as the difference in proportions of patients moving up and down for cases and non-cases separately