Discussion
We found relatively few systematic reviews that focused on evaluating CAM interventions in a pediatric population. This contrasts sharply with the 1468 PedCAM randomized trials known to exist [17,18]. This suggests that the pediatric CAM community have not synthesized all of the available evidence. We found information to support this contention. In our efforts to identify PedCAM randomized trials we identified 36 studies that evaluated the efficacy and/or effectiveness of CAM interventions, such as hypnosis, for the management of pain in a variety of settings. However, we were unable to identify any systematic reviews that had pooled this evidence.
In 1989 Crowley and colleagues [19] published a systematic review evaluating the benefits of corticosteroids (versus placebo) in reducing mortality and morbidity (e.g., respiratory distress syndrome) in expectant mothers of premature infants. Their review showed that corticosteroids were significantly more effective in reducing mortality and morbidity. These researchers also noted that this evidence was available 10 years earlier (i.e., 1979) had it been synthesized. Because of the delay in pooling this evidence it is likely that some children suffered unnecessarily. The results of this review were subsequently used as the logo of the Cochrane Collaboration.
Our results indicate that the quality of reporting of the PedCAM systematic reviews, and their conventional medicine comparators, is similar and less than optimal, with considerable room for improvement. The quality of reports of systematic reviews can influence reviewers' conclusions concerning the effectiveness of an intervention. After examining 51 systematic reviews on the effectiveness of spinal manipulation reviewers' were more likely to judge the intervention as positive if the report was of high quality [20].
Resources exist to help ensure that appropriate methods are used to conduct and report systematic reviews [21-24]. Another way to improve the quality of reporting of PedCAM systematic reviews is for more pediatric journals to endorse the QUOROM statement [25]. A multi journal evaluation of QUOROM has recently been completed and the results are being written up currently. Examining the quality of reporting is 'after the fact' when the review is already completed. QUOROM can also be used by granting agencies to encourage prospective systematic reviewers to improve the conduct of their reviews. This is already starting to happen in the conduct and reporting of randomized controlled trials [26].
Conducting and reporting systematic reviews with the highest possible quality is likely to minimize the possibility that their results are influenced by bias, enabling clinicians to be more confident of using them in their practice. Reports of PedCAM systematic reviews seem particularly weak in terms of the comprehensiveness in their search to identify primary studies. For example, excluding a report identified uniquely in Embase, compared to its inclusion, can statistically exaggerate the estimates of an intervention's effectiveness by 6%, on average [13].
Systematic reviewers of the CAM literature appear to be more conscious of the consequences excluding primary studies published in languages other than English. Excluding CAM trial reports in languages other than English, compared to their inclusion, is likely to exaggerate the estimates of an intervention's effectiveness by 37%, on average [27]. This result is interesting from two perspectives: it is contrasts that found when examining the conventional medicine interventions (i.e., no effect when excluding reports of trials in languages other than English), and most of the methodological research to date has focused on the impact of bias within conventional interventions. There is an important need to develop a research agenda that focuses specifically on the impact of bias when pooling CAM interventions.
This study had several limitations. Our focus was on the quality of reporting of PedCAM systematic reviews. It is possible that the reviews were appropriately conducted but had deficiencies in their reporting. Despite the paucity of data addressing this important question, the evidence that is available points in the direction of a reasonably good correlation between how investigators conduct their research and how it is subsequently reported [28], [29]. We only selected a sample of CAM studies to compare to reviews of conventional interventions. It is possible that our sample is not representative and that our results cannot be generalized to all PedCAM systematic reviews. We selected the reports to enable comparators investigating the same disease. Given that the quality of reporting of the sample is very similar to the quality of all 47 reviews we believe that our sampling approach is representative and enables us to generalize the observed results.
Evidence based health care continues to make important contributions to the well being of children. To ensure the pediatric community can maximize the potential use of interventions it is important to ensure that systematic reviews are conducted and reported at the highest possible quality. Such reviews will be of benefit to a broad spectrum of interested stakeholders.