PMC:4979053 / 19306-34063 JSONTXT

{"project":"2_test","denotations":[{"id":"27600350-12538238-69476975","span":{"begin":5990,"end":5992},"obj":"12538238"}],"text":"3. Results and Discussion\n\n3.1. Data Description\nAnalyses involved two samples (A498, a tumor line, and hREF, a pool of healthy tissues), each with three technical replicates for all platforms considered (Affymetrix GeneChip© miRNA Array, Agilent Human miRNA Microarray (V1) and Illumina humanMI_V2). MiRNA selection described in the Experimental Section, resulted in a total of 813 human miRNAs considered for analysis, which account for 95.99% of human miRNAs on Affymetrix platforms, 95.53% on Agilent and 94.76% on Illumina (see Figure 1). Pairwise intersections of human miRNA lists revealed that the larger overlap occurred when Affymetrix and Agilent were considered (830 miRNAs, 97.99% of Affymetrix hsaand 97.53% of Agilent hsa), whereas Illumina showed a slightly poorer degree of overlap with both Affymetrix (817 miRNAs, 96.46% of Affymetrix and 95.22% of Illumina) and Agilent (815 miRNAs, 95.77% of Agilent and 94.99% of Illumina).\nFigure 1 Venn diagram of common human miRNAs across platforms. Density plots and boxplots for both samples in Figure 2 showed that the distribution of intensity values of common miRNAs was very different between platforms, with a marked skewness for Affymetrix and Agilent (red and green) and peaks of density for relatively low values. The Illumina platform was characterized by a slightly more symmetrical behavior. Notably, the first replicate of sample A498 had some technical problems for both Affymetrix and Agilent (solid red and green lines, lower right panel), whereas no similar pattern was seen for Illumina. In fact, the quantile distribution appeared to be quite different between technical replicates for these two platforms, despite similar inter-quartile ranges (see Table 1), suggesting that systematic bias had occurred in the profiling of the sample. The same plots were produced for normalized data and showed quite a different distribution, at least in terms of location, in particular for Illumina platform, whose right tail was relevantly reduced (see Supplementary Material, Figures S1 and S2).\nmicroarrays-03-00302-t001_Table 1 Table 1 Quartiles and inter-quartile range for the three platforms.\nFigure 2 Box and density plots for both samples. The left column refers to hREF and the right column to A498. Plots refer to non-normalized log2-transformed data. (Lower panels) Solid lines represent the technical replicate labeled as 1 in the datasets, whereas dashed lines and dotted lines represent technical Replicate 2 and 3, respectively.\n\n3.2. Intra-Platform Reliability\nResults for the pairwise CCCs and the overall CCC for each sample and platform using common human miRNAs are reported in Table 2. Globally, the hREF sample showed better features of reliability, in particular for the Agilent and Illumina platform (OCCC: 0.994 (CI 95%: 0.993–0.995) and 0.994 (CI 95%: 0.994–0.995), respectively). On the other hand, the A498 sample showed lower values of OCCC, which dramatically decreased when considering the Affymetrix platform (0.927, CI 95%: 0.906–0.941). According to the non-inferiority threshold of 0.96 defined in the Experimental Section, Agilent and Illumina could be consistently declared repeatable across the two samples considered, whereas Affymetrix resulted in repeatable only for sample hREF.\nAs a means of comparison to previous studies, these results were obtained also on data filtered using the detection call approach, i.e., by removing all of the miRNAs that were called “present” on no more than two samples (see Table 3). This information was available only for Agilent and Illumina and led to a reduced dataset consisting of 347 human miRNAs. It can be noticed that estimates were mostly unchanged for the Illumina platform, though with a slight improvement after data filtering, whereas for Agilent, the pattern was not clear, though the differences appeared to be larger for A498, where filtering led to some relevant improvements, both in the pair-wise CCC and in the OCCC.\nmicroarrays-03-00302-t002_Table 2 Table 2 CCC and Overall CCC with bootstrap 95%CI (common miRNAs) CCCs and the OCCC were computed on the 813 miRNAs common across all platforms considered for the study, and bootstrap 95%CI for the Overall Concordance Correlation Coefficient were computed using 1000 bootstrap samples and the percentile method [32].\nmicroarrays-03-00302-t003_Table 3 Table 3 Concordance correlation coefficient (CCC) and overall CCC with bootstrap 95% CI (detection call filtering). CCCs and the OCCC were computed on the 347 miRNAs common across Illumina and Agilent platforms after detection call filtering (present on at least three samples), and bootstrap 95% CI for the overall CCC were computed using 1000 bootstrap samples and the percentile method [32]. CCCs and OCCC using all human miRNAs and all miRNAs (human and non-human) were also evaluated on each platform (see Supplementary Material, Tables S1 and S2). When considering all of the human miRNAs on each platform (847 for Affymetrix, 851 for Agilent and 858 for Illumina; Table S1), results were almost identical to the situation with only common miRNAs used, the only difference lying in the Affymetrix platform for sample A498, which showed an OCCC of 0.93 (CI 95%: 0.911–0.943), a slightly higher value than those seen for common miRNAs. Using all of the miRNAs measured on the platform (7815 for Affymetrix, 921 for Agilent and 1145 for Illumina) returned values that confirmed patterns of agreement already depicted in Table 2 and Table S1, with the exception of line A498 for Affymetrix, which showed an increase, both in point and interval estimates for the OCCC (0.954, CI 95%: 0.950–0.957, see Table S2). The Agilent platform on line A498, on the contrary, showed similar patterns of moderate-to-poor agreement throughout all miRNA selection strategies.\nAdditionally, the normalization effect was evaluated by estimating CCC and overall CCC on data normalized according to quantile and loess normalization [38], which resulted in a general increase of both pairwise and overall concordance for both samples and for all platforms. In particular, also sample A498 profiled with Affymetrix showed good patterns of repeatability, with its CI 95% lying above the threshold of 0.96. Detailed results were reported in Tables S3 and S4 of the Supplementary Material.\n\n3.3. Between-Platform Agreement\nTo perform agreement evaluation, miRNA intensities were averaged across technical replicates for each array and both samples. Then, pairwise array agreements were evaluated in terms of miRNA lying within the modified agreement interval described in the Experimental Section. Estimates of the measurement error model for error-variance ratio λ equal to one, presented in Table 4, show that the relationship between Agilent and Illumina was the one that is closest to the agreement line with intercept zero and slope one for both samples. On the other hand, models that include the Affymetrix platform for line A498 showed a very negatively large intercept (−12.4128 and −17.4064), which possibly reflected the technical bias already highlighted in the previous section. However, if line hREF is considered, Affymetrix was confirmed to be the array deviating most from the line of perfect agreement with both Illumina and Agilent, whereas these two showed patterns very close to concordance (slope = 1.0925, CI 95%: 1.0371–1.1479). Since the confidence intervals for the intercept and the slope suggested an intercept different from zero and a slope different from one for all comparisons in both samples, the agreement intervals following Formula 11 were built.\nmicroarrays-03-00302-t004_Table 4 Table 4 Estimates of the linear measurement error model, λ = 1. In Figure 3 and Figure 4, graphical results for lines A498 and hREF, respectively, were reported. When a value k of tolerance equal to 0.05×813 ≃ 41 was set, thus considering the platforms to be in agreement if no more than k measurements lay outside these intervals, the pair Illumina-Agilent was the only one that could be said to be in agreement for both samples. In fact, 97.79% (CI 95%: 96.52–98.68) of miRNAs for hREF and 95.45% (CI 95%: 93.78–96.78) of miRNAs for A498 were concordant between the two platforms (795 and 776 miRNAs in agreement for cell lines hREF and A498, respectively). On the other hand, the remaining pairs showed poor patterns of concordance. The Affymetrix platform on line A498 was poorly concordant with both Agilent and Affymetrix, possibly because of the issues previously described, but also for line hREF, it showed a poor degree of agreement. In particular, only 82.53% (CI 95%: 79.75–85.08) and 82.78% (CI 95%: 80.01–85.31) of miRNAs, corresponding to 671 and 673, were found to lie within the agreement interval for comparison to Agilent and Illumina, respectively.\nFigure 3 Agreement intervals for line hREF, λ=1. (A) The comparison Agilent-Affymetrix; (B) the comparison Illumina-Affymetrix; and (C) Illumina-Agilent. The samples/miRNAS have been plotted in ascending order according to their value on the second platform in the y-axis label, so that the x-axis only contains a progressive value from one to 813 according to such ordering.\nFigure 4 Agreement intervals for line A498, λ=1. (A) The comparison Agilent-Affymetrix; (B) the comparison Illumina-Affymetrix; and (C) Illumina-Agilent. The samples/miRNAs have been plotted in ascending order according to their value on the second platform in the y-axis label, so that the x-axis only contains a progressive value from one to 813 according to such ordering. These results relied on the assumption that the ratio of the error variances from the two methods being compared was equal to one. The main advantage of this assumption was that previous knowledge of the error variances was not required; however, it was likely to be violated when the methods to be compared had very different analytical properties, such as microarray platforms (see, for instance, Figure 2). The technical replicates were used to fit a random effects model for each combination of cell line and array, estimated the error variance as the residual error of the model itself and computed the parameter λ as the ratio of the error variance of Y and X (see Table 5). The confidence intervals for λ did not include one in any comparison, thus suggesting a different measurement error for each platform.\nmicroarrays-03-00302-t005_Table 5 Table 5 Estimates of λ and CI 95%. Values were obtained as the ratio of σϵ2 (error variance of Y) and σδ2 (error variance of X), estimated via random effects models. Model parameters estimates when λ is estimated according to a random effects model are reported in Table 6. The estimates of the slopes and of the intercepts suggested that also in this case, the interval to be preferred should be the one described in Formula 11.\nmicroarrays-03-00302-t006_Table 6 Table 6 Estimates of the linear measurement error model, λ estimated. Illumina and Agilent showed the best patterns of concordance for both samples, resulting in a percentage of miRNAs in agreement exceeding the 95% threshold previously discussed: 97.54% (CI 95%: 96.23–98.49) for hREF and 96.3% (CI 95%: 94.77–97.50) for A498, corresponding to 793 and 783 miRNA (see Figure 5 and Figure 6).\nIn general, estimating the value of λ led to an increased number of miRNAs within the determined agreement interval, in particular for the Affymetrix-related comparisons (see Table 7).\nPerforming the same analysis on detection call filtered data (thus, only on 347 miRNAs and comparing Agilent and Illumina platforms only) led to similar results, though with some improvement for line A498. In particular, if λ was set to one, the proportion of concordant miRNAs was equal to 97.69% (CI 95%: 95.51–99.00) for line hREF and to 98.56% (CI 95%: 96.67–99.53) for line A498. Estimating the error variance ratio (0.629, CI 95%: 0.557–0.710 for hREF; 0943, CI 95%: 0.835–1.065 for A498) led to slightly improved results, the proportion of concordant miRNAs being equal to 99.14% for hREF (CI 95%: 96.67–99.53) and to 98.56% (CI 95%: 97.49–99.82) for A498.\nFigure 5 Agreement intervals for line hREF, λ estimated. (A) The comparison Agilent-Affymetrix; (B) the comparison Illumina-Affymetrix; and (C) Illumina-Agilent. The samples/miRNAs have been plotted in ascending order according to their value on the second platform in the y-axis label, so that the x-axis only contains a progressive value from one to 813 according to such ordering.\nFigure 6 Agreement intervals for line A498, λ estimated. (A) The comparison Agilent-Affymetrix; (B) the comparison Illumina-Affymetrix; and (C) Illumina-Agilent. The samples/miRNAs have been plotted in ascending order according to their value on the second platform in the y-axis label, so that the x-axis only contains a progressive value from one to 813 according to such ordering. The same analysis was performed also on quantile and loess normalized data, with contrasting results. For sample A498, there was a general increase in the performance of all platforms, yet only Agilent and Illumina could be considered to be concordant (net of bias correction), whereas for sample hREF, there was a general decrease, at least when λ = 1 was considered, in the proportion of concordant miRNAs. This difference was substantially relevant when Affymetrix and Illumina were compared (see Tables S8 and S12 in the Supplementary Material). The fact that differences between normalized and un-normalized data were more relevant when λ was assumed to be one could be due to the effect of the normalization procedure on the error variance ratio: in particular, when normalization is performed, there is a use of information carried by the data that can lead to a reduction in the residual error variance and in the ratio λ and, eventually, to more stable results and, thus, more concordant measurements. On the other hand, differences between platforms are already taken into account when λ is estimated, so that a normalization procedure could only limitedly improve results.\nmicroarrays-03-00302-t007_Table 7 Table 7 miRNA in agreement between arrays. Number (n) and proportion (%) of miRNAs lying in the different agreement intervals, estimated according to the measurement error model parameters estimated by setting λ = 1 and by estimating it via random effects models. Confidence intervals for the proportions were computed using the Clopper–Pearson exact method [39]. †: the platform pair is in agreement. Results for model parameter and λ estimation on normalized data are available in the Supplementary Material, from Tables S5 to S7 for quantile normalization and from S9 to S11 for loess normalization.\n"}