Article Access Statistics   Viewed  594   Printed  5   Emailed  0   PDF Downloaded  64   Comments  [Add]  

Click on image for details.



Year : 2016
 Volume
: 19  Issue : 1  Page
: 144145 

Some clarifications regarding multiple comparisons 

Andrew V Frane
Department of Psychology, University of California Los Angeles, Los Angeles, CA 90095 1563, USA
Click here for correspondence address and
email
Date of Web Publication  31Dec2015 




How to cite this article: Frane AV. Some clarifications regarding multiple comparisons. Ann Card Anaesth 2016;19:1445 
The Editor,
Barkan ^{[1]} reviewed some basic statistical concepts used in research and rightly called for increased statistical literacy among clinicians. The study is largely correct but contains several statements that may foster more confusion than clarityparticularly on the topic of multiple comparisons.
For example, it was claimed that if each of 20 comparisons is tested at the 0.05 level, then "there's a virtual certainty that at least one will yield a false positive result." On the contrary, as noted in Bland and Altman, ^{[2]} which was apparently misinterpreted in Barkan, ^{[1]} the maximum probability of at least one false positive in the given scenario is approximately 0.64not "virtual certainty" by any reasonable definition. Moreover, the probability is even lower if the test statistics are positively correlated and/or if any null hypotheses are false. Barkan may have confused the familywise Type I error rate (FWER; the probability of obtaining at least one false positive in a given set of comparisons) with the perfamily Type I error rate (PFER; the expected mean number of false positives per set, as a theoretical longterm average). Indeed, the maximum PFER in the given scenario is 1, but that by no means indicates 100% chance of error.
It was also claimed that although the Bonferroni procedure is considered conservative, "Equally rigorous but less stringent techniques such as the false detection (sic) rate are now in use." However, the false discovery rate (FDR) is not a "technique" in itself, and techniques for controlling FDR are considerably less rigorous than the Bonferroni procedure (which controls the PFER). In general, one adjustment cannot simultaneously be both "equally rigorous" and "less stringent" relative to another. The more stringent the thresholds for significance the more rigorous the control of Type I errors (false positives). PFER control is more rigorous/stringent than FWER control, and FWER control is more rigorous/stringent than FDR control. Thus, controlling PFER always controls FWER and controlling FWER always controls FDR, but not vice versa.
It was claimed that all adjustments for multiple comparisons "adjust individual comparison thresholds, so the final statistical significance for all comparisons combined is P < 0.05." However, although the Bonferroni procedure controls the sum of probabilities of Type I error "for all comparisons combined," FDR control does not provide that protection. ^{[3]} Note that different methods of adjustment address different error rates, and which error rate is relevant depends on the strength of inference that is required. ^{[4]} For instance, procedures that control only FDR and not FWER are generally not adequate for confirmatory clinical trials. ^{[5]}
It is also notable that although advantages of using confidence intervals rather than P values alone were correctly noted in Barkan, ^{[1]} simultaneous confidence intervals for multiple comparisons were not mentioned. Contrary to a common misconception, the issue of multiple comparisons is just as relevant to confidence intervals as to P values. ^{[4]}
Due to space constraints, additional errors and omissions in Barkan ^{[1]} are not addressed in this letter. Suffice to say, improving statistical literacy (e.g., on the topic of multiple comparisons) is certainly a worthy goal. Instructive publications that are clear, simple, and correct are essential to that goal.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References   
1.  Barkan H. Statistics in clinical research: Important considerations. Ann Card Anaesth 2015;18:7482. [ PUBMED] 
2.  Bland JM, Altman DG. Multiple significance tests: The Bonferroni method. BMJ 1995;310:170. 
3.  Finner H, Roters M. On the false discovery rate and expected Type I errors. Biom J 2001;43:9851005. 
4.  Benjamini Y. Simultaneous and selective inference: Current successes and future challenges. Biom J 2010;52:70821. 
5.  Dmitrienko A, Bretz F, Westfall PH, Troendle J, Wiens BL, Tamhane AC, et al. Multiple testing methodology. In: Dmitrienko A, Tamhane AC, Bretz F, editors. Multiple Testing Problems in Pharmaceutical Statistics. Boca Raton, FL: Chapman and Hall; 2010. p. 3598. 
Correspondence Address: Andrew V Frane Department of Psychology, University of California Los Angeles, 7351 Franz Hall, Los Angeles, CA 90095 1563 USA
Source of Support: None, Conflict of Interest: None  Check 
DOI: 10.4103/09719784.173033








