2
Statistics in articles published in TRANSFUSION The report by Kanter and Taylor’ in this issue of TRANSFUSION raises important questions about the quality, validity, and accuracy of the statistics used in articles published in the journal. One of the interesting observations of Kanter and Taylor is that in less than half of the articles published in the journal were statistics a sufficiently important part of the manuscript to warrant inclusion in the study. This probably relates to the types of articles that TRANSFUSION publishes, including reviews, controversies, and case reports, as well as origi- nal research and articles on subjects other than strict labo- ratory science. In the opinion of Kanter and Taylor, a substantial pro- portion of articles were characterized by either an error in the statistical description of the data, a failure to de- scribe clearly the statistical tests performed, or the use of an incorrect statistical test or the occurrence of an er- ror in test calculation or interpretation. Of greatest con- cern is their contentionthat 22 percent of the articles using statistics reported a conclusion not supported by the data. Clearly, this is a situation that requires serious attention by the editors of this journal and corrective action where indicated. Is this an indication that the use of statistics is sub- stantially worse among those involved in transfusion medicine research than among those involved in other areas of biomedical research? It has been known for some time that the use of statistics in the biomedical research that is published in medical journals is sometimes not ideal and is subject to the variable concerns of experi- enced statisticians. A number of references in the article by Kanter and Taylor attest to the existence of similar problems in other journals--even those as prestigious as the British Medical Journal, JAMA, and the New England Journal of Medicine-and in other fields of biomedical research such as psychiatry, tropical medicine, arthritis and rheumatism, infectious disease, anesthesiology, or- thopedics, and pathology. Articles in which the statistics were not describedthor- oughly or done exactly properly may not provide the reader with a clear picture of the data, but a failure to use statistics that are technically correct does not neces- sarily invalidate the results of a study. Of greater con- cern are reports of studies in which the conclusions may be based on incorrect or inaccurate statistical analysis. Such a publication reflects badly on the authors and the journal, and in extreme cases could compromise patient safety. The statistical shortcomings described by Kanter and Taylor range from those that might be considered “edi- torial” problems to the statistical equivalent of a poor choice of words to the development of conclusions that are not supported by the data. However, we are not aware of articles that have been published in the journal in which the statistical analysis is so faulty that the authors should be asked to consider retracting the report. Not all statisticians agree on which statistical ap- proaches should be used in different situations or on how those approaches should be applied. Others might even have a different interpretation of the analysis done by Kanter and Taylor. As recently summarized by Boen: “Beware of the statistician who says there is only one right way to design studies or analyze data. There is always more than one good way, and never is there only one right Boen had also elaborated: “Statistical tests are statis- ticians’ best-sellers and their worst products. They are the worst in the sense that they seem as if they meet a need for making decisions in the face of uncer- tainty .... One problem with statistical tests is that statisti- cal significance is too easy to obtain or avoid, depend- ing on the researcher’s purpose and resources.... [Sltatistical significance is just as dependent on extrane- ous factors as on main factors. Variability, sample size, and study design have as much bearing on attainment of statistical significanceas does the amount of e f f e ~ t . ” ~ ( P ~ ~ ~ ) This latter point is also emphasized in the accompa- nying editorial by C ~ n n e t t . ~ The involvement of capable biostatisticians in the early stages of study design and through the process of data acquisition is the most effec- tive way to ensure a high-quality study with effective use of statistics. One factor contributing to these difficulties derives from the unstructured nature of the statistical community, the relatively small number of individuals who can provide effective consultation in biomedical research, and the substantial limitations in what can and cannot be accomplished.2 What does all this mean to transfusion medicine re- search and to the mechanisms used to assess reports sub- mitted to this journal? First of all, we agree with Kanter and Taylor that it would be prudent to give more atten- tion to statistical methodology in all research, including that in transfusion medicine. For years, the editors have discussed the specific mechanisms to be used to evalu- ate the statistical component of manuscripts submitted for publication. It has not been considered practical to send every manuscript to a statistician for review of sta- tistical content. Since very few biostatisticians are in- volved in transfusion medicine research, it has not been possible to find a group who are willing to take on such Way.’’2(~942) 654

Statistics in articles published in TRANSFUSION

Embed Size (px)

Citation preview

Page 1: Statistics in articles published in TRANSFUSION

Statistics in articles published in TRANSFUSION

The report by Kanter and Taylor’ in this issue of TRANSFUSION raises important questions about the quality, validity, and accuracy of the statistics used in articles published in the journal. One of the interesting observations of Kanter and Taylor is that in less than half of the articles published in the journal were statistics a sufficiently important part of the manuscript to warrant inclusion in the study. This probably relates to the types of articles that TRANSFUSION publishes, including reviews, controversies, and case reports, as well as origi- nal research and articles on subjects other than strict labo- ratory science.

In the opinion of Kanter and Taylor, a substantial pro- portion of articles were characterized by either an error in the statistical description of the data, a failure to de- scribe clearly the statistical tests performed, or the use of an incorrect statistical test or the occurrence of an er- ror in test calculation or interpretation. Of greatest con- cern is their contention that 22 percent of the articles using statistics reported a conclusion not supported by the data. Clearly, this is a situation that requires serious attention by the editors of this journal and corrective action where indicated.

Is this an indication that the use of statistics is sub- stantially worse among those involved in transfusion medicine research than among those involved in other areas of biomedical research? It has been known for some time that the use of statistics in the biomedical research that is published in medical journals is sometimes not ideal and is subject to the variable concerns of experi- enced statisticians. A number of references in the article by Kanter and Taylor attest to the existence of similar problems in other journals--even those as prestigious as the British Medical Journal, JAMA, and the New England Journal of Medicine-and in other fields of biomedical research such as psychiatry, tropical medicine, arthritis and rheumatism, infectious disease, anesthesiology, or- thopedics, and pathology.

Articles in which the statistics were not described thor- oughly or done exactly properly may not provide the reader with a clear picture of the data, but a failure to use statistics that are technically correct does not neces- sarily invalidate the results of a study. Of greater con- cern are reports of studies in which the conclusions may be based on incorrect or inaccurate statistical analysis. Such a publication reflects badly on the authors and the journal, and in extreme cases could compromise patient safety.

The statistical shortcomings described by Kanter and Taylor range from those that might be considered “edi-

torial” problems to the statistical equivalent of a poor choice of words to the development of conclusions that are not supported by the data. However, we are not aware of articles that have been published in the journal in which the statistical analysis is so faulty that the authors should be asked to consider retracting the report.

Not all statisticians agree on which statistical ap- proaches should be used in different situations or on how those approaches should be applied. Others might even have a different interpretation of the analysis done by Kanter and Taylor. As recently summarized by Boen: “Beware of the statistician who says there is only one right way to design studies or analyze data. There is always more than one good way, and never is there only one right

Boen had also elaborated: “Statistical tests are statis- ticians’ best-sellers and their worst products. They are the worst in the sense that they seem as if they meet a need for making decisions in the face of uncer- tainty .... One problem with statistical tests is that statisti- cal significance is too easy to obtain or avoid, depend- ing on the researcher’s purpose and resources.... [Sltatistical significance is just as dependent on extrane- ous factors as on main factors. Variability, sample size, and study design have as much bearing on attainment of statistical significance as does the amount of e f f e ~ t . ” ~ ( P ~ ~ ~ )

This latter point is also emphasized in the accompa- nying editorial by C ~ n n e t t . ~ The involvement of capable biostatisticians in the early stages of study design and through the process of data acquisition is the most effec- tive way to ensure a high-quality study with effective use of statistics. One factor contributing to these difficulties derives from the unstructured nature of the statistical community, the relatively small number of individuals who can provide effective consultation in biomedical research, and the substantial limitations in what can and cannot be accomplished.2

What does all this mean to transfusion medicine re- search and to the mechanisms used to assess reports sub- mitted to this journal? First of all, we agree with Kanter and Taylor that it would be prudent to give more atten- tion to statistical methodology in all research, including that in transfusion medicine. For years, the editors have discussed the specific mechanisms to be used to evalu- ate the statistical component of manuscripts submitted for publication. It has not been considered practical to send every manuscript to a statistician for review of sta- tistical content. Since very few biostatisticians are in- volved in transfusion medicine research, it has not been possible to find a group who are willing to take on such

Way.’’2(~942)

654

Page 2: Statistics in articles published in TRANSFUSION

EDITORIAL 655 TRANSFUSION 1994-Vol. 34, No. 8

a substantial task as volunteer reviewers. Alternatively, obtaining statistical reviews from paid consultants would add a substantial cost to the operation of the journal. However, it is neither acceptable nor necessary to settle for poor quality or inaccurate manuscripts in an effort to accommodate the competing demands on the funding for the operation of the journal. As a result of this report,' we are implementing the following additional procedures to attempt to ensure the quality and accuracy of statisti- cal methods for data handling and analysis in articles published in TRANSFUSION

1. The editors will provide increased attention and scrutiny to the statistical issues in manuscripts as the review and decision-making processes are being carried out.

2. The editors will make a more conscious effort to decide early in the review process whether additional statistical review is needed.

3. The forms used by all reviewers will be modified to include a specific question as to whether the reviewer believes a statistical review is necessary. 4. Manuscripts involving the critical use of statistics

will be sent to a biostatistician for an initial recommen- dation as to whether a full statistical review is warranted.

We believe that these additional procedures will en- sure closer scrutiny of the use of statistics in the journal and will be compatible with the resources available for operating the journal. This approach will not delay pub- lication of scientific work but will further assure the read- ers that the statistical procedures used in the journal are appropriate, properly carried out, and accurately de- scribed.

JEFFREY MCCULLOUGH, MD RICHARD H. ASTER, MD

JOSEPH R. BOVE, MD GEORGEGARRATIY, PHD, FIMLS, FRCPATH

PEIERISS~ , PHD, FIMLS, FIBIoL, FRCPATH HAROLD OBERMAN, MD

HERBERT PERKINS, MD Editors

References 1. Kanter MH, Taylor JR. Accuracy of statistical methods in

TRANSFUSION: a review of articles from July/August 1992 through June 1993. Transfusion 1994:34:697-701.

2. Boen JR. What statistics can and cannot do. Transfusion

3. Connett JE. Biostatistical red flags (editorial). Transfusion 1993;33 :940-3.

1994;34:65 1-3.