| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
EDITORIAL |
Australia
"Moreover, inadequate reporting borders on unethical practice when biased results receive false credibility" The CONSORT Statement 2001.1
A publication in a peer-reviewed journal has many implications. The reader will assume that there has been full editorial supervision at all stages of the submission and acceptance, that reviewers who understand the subject and the nuances of the study have been selected, that data collection is adequate and appropriate, and importantly in this day of increasingly complex statistical tests (performed using small computers) that the design and analysis is appropriate and can be followed, that the study has been conducted appropriately, results have been described accurately, and the discussion takes into account existing evidence. Voluntary unpaid reviewers, often with little support or training, will do many of these tasks. The authors will make similar assumptions and consider after the process is complete that the report meets the standards of correctness (i.e. the truth).
It is also important to acknowledge the potential variations among readers. A physician or researcher with detailed knowledge in the area will view a paper in a different light to someone seeking knowledge about the subject or wishing to use similar methodology. A lay person with little knowledge may seize upon the results summarized in the abstract, which should accurately reflect and summarize, in the manner of an executive statement, the key findings and conclusions set out in the body of the manuscript, and take them as true. Reports of trials may have important financial implications for drug and device companies. Is this process adequate for this day and age? Is too much pressure placed on peer reviewers to determine whether the ‘truth’ has been determined? Much has been written regarding the peer-review process in the past. What if there is a failure or inadequacy at this stage, might this not encourage or permit inadequate reporting? At the editorial stage, who controls these gatekeepers?
Much of the review process is done in a confidential manner, some would say secretive. This may hide inadequacies of the reviewer who may have no training in this important process. Some journals provide guidelines for reviewing papers, but little support for the cognitive process is available. Editorial decisions are frequently passed back to the reviewers, but little feedback on the adequacy of the review is given. All of this involves time and effort, lacking often at both the editorial level and at the reviewer level. It is difficult to berate an untrained and unpaid volunteer.
The International Committee of Medical Journal Editors provides guidelines for authors (http://www.icmje.org), but few words for reviewers. Several reporting guidelines have been published, including CONSORT (http://www.consort-statement.org) and STROBE (http://www.strobe-statement.org). The CONSORT statement provides a useful basis for reporting randomized trials and has been acknowledged by almost all major journals. Not only should randomized controlled studies be viewed in this manner, but those who report other trials can reasonably use the checklist to ensure the credibility of their study. Other recommendations for meta-analyses and observational studies have followed.
How can critical review be improved? A reviewer has deductive powers and communication skills. Kassirer (a former editor of the New England Journal of Medicine, who resigned citing interference with editorial freedom) and Campion2 (Senior Deputy Editor of the New England Journal of Medicine) commented in 1994 on the process as they saw it. The first aspect was the detection of flaws such as bias, design problems, and performance issues. After assessment of all these points, each reviewer is likely to have a rejection threshold: a point at which all the positives in favor of publication are outweighed by the negatives. They suggested that a reviewer assessment might have false positives or false negatives, leading to incorrect recommendations.
To study what might go wrong, let us take a recent publication in a prestigious medical journal, the New England Journal of Medicine. In the January 26 edition, a paper appeared that was authored by a notable anesthetist and lead or co-author of several papers in the same journal on related subjects.3 Several articles in non-medical publications, the Wall Street Journal and the Washington Post, interestingly accompanied it. The report analyzed a registry of over 5,000 patients who were enrolled from 1996 to 2000. It used several complex methods of statistical analysis and concluded that aprotinin, in use for over 20 years in the treatment of mainly high-risk cardiac surgery patients, should be removed from the shelves. In the methods section, the authors provided a CONSORT diagram of the enrollment process. It is presumed the authors were familiar with the contents of the CONSORT statement. One can examine this study in the light of the CONSORT checklist and assess the probable adequacy of the review process. The abstract does not accurately reflect the contents of the body of the paper. The specified objectives of the study refer to the treatment of ST-elevation myocardial infarction (STEMI) and acute coronary syndrome (ACS). In the methods, results, and discussion, no mention is made of STEMI or ACS. Recruitment favored smaller units in many countries where policies for the use of the studied agents and their availability were not uniform. Treatments were allocated according to local practice, with little data reported on the rationale for decisions. One hypothesis is stated regarding STEMI and ACS, but not further discussed in the text. Several definitions vary from standard practice, e.g., myocardial infarction. The statistical methods used were complex and incompletely described. Many co-variates were not listed, such that the propensity score could not be externally validated. To counter the non-randomized nature of the study, this method can only rely on collected co-variates. No mention was made in the discussion of the impact of the treatments on STEMI or ACS. No comment was made of the inherent bias of the treatment allocation used. No acknowledgment was made of previous analyses with differing conclusions from the same dataset. Multiple statistical tests had been used, and results differed in the text and tables. This publication contradicts at least 64 randomized controlled trials and the weight of at least 2 meta-analyses.4,5 Scant mention of this body of evidence is given. As to its overall strength in this era of evidence-based medicine, no weight can be given to this report.
How is it that this article appears in such a purportedly prestigious journal as the New England Journal of Medicine? Is there a failure of the review process to examine the paper in the light of the comments above? Is there a hidden agenda of the editors? Should, as has been suggested in the past, the reviews be transparent? Given there have been several publications in the New England Journal of Medicine from this group, is there a publication bias for this methodology, flawed as it is? 6,7
So what is necessary to ensure accurate reporting of research? What should be the responsibilities of the editors/publishers? Should they provide referees with some training in the process or at the very least, detailed guidelines of what are the basic requirements for review? Furthermore, research into peer review is needed to support these concepts. Altman8 provided some suggestions to editors for improvement. It will take an effort to educate not only authors (researchers), who in many cases are under pressure to produce publications, but also ethics committees. Reviewers need to understand the responsibility that lies on their shoulders. Criticism of poor methods should not disturb a reviewer. When complex statistics are being used, expert statistical review is needed. Journals should insist on data and methodology, which are not an integral part of the manuscript, be provided and published electronically, enabling a critical review by an outside party.
REFERENCES
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
