Peer Review Yesterday, Today and Tomorrow

Scientific journals began in 1665 and soon a system had evolved whereby manuscripts were referred by the editor to experts in the field.  Early on, it was emphasized that the system could not guarantee the results reported, but this system of refereeing papers to help the editor select and then improve the best of them gradually spread. Until 30 years ago, the peer review system remained virtually ignored despite its crucial role in the publication of science and the life of scientists.  In 1983, my colleague at the New England Journal of Medicine, John Bailar, together with Kay Patterson proposed a research agenda to look into peer review. I moved to JAMA, whose editor, George Lundberg, gave strong support to a congress on the subject.  We insisted that all the presentations should consist of the results of research into peer review and related aspects of scientific publication, rather than editors expressing evidence-free admiration for their system. This very public expression of support had a dramatic effect on the amount of interest in, and research on, peer review, and with successive Congresses, from the first in 1989 the number of submissions grew substantially. What do we now know about journal peer review?  We found out that peer review had not prevented numerous cases where dozens of retractions of published articles had occurred.  So, as a quality assurance system, it could not detect even the grossest fabrications.  Single and multi-journal studies of variants of peer review, blinding reviewers and so on, have advanced our knowledge, but have failed to tell us much about peer review’s predictive powers when viewed as a test of quality. In short, systematic review of hundreds of small studies have failed to show strong, convincing, objective evidence that it “works”. But that is not the way editors, editorial boards, reviewers, authors and readers feel about it.  Instead of using this evidence as a reason to save the costs and delays of peer review, journals show ever more enthusiasm for it. The advent of electronic publication and of open access journals will afford enormous opportunity for experimentation.  Already numerous papers are being published on systems that might improve peer review, but none of these papers are worth much unless the authors formally and rigorously assess them against other methods.  Only then can we make progress. Meanwhile, we should look into the remarkable disparity between the evidence on this time-consuming and expensive system and our enthusiasm for it.