The Subjectivity of Meta-Analysis in Practice

An astonishing volume and diversity of evidence is available for many hypotheses in the biomedical and social sciences. Some of this evidence – usually from randomized controlled trials (RCTs) – is amalgamated by meta-analysis. The status of RCTs as the ‘gold-standard’ of evidence in medicine is debated. It is usually meta-analyses, though, rather than RCTs, which are considered the best source of evidence: meta-analysis is thought to be the platinum standard of evidence. However, I argue that meta-analyses fall far short of that standard. Different meta-analyses of the same primary evidence can reach contradictory conclusions. Meta-analysis fails to provide objective grounds for guiding belief because numerous decisions must be made when performing a meta-analysis, which allow wide latitude for subjective idiosyncrasies to influence the results. I end by discussing an older tradition of evidence in medicine: the plurality of reasoning strategies appealed to by the epidemiologist Sir Bradford Hill (1897 - 1991).