Standardised mean difference in metaanalyses - How reliable is it in practice?

The use of the standardised mean difference (SMD) is common in meta‐analyses, as it allows outcomes
of a similar nature, but measured on different scales, to be combined. The application of SMDs,
compared with that of the raw mean difference, can be complex. Despite this complexity, there have
been few studies of the reliability of this effect measure in practice.
The aims of this PhD were to investigate the difficulties that may arise when researchers use SMD as
an effect measure and to determine the scope for reviewer bias. Three studies were undertaken in
order to fulfil these aims. In the first study, we evaluated the reproducibility of meta‐analyses using
SMDs. In the second study, we determined the observer variation when extracting data for the
computation of SMDs. In the third study, we investigated the range of SMDs that could be calculated
based on the same outcomes from the same trials.
The results from the three studies demonstrate that data extraction is prone to error, which can
negate or even reverse the findings of studies. Disagreements were common and often larger than the
effect of commonly used treatments and multiplicity of data was frequent and could impact
importantly on meta‐analytical results.
The conclusion is that readers should be aware that meta‐analyses using SMDs may have included
incorrect or selectively extracted data. Readers should seek assurances that data collection methods
were rigorous and clearly predefined. Reliability of meta‐analyses using SMDs might be improved by
having more detailed review protocols, more than one observer, and investigators with statistical
expertise, but this has to be confirmed by future research.