ESMI Seminar: Methodology Guidance - where are the methods?

There is an increasing stream of published “methodology guidance” covering the conduct and reporting of almost every kind of observational study, randomised trial, systematic review and meta-analysis. In some cases, journals require that this guidance is followed as a condition for publication. But what are the methods that generate this guidance?

Using Network Meta-analysis (NMA) as an example, we look at guidance issued by GRADE (Grading of Recommendations Assessment, Development, and Evaluation) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). We find that, in common with the majority of “methodology guidance”, the GRADE and PRIMA extensions for Network Meta-analysis are based on no more than opinion, and issued without any explicit empirical or theoretical under-pinning. We identify aspects of GRADE-NMA and PRSIMA-NMA that are incorrect or even incoherent, and illustrate alternatives. 

The current approach to methodology guidance is leading to ossification and hegemonism, and feeds a “post-truth” agenda in which the investigator’s priority is conformity with guidelines rather than a search for scientific truth. We distinguish between prescriptive and descriptive methodology guidance, and suggest that a debate is needed on what form methodology guidance should take.