Building research evidence towards reproducibility of animal research
Since our debut in late 2006, PLOS ONE has strived to promote best practices in research reporting as a way to improve reproducibility in research. We have supported initiatives towards increased transparency, as well as the gathering of evidence that can inform improvements in the quality of reporting in research articles. In line with this commitment, PLOS ONE collaborated in a randomized controlled trial (RCT) to test the impact of an intervention asking authors to complete a reporting checklist at the time of manuscript submission. The results from this trial have recently been posted on bioRxiv (1) and provide a further step toward building the necessary evidence base to inform editorial interventions towards improving reporting quality.
The PLOS ONE publication criteria represent a commitment to reproducibility in all scientific disciplines, for example, we require that experiments, statistics and other analyses be performed to a high technical standard and described in sufficient detail. Additionally, articles need to adhere to appropriate reporting guidelines and as a multi-disciplinary journal, we encourage authors to comply with reporting guidelines for their relevant field(s) of research. For example, we require that authors follow CONSORT or TREND guidelines for reports of clinical trials (randomized and non-randomized trials, respectively) and PRISMA for systematic reviews and meta-analyses.
As part of our policies for studies reporting research on animals, we encourage authors to comply with the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines and submit a completed ARRIVE checklist with their manuscripts. This checklist consists of 20 items describing information on study design, experimental animals, housing and husbandry, sample size and so forth.
PLOS ONE has supported the ARRIVE guidelines since their development by the NC3Rs and their publication in 2010. Although the ARRIVE Guidelines have since been endorsed by a myriad of journals, funders and various learned societies, their impact on reporting quality is unclear, which prompted discussions about the need for research in this area. A number of findings suggest that reporting guidelines (such as CONSORT), have the potential to improve the quality of methodological reporting (2, 3). More recently, an observational study led by MacLeod et al. (4) involving Nature journals publishing life sciences research reported that a change in editorial policy, including the mandate of a customized checklist, was associated with improved reporting. To our knowledge, however, there do not appear to be any studies that investigate an association between reporting guidelines (in the form of a checklist) and completeness of reporting using a randomized controlled design.
PLOS widely supports efforts aimed at gathering empirical evidence about the impact of guidelines on reporting. To further such efforts in relation to the ARRIVE guidelines, we began a collaboration with researchers from the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES) in 2014. This group, led by Malcolm Macleod and Emily Sena designed a trial to assess whether solely requesting completion of the ARRIVE checklist at submission to PLOS ONE, without further editorial checks for compliance, would have any impact on the quality of the reporting within the published article. Under the leadership of former PLOS ONE Senior Editor, Liz Silva and former PLOS Advocacy Director Catriona MacCallum, a small team of our editorial staff became actively involved in devising the randomization and data collection phases of this trial. We felt it critical to take advantage of the high volume of laboratory-based in vivo studies submitted to the journal, to support an adequate sample size for the study and the generation of a substantial dataset (i.e. nearly 1700 randomized journal submissions) for analysis.
The results of the RCT, which are now available in a preprint on bioRxiv, demonstrate that solely requesting authors to complete the ARRIVE checklist at submission without further editorial checks for compliance has no effect on the quality of reporting of such studies. Although perhaps not surprising, this result suggests that journal requests for the completion of a checklist, on their own, are not sufficient to observe an improvement in the reporting of the studies. The intervention aimed at simplicity by design, looking to integrate the request for the ARRIVE checklist within existing editorial workflows. While PLOS ONE undertakes consistent check on requirements related to other reporting requirements (such as CONSORT and PRISMA checklists as indicated above), the lack of repeated requests to the authors as part of the trial may have been a factor explaining the result. Other factors may have also contributed, including the time of the request (at submission instead of during revision when authors are expected to amend their manuscript) and the relatively long list of items (20+ in the ARRIVE checklist) to be incorporated by authors. We feel that additional incentives to authors such as empirical evidence that demonstrates the benefits of thorough reporting on reproducibility may improve compliance.
Altogether, we believe that a multi-faceted approach involving different stakeholders is needed. Partly prompted by the results of this trial, the ARRIVE working group is seeking to revise the ARRIVE guidelines so that their goal of improving transparency and standards of reporting can be fully achieved. At the institutional level, further training and support would be beneficial so that researchers incorporate elements of the reporting requirements at each step of the research planning and process, and not only at the time of preparing a manuscript for publication. Journals can also support awareness about reporting practice by developing clear submission guidelines and looking into approaches to strengthen compliance with reporting requirements. In our experience on PLOS ONE, compliance is stronger for those submissions where we have clear requirements for reporting and associated checks during the editorial process.
This blog post was co-authored by PLOS ONE Editor Alejandra Clark and former PLOS ONE Editor Gina Alvino, who is now a Training and Applications Analyst at University of California, San Francisco’s Institutional Animal Care and Use Program.
References
- Hair K, Macleod MR, Sena ES and IICARus Collaboration. A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus). 2018 bioRxiv 370874, doi: https://doi.org/10.1101/370874
- Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic reviews. 2012 Nov 29;1(1):60.
- Fuller T, Peters J, Pearson M, Anderson R. Impact of the transparent reporting of evaluations with nonrandomized designs reporting guideline: Ten years on. American journal of public health. 2014 Nov;104(11):e110-7.
- Macleod MR, The NPQIP Collaborative group. Findings of a retrospective, controlled cohort study of the impact of a change in Nature journals’ editorial policy for life sciences research on the completeness of reporting study design and execution. 2017 bioRxiv 187245; doi: https://doi.org/10.1101/187245
Featured Image: Jack Snyder via Wikimedia Commons CC-BY