Reproducible research

8572856136_f746c2dfa3_o.jpgBy Cyril Pernet, University of Edinburgh.

This blog post was first published at

Feedback from reproducible science workshops

Only a minority of scientists think there is no 'reproducibility crisis' (Nature 533, 452–454), yet many are not engaging in reproducible practices. Results from a recent survey among psychology researchers suggest that discussion and education about the utility and feasibility of practices like data sharing are needed if we want the community to adopt those standards. In short, people don't know those practices and don't want to journey there.

One of the things I did during my Software Sustainability Institute Fellowship, was to run a series of small group workshops for post-graduate students and principal investigators, on data sharing, code sharing, and good practices around code. This took place the last of September 2018 in Oxford the 25th, Birmingham the 27th and Glasgow the 29th, and in…

Continue Reading

16121281031_5b58dfc131_z.jpgBy Laura Fortunato, University of Oxford

Reproducible Research Oxford is a project based at the University of Oxford, launched in October 2016. The project aims to lay the groundwork for a culture of research reproducibility across the University, focusing on training in the effective use of computational tools in research. These tools are widely used in some disciplines, and they can enable researchers to easily track the process leading from data to results, so that it is fully reproducible. However, researchers often lack the opportunities, incentives and confidence to make best use of these tools.

As part of the project, we have set up a partnership between the University and Software and Data Carpentry, non-profit volunteer organisations focused on teaching researchers across disciplines the computing and data skills they need for effective and reproducible research. Since the start of the project, we have ran four Software Carpentry workshops, one Data Carpentry workshop—the first to be held in Oxford!—and we have hosted the first Oxford-based Software/Data Carpentry instructor training. So far, we have provided training to upwards of 100 learners from across the University who attended our workshops, in addition…

Continue Reading

34665071964_79a52d2bba_z.jpgBy Ana Todorović, Oxford University

In September 2017, we started the school year at Oxford with a day of talks on robust research practices. Originally envisioned as a satellite meeting of the Autumn School in Cognitive Neuroscience, it ended up spilling over into the Autumn School as well, which meant that incoming masters students got their official welcome to the programme in the form of four lectures on scientific reproducibility.

The Oxford Reproducibility School was spurred into action by Kia Nobre, head of the Experimental Psychology department, and was organized by Dorothy Bishop, Ana Todorovic, Caroline Nettekoven and Verena Heise. And although their primary areas are psychology and cognitive neuroscience, the Oxford Reproducibility School was aimed at discussing problems in empirical science in general, as well as best research practices. 

We had talks that outlined the root causes of the reproducibility crisis. Talks that discussed novel statistical approaches. Talks that covered combined academic and industry practices in pharmaceutical research. Talks about efficient computing, shared analysis pipelines, data storage and ethical practices when uploading that brain scan to an online repository. Talks that covered teaching undergraduates the right way instead of having them unlearn what they first encountered in their statistics courses. Talks about preregistration, and conversely talks about exploratory…

Continue Reading


By Blair Archibald, Software Sustainability Institute fellow.

Software is (almost) never written or run in isolation. Instead, it builds on top of a wide range of dependencies from compilers and language runtime environments to application specific libraries. This is a huge challenge for reproducible research. Not only should the software we write be sustainable (e.g well versioned, documented, and tested) but the environments that the software exists within also needs to be documented and, ideally, recreatable.

Many have suggested virtual machines/containers as one solution to this problem (for instance the recent Docker Containers for Reproducible Research Workshop (C4RR)), where you ship not just your computational code but also the environment alongside. While this is a good start on tackling this problem I'm not sure it's fully sufficient. Often the environment for the image is constructed using a standard Linux distribution's package manager, and these usually, by default, install the newest (stable) possible version of a package, meaning that two people running a VM/container at two…

Continue Reading

Video and Reproducibility in the Behavioural SciencesBy Joy Lorenzo Kennedy (Scientific Support Specialist), Karen Adolph (Principal Investigator), & Rick O. Gilmore (Principal Investigator)

Behavioural researchers are under fire due to low levels of reproducibility; more than half of the findings from psychological studies cannot be replicated. On the reasonable assumption that both the original researchers and the replication team are competent, why are they getting different results? Inability to replicate findings might be a side effect of the inability to reproduce methods and procedures with sufficient fidelity. In fact, as behavioural research expands beyond its traditional white, middle-class participant base, one of the most notable findings is the extent to which small differences in context create differences in behaviour and development. As such, slight variations in instructions, stimuli, procedures, experimenters, testing room, and so on can lead to different outcomes. No matter how detailed the method section in published works, full explication of a methodology is impossible due to the limitations of text and static images to convey the full richness of the testing situation. This makes it difficult to ensure that researchers can truly…

Continue Reading

Docker Containers & Reproducible ResearchBy Raniere Silva, Community Officer, Software Sustainability Institute.

Last year, during the First Conference of Research Software Engineers, Iain Emsley, Robert Haines and Caroline Jay hit on the idea to organise a meeting about Docker and how researchers are using it. Ten months later, 60 researchers, developers and librarians met in Cambridge for the Docker Containers for Reproducible Research Workshop (C4RR).

The workshop consisted of one sponsored keynote by Microsoft, 20 talks and four lightning talks and participate in one of two demo sessions. There were many success stories involving containers and, when high performance computing (HPC) was involved, the use of  Singularity as a good alternative to Docker.


If I had to select one talk from C4RR to summarise the workshop, my choice would be Building moving castles: Scaling our analyses from laptops to supercomputers by Matthew Hartley, et al. With some images from Hayao Miyazaki’s…

Continue Reading

Docker Containers & Reproducible ResearchBy Raniere Silva, Community Officer.

Docker Containers for Reproducible Research Workshop (C4RR) is only a month away, 27-28th of June 2017 at the University of Cambridge. This workshop offers many talks on the use of containers applied to improve reproducibility on desktop, cloud and HPC environments and some practical sessions.

For those interested in HPC, some talks will surely make the workshop worth for all our attendees, Michael Bauer's one about Singularity, Matthew Hartley's one about ways to make the transition from the desktop to the HPC smother and Jeroen Schot's one describing how the Dutch National e-Infrastructure is empowering containers.

Meanwhile, the talks from Nick James, David Mawdsley and Matthew Upson are aimed at attendees who are more interested in reproducibility. Nick will talk about an open source data analysis pipeline from the European Bioinformatics Institute that employs containers. If you are an R user and are looking for ways to use Knitr with Docker to make easy for your colleagues to reproduce your R Markdown documents, David's talk is for you. And Matthew will take the attendees through a journey…

Continue Reading

Practice of Reproducible ResearchBy Justin Kitzes, University of California, Berkeley

We are very happy to announce the launch of our open, online book The Practice of Reproducible Research, to be published in print by the University of California Press later this year. In short, this book is designed to demonstrate and teach how research in the data-intensive sciences can be made more reproducible. The book centres on a collection of 31 contributed case studies, in which experienced researchers provide examples of how they combined specific tools, ideas, and practices in order to improve the reproducibility of a real-world research project. These case studies are accompanied by a set of synthesis chapters that introduce and summarise best practices for data-intensive reproducible research.

Within the overall context of reproducibility, our book focuses specifically on the goal of achieving computational reproducibility in individual research projects. We defined a research project as computationally reproducible if a second investigator can recreate the final reported results of the project, including key quantitative findings, tables, and figures, given only a set of files and written instructions. This focus reflects our belief that computational reproducibility forms a first and most foundational goal for individual investigators interested in the broad goals of reproducible…

Continue Reading

For all enquiries about sponsorship, please contact Graeme Smith.

Sponsorship Packages for Docker Containers for Reproducible Research Workshop


Scholarship Bronze Silver Gold Platinum

Free tickets

✔ (1)

✔ (1)

✔ (1)

✔ (2)

✔ (4)

Thanks (on website)

✔ (name)

✔ (small logo)

✔ (medium logo)

✔ (large logo)

✔ (huge logo)

Continue Reading

Docker Containers, Reproducible ResearchSubmit your abstract by 31st March 2017 at midnight.

Presenters are invited to submit abstracts for 15-minute talks (plus 5 minutes for questions) and lightning talks on the following subjects:

  • Examples of use—positive or otherwise and lessons learned
  • Position papers
  • Applications for Reproducible Research
  • Other use cases
  • Building other tools around container ecosystem
  • Comparing different types of containers
  • The future and challenges for adoption, or lack thereof, in specific communities

The Software Sustainability Institute’s Docker Containers for Reproducible Research Workshop (C4RR) will take place on from 27th to 28th June 2017 in Cambridge. C4RR aims to gain insight into the topics of containers technologies and how these impact and will impact on research. It is…

Continue Reading
Subscribe to Reproducible research