By Linda Nab, Daniela Gawehns, Paloma Rojas Saunero, Ricarda Proppert, and Software Sustainability Institute Fellow Anna Krystalli
On Saturday 30 November, 44 excited researchers gathered at the Leiden University Library to join the very first ReproHack – short for Reproducibility Hackathon – in the Netherlands. In a nutshell, a ReproHack is aimed at assessing the reproducibility of scientific research that publicly shared code and data. During a ReproHack, participants get to work with other people’s material in a low pressure environment and record their experiences on a number of key aspects, including reproducibility, transparency and reusability of materials.
There we go! ? See you in a bit! pic.twitter.com/i65Yi7ssbf
— ReproHack ♻️ (@ReproHack) November 30, 2019
The first few ReproHacks were run by Anna Krystalli as part of Opencon satellite events. These events were picked up on by Daniela and her call on Twitter led to the formation of the so-called “Leiden ReproHack Ladies”. The rest is history: hard work in the leading months to the event led to the first Dutch ReproHack.
Hacking away to reproduce papers! @ReproHackNL @ubleiden @CDSLeiden pic.twitter.com/jyVvMs07Hp
— Kristina Hettne (@KristinaHettne) November 30, 2019
The event in summary
Anna Krystalli took the honour of kicking off the event and shared her perspective to research reproducibility and described the concept of “research compendia”, which provides a standard and easily recognisable way for organising the digital materials of a project to enable others to inspect, reproduce, and extend the research. After Anna’s talk, participants formed teams and began the challenge of reproducing their chosen papers.
People picking papers to reproduce ? pic.twitter.com/JP8T1qkIpg
— ReproHack ♻️ (@ReproHack) November 30, 2019
Lunch break provided a great opportunity for networking and discussing hacking progress, difficulties and experiences so far. After lunch, the day continued with a talk by John Boy, who shared his vision for open science beyond the reproducibility crisis and generated an enriching discussion among the participants.
Also was very inspired by the talk of @jboy, who advocates to make open science a collective effort and asks us to value different epistemologies in the process. Transparency can be useful for quantative research, but is not always the way to go for more qualitative work. pic.twitter.com/v6VQ6rs3ed
— Ilse van de Groep (@IlsevdGroep) November 30, 2019
By the end of the afternoon, Mateusz Kuzak from the Netherlands eScience center improvised a hands-on tutorial on making software citable and introduced FAIR software for promoting good research software development. The day ended with a group discussion about the different hacking experiences and overall feedback of the event, followed by refreshments and networking. And last but not least, future ReproHack plannings!
So exciting! Many more #reprohacks to come! ?? pic.twitter.com/qkfUPFL75X
— ReproHack ♻️ (@ReproHack) November 30, 2019
The 44 participants formed a diverse group of researchers from the fields of science and engineering, all eager to build towards more open and reproducible research practices. Of 19 papers with code and data, 24 feedback forms were filled, five papers were successfully reproduced and six were nearly reproduced. The filled feedback forms were sent to the authors of the papers after the event, who appreciated the received feedback.
Thank you so much for your time @barbaravreede! your suggestions and feedback for our paper, coming from @ReproHackNL, are super useful. I'll make sure to implement them pretty soon! ?
— Marce (@Fichulina) December 5, 2019
What an accolade! Participants at the @ReproHackNL managed to reproduce the results of a study we published as a preprint:https://t.co/3XkMz4SGX2
— Paul Schneider ?? (@waq0r) December 5, 2019
We got very useful feedback that will help to improve the reproducibility of our R code + data – Thank you! https://t.co/SQ0GoOcCeW https://t.co/iz9G10XulB pic.twitter.com/f2iWrute9b
Brief evaluation of the event
The organising team attempted to reproduce the ReproHack by using the material available on github. To make organising future ReproHacks easier, experiences will be written down in an organiser’s guideline. Follow ReproHack on Twitter to stay tuned for these guidelines. The excitement of ReproHackNL’s participants made the event a success.
Had a really great day at @ReproHackNL reproducing analyses and learning what good and less good data documentation looks like! Thanks to the amazing team for organizing and feeding me ridiculous amounts of vegan bitterballen ?
— Maike Dahrendorf (@maikelenad) November 30, 2019
Had such a great time at the @ReproHackNL today! Thanks to @maikelenad @SandraJGeiger & @jollytulip for reproducing some papers with me today, I learned a lot ?
— Myrthe Vel Tromp (@rnyrthe) November 30, 2019
Read how Mateusz Kuzak experienced the ReproHack in his blog. Improvements include a better internet connection and guidelines for participants on how to review the reproducibility of a paper.
@ReproHackNL The start of the ReproHack for me today :) pic.twitter.com/p2yt8UOtoU
— Gerrit-Jan de Bruin (@GerritJandBruin) November 30, 2019
Big thank you
ReproHackNL would not have existed without sponsors: Leiden institute of advanced computer science, the data science research programme, data archiving and network services, young academy Leiden and Erasmus medical center. Also, a big shout out for Leiden university library for opening their doors. The Software Sustainability Institute kindly sponsored Anna’s trip to the Netherlands.
The ReproHack community is slowly growing. More and more people volunteer to organise ReproHacks at their own universities and research institutes. Are you excited to become part of the ReproHack community? Check here (website under construction!) if there is an event nearby or commit a paper for a future ReproHack and follow us on Twitter.
Want to discuss this post with us? Send us an email or contact us on Twitter @SoftwareSaved.