By Joy Lorenzo Kennedy (Scientific Support Specialist), Karen Adolph (Principal Investigator), & Rick O. Gilmore (Principal Investigator)
Behavioural researchers are under fire due to low levels of reproducibility; more than half of the findings from psychological studies cannot be replicated. On the reasonable assumption that both the original researchers and the replication team are competent, why are they getting different results? Inability to replicate findings might be a side effect of the inability to reproduce methods and procedures with sufficient fidelity. In fact, as behavioural research expands beyond its traditional white, middle-class participant base, one of the most notable findings is the extent to which small differences in context create differences in behaviour and development. As such, slight variations in instructions, stimuli, procedures, experimenters, testing room, and so on can lead to different outcomes. No matter how detailed the method section in published works, full explication of a methodology is impossible due to the limitations of text and static images to convey the full richness of the testing situation. This makes it difficult to ensure that researchers can truly replicate each other’s work.
Video can help. Video captures the complexity of behaviour with unsurpassed fidelity, and is a uniquely rich source of documentation of aspects of the research context that might seem unimportant, or that are hard to explain. Video can accurately represent what participants saw and heard, how researchers interacted with them, and how participants behaved. Regardless of whether video is the primary source of research data, every behavioural study can—and should—use video to document instructions to participants, stimuli, procedures, and the details of the testing context. Video captures the nuances in implementation of procedures in a way that no other format can, and thereby allows more exact replication of those procedures in other labs. Sharing procedural videos is thus one step closer to ensuring transparent and reproducible science.
Databrary, a free, online data library, was created by behavioural scientists as a platform for sharing video and is ideally suited to sharing methods with other researchers. In addition to procedural documents such as stimuli, coding manuals, analysis scripts, and output, researchers can upload videos from practice run-throughs, pilot participants, or actual data collections, giving other researchers a detailed demonstration of how they actually conducted their study. Databrary also stores demographics from the original sample, allowing replication of the study with participants from a similar background and extension of the study to those from different backgrounds to examine the robustness of results. Databrary thus provides a unified repository for all the data files, procedures, and metadata that researchers need to reproduce each other’s work.
Sharing these kinds of research data, particularly data that contain personally identifying information (PII), requires special caution. Databrary consulted with legal teams, institutional review boards, and other pioneers in data sharing to develop a framework to maximise both access to, and confidentiality of, data containing PII. Databrary restricts data access to researchers with a signed data access agreement in which their home institutions vouch for their proper research conduct. And Databrary developed permission language for researchers to use with participants to explain the purpose of data sharing and to obtain their explicit consent to do so. As of July 2017, over 600 researchers have gained access to Databrary, and many are actively collecting data with the intent to share.
Datavyu is a free, open-source, and fully customisable tool for annotating and scoring videos, and can aid in reproducibility. Researchers can use Datavyu to “carve” videos up into meaningful, easily retrievable units, and thus to analyse each other’s methods. For example, the duration of trials, or the time between trials, can be calculated easily and accurately and compared across the original and replication studies. Rapport with the investigator or participant comfort can likewise be coded and compared across studies. For studies where video is the primary source of data, researchers can reference the original Datavyu coding files, and link to exemplars of coded behaviors so they are able to more accurately replicate the original coding. Coding in Datavyu can be done with a single keystroke, and minimal hand movements control playback and coding with ease.
The real test of reproducibility is the accurate replication of procedures, demographics, and analyses across studies. Databrary and Datavyu are tools that ensure that replication studies are conducted with sufficient fidelity that researchers are actually measuring the same thing in the same way. As a result – like penicillin – a failure to replicate can become an “aha!” moment, with different findings providing meaningful insights into what does and does not matter for the behaviour being studied.