We are pleased to announce a new guide on the Responsible use of artificial intelligence in research software, written by Scott C. Edmunds.

The rapid uptake of AI-assisted and AI-generated (or “vibe”) coding may be one of the biggest shifts in research methodology since the advent of the computer. While this may speed up development, it comes with concerns about code quality, plagiarism, and risks of deskilling. Advances in this technology have been so rapid they are outpacing the development of ethical and legal policies, increasing risks of falling foul of copyright and plagiarism regulations. 

What are the risks in using AI-coding?

While these new AI-coding assistants have amazing potential, they should not be seen as taking over coding from human developers but rather augmenting their capabilities. As with the usage of AI tools in writing text, when using AI to write code the responsibility has become less clear. While most researchers may have sufficient English knowledge to understand, edit, and take responsibility for AI-generated text, research programmers who may understand their discipline well, may not have sufficient knowledge to understand exactly what their AI-generated code does. This "debugging cliff" is potentially even more problematic as AI-generated code has a tendency to produce more errors, an analysis of real-world open source pull requests showing AI-generated code introducing 1.7x more defects. There are also uncertainties about researchers’ responsibility and accountability for AI-generated code. Particularly in a research software context where software and computational research are submitted for publication in scientific journals, and the reviewers and editors may have similar difficulties assessing AI-generated code. 

Further uncertainty has surrounded licensing and attribution as fully autonomous AI outputs have no human author behind them. This leaves questions on whether you can patent or copyright AI-generated software, and also complicates open-source licensing. With the ability to translate code from one language to another using AI it is also unclear how translation relates to licensing. As with generative AI trained on text and images, there is a lack of transparency regarding the sources that were used for training coding assistants. With lawsuits filed against GitHub claiming that their Copilot coding assistant was trained on public projects without giving permission or appropriate credit to the original creators. With risks that these coding tools could be plagiarizing the efforts of others without the users’ knowledge, and inadvertently leading to data security issues through the accidental release of encrypted keys and breaches of data privacy laws. Training and running the LLMs behind these coding tools has required massive energy consumption, also raising sustainability concerns (see the related module on green computing).

How can you use AI-coding safely and responsibly?

While specific ethical guidelines for publishing and peer-review of AI-coded research are yet to be published at the time of drafting, by following the principles and general guidelines for usage of AI in research it is possible to responsibly use these tools while minimizing the legal and ethical risks. The rise of these tools has shifted the cognitive load for developers from creation to verification, making validation crucial to check if code is doing what it says it is doing, and mitigating overfilling and bias. Developers are adopting best practices that include using additional tests, and reserving vibe coding and less reliable outputs more for personal or low stakes projects. If you decide to share AI-generated code publicly, then this should follow ethical best practice guidelines developed by COPE and WAME for AI usage in scholarly publications. Authors who use AI tools must be transparent in disclosing their use and are the ones responsible for the content of the software and manuscript. This includes any parts produced by an AI tool, and authors are liable for any legal or ethical breaches the AI may inadvertently cause. The black-box nature of most AI-technologies makes transparency and reproducibility a challenge, but explainable AI techniques and integrating the usage of AI standards will assist in the validation and checking process, and improve trust and transparency to better facilitate re-use. As with any software, you must remember to provide good documentation to help others to use and maintain it in the future. For ML research there are many standards facilitating discoverability, access, and re-use such as the Crossant-ML metadata format. There are also a number of checklists and tools (e.g. REFORMS and DOME-ML) that can help focus on the essential information AI-driven research needs to capture, to facilitate sharing and more sustainable re-use of models. 

One final challenge is the risk of deskilling, with increasing dependence on these tools making it harder for young developers to understand and learn how to code and troubleshoot independently, but organisations such as The Carpentries are working on training materials tackling this issue.

Conclusions

For safe and ethical use of AI-coding the main things to take note of are scrutiny, disclosure, and accountability. 

AI-generated code needs to be carefully reviewed, edited, and tested by humans before sharing. Actively monitoring progress to catch problems early, alongside critically reviewing all generated code to ensure it meets scientific standards.

Disclose any LLM use alongside your code in your README and at the top of relevant modules. Upon submitting software and resulting work to a journal you should disclose how the Generative AI tools were used in their development, and be clear about what human review was performed on these outputs. This can be outlined in the covering letter and relevant sections in the paper.

Ultimately you as the human author take responsibility for the quality, correctness, and provenance of all code and the results it produces, regardless of how it was generated. If you are concerned about reliability, unclear provenance and licensing, you may want to reserve this code for personal projects rather than make it public.

For further information see related guides and Software Carpentry modules on Coding with AI.

Further Reading

Katz D, Hosseini M, Edmunds SC. Guest Post – Code Plagiarism and AI Create New Challenges for Publishing Integrity. Scholarly Kitchen.  https://scholarlykitchen.sspnet.org/2025/08/28/guest-post-code-plagiarism-and-ai-create-new-challenges-for-publishing-integrity/  

COPE Council. COPE position - Authorship and AI. https://doi.org/10.24318/cCVRZBms

WAME Recommendations on Chatbots and Generative Artificial Intelligence in Relation to Scholarly Publications. WAME. https://wame.org/page3.php?id=106  

Bridgeford EW, Campbell I, Chen Z, Lin Z, Ritz H, Vandekerckhove J, Poldrack RA.  Ten Simple Rules for AI-Assisted Coding in Science. aRxiv 2025.  https://doi.org/10.48550/arXiv.2510.22254 

Edmunds SC et al., (2026) ‘Integrating Machine Learning Standards in Disseminating Machine Learning Research’, Data Science Journal, 25(1), p. 1. Available at: https://doi.org/10.5334/dsj-2026-001 

Ray PP. A Review on Vibe Coding: Fundamentals, State-of-the-art, Challenges and Future Directions. TechRxiv. May 09, 2025. doi:36227/techrxiv.174681482.27435614/v1

Coding with AI https://southampton-rsg-training.github.io/coding-with-ai 

The research software community is nothing if not diverse. It brings together practitioners loosely connected by their relationship with the broad and sometimes nebulous idea of 'research software'. The day-to-day experience of, say, an engineer developing and maintaining computational fluid dynamics simulations is very different from that of a data architect supporting clinical data pipelines. It is unsurprising, then, that this diversity of experience gives rise to a wide range of hopes, concerns, and uncertainties - particularly within the context of Generative Artificial Intelligence (GenAI).

Research software’s rich diversity of experience is explicitly recognised in the Alliance for Data Science and AI’s (ADSA) and US-RSE’s recent community position statement on GenAI in the RSE workplace:

“The conversation around software engineering in the age of GenAI has […] left relatively little space for nuanced discussions of […] the complexities of developing research software in a future where GenAI is widely available.”

The nuance and complexity identified by ADSA make the development of GenAI guidance and policy particularly challenging. In the context of a highly diverse, reactive, and evolving discipline like research software engineering, there is no “one-size-fits-all” approach to GenAI governance. Rather, we must develop systems that capture the nuanced and shifting concerns of practitioners, leaders, funders and other stakeholders; systems that can also act as an evolving reference for best practice. The Responsible AI Risk Register for RSE (RAIRR) is intended as one such system.

RAIRR is a living register designed to support open discussion of how GenAI adoption is affecting research software practice. It allows contributors to document risks identified in their day-to-day work, alongside possible mitigations, examples, and shared responsibility for responding to them. We expect the register to become a useful resource to inform the development of AI policies that are grounded in the actual experience of RSE practitioners, rather than hype, promises, expectations or unfounded fears. RAIRR is open for anyone to use and contribute to, being particularly relevant to practitioners, leaders and policymakers.

Pilot and technical implementation

We introduced RAIRR at the Collaboration’s Workshop 2026 as an opportunity to test the idea in practice. Working in small groups, attendees were asked to identify the ways in which GenAI adoption might lead to harm, failure, or undesirable outcomes in their professional work. Crucially, they were also asked to consider valid approaches to mitigation of these risks, and suggest examples of best practices they have seen, either in institutional guidance or individual practice. This provides a pragmatic and actionable account of practitioners' perception of AI risk in research software. 

An example of one such risk is shown below (contributed by Paddy McCann’s group), which highlights the potential for the accidental reuse of licensed code in software with incompatible licenses (e.g. the use of copyleft-licensed code within more permissively licensed software). Given the provenance of AI-generated code is not visible to developers using GenAI (i.e. there is no way of knowing whether the generated code is mostly or entirely based on code with a restrictive license), there is a real risk that such code will be unintentionally reused in violation of licenses.

Risks can be added to and updated in the register through issue templates provided in the repository. As well as adding new risks or updating existing ones, the templates also allow users to add relevant or interesting resources. Since each risk is directly associated with a GitHub issue, users wishing to discuss a risk further can easily do so by adding a comment to the issue. This facilitates open discussion and allows users to add additional context or framing to existing risks. Of course, some users may wish to submit more sensitive risks – in such cases, an alternate route for anonymous contributions via an online form is also available.

What's next?

The register is open to new contributions describing risks around AI adoption within research software practices. We also welcome discussion of existing risks via their associated GitHub Issues, and encourage people to add examples of how risks can be mitigated, and to point to examples of best practice. We will be running a further workshop at RSECon26 in Sheffield, so please keep an eye out for opportunities for collaborative contribution sessions. 

Contributing to RAIRR

You can view the full register and contribute to it here. Please make sure to read the README.md and CONTRIBUTING.md for more information about what makes a good contribution to the register.

The register is maintained by the SSI Responsible AI Study Group – a recently established group interested in understanding ways in which AI may be adopted safely, fairly, and constructively within research software practices. The group is open to new members from both within the SSI and externally – if you’re interested in joining, then please contact joseph.shingleton[at]glasgow.ac.uk or oscar.seip[at]manchester.ac.uk.

We are pleased to announce the Creating and Evaluating Datasets in AI guide, written by Stef Garasto, which offers a practical introduction to responsible data collection, evaluation and documentation for Artificial Intelligence algorithms.

In response to the growing number of reports about the use of generative AI chatbots by learners at workshops, The Carpentries has launched a bonus module to Instructor Training, GenAI in Workshops, in which instructors will explore what widespread genAI use means for the way they teach.

AI models - algorithms that learn statistical patterns from data to perform a given task - need large datasets to infer the right patterns between their input (what is known) and their outputs (the task’s targets). AI development often focuses on designing novel, larger or more complex models to improve task performance. However, the data used for AI development is also of crucial importance, not only because it can impact the quality of the results obtained, but also for transparency and accountability purposes. 

Getting the best out of a dataset involves proceeding carefully at all stages of data design, starting from planning. It is important to consider not only how the data will be collected, but also what data will be collected. Here, researchers may want to think about how to balance data minimisation with data coverage - can we achieve the same goal with less data? For example, if working with texts: do they need to include different writing styles (coverage)? Do they contain unnecessary, and therefore removable, personal data (minimisation)? Indeed, minimising the amount of personal data collected is also explicitly a GDPR (General Data Protection Regulation) requirement. That said, data minimisation can be relevant more broadly, given that both collecting and storing data can be expensive in terms of computational resources, funding and time. This is also why the collection process is typically iterative, starting from a pilot dataset, then refining the requirements and expanding, or reducing, the dataset.

There are multiple data collection methods that may be employed, including:

1. Web scraping, or, generally, automated collection from web sources. This practice is common yet debated, for example because of the risk of ingesting personally identifiable information, and therefore best approached with care.
2. Simulations and sensors, where data is generated from computer programmes (for example, weather forecast models) or instruments (for example, microscopy data). 
3. Data collection from participants, whether targeted (directly recruiting participants) or crowdsourced (soliciting contributions from volunteers). 

Each different method can support different goals for why we may want to collect a dataset. For example, training a model typically necessitates a larger dataset size, and therefore tends to involve some element of automated collection from web sources, or simulations.  Data collection from participants is often used for smaller or benchmark datasets, for example to evaluate existing models. However, this distinction is not absolute: Mozilla Common Voice is an example of an open-source dataset created via crowdsourcing from volunteer contributors, yet able to support model training too. 

Irrespective of the collection method, a recommended best practice for responsible data collection is that of maintaining extensive and rigorous metadata documentation. It can be helpful to use documentation standards like Datasheets for Datasets or Data Cards. Metadata themselves can be stored in a machine readable format.

Documentation includes keeping track of the provenance of data sources and, where relevant, the licences and terms under which they were released. Different portions of the data could also come from sources with varied licensing requirements - for example, some might allow commercial use and others do not, increasing the need for maintaining precise documentation.  Furthermore, even permissible licenses or terms of services may still not imply there is sufficient awareness or consent from the people represented in the data (for datasets involving human subjects). Implementing opt-in or, depending on the circumstances, opt-out mechanisms can help increase the level of transparency and accountability in the data collection process (the BigCode project is an example of opt-out data governance). 

Metadata documentation can also encompass describing why the dataset was created, and what it should and should not be used for. Similarly, it can incorporate a specific training/test split that dataset users should apply. Training/test split means creating a non-overlapping partition of the data: the model learns the desired patterns from the training subset and its generalisation capabilities are evaluated on the test. The test subset should be as indicative as possible of the model’s performance on new data, otherwise we might obtain an optimistic, rather than realistic, measure of generalisation capability. For example, if a medical dataset contains multiple samples from the same patient, the training and test sets could be split by patient, ensuring that data from a single patient does not appear in both subsets. 

After creating the dataset, exploring it is necessary to better understand its content and structure. Here, important steps include calculating descriptive statistics, checking for duplicates and outliers, detecting (and removing) potentially harmful or illegal content, and identifying potential gaps in representativeness and coverage. Personally identifiable information included in the dataset should be removed. 

Exploring the distribution of the target labels or values is also crucial to check for gaps, imbalances or errors. This is because labels, as the desired output of the AI algorithm, are typically the main signal from which the model learns how to perform a task. Datasets may not always include labels when collected. Various data points might therefore need to be annotated with such labels, for example by domain experts or volunteers - a process that is complex but crucial, and therefore worth resourcing appropriately. When annotations are performed by multiple labellers, inter-annotator agreement measures can indicate how reliably each sample can be labelled. Disagreements between annotators may reflect not just labelling errors, but also intrinsic uncertainty in what the right label should be. Indeed, the categories that we use to classify data points rely on human discretion - both to determine which category should be assigned to a data point, but also in deciding which categories should be used in the first place. This can lead to bias entering the dataset and then being propagated further by an algorithm trained to replicate the same biased labels.

The data exploration work can also continue after a (partial) model is trained, for example by measuring which data points a model is more or less familiar with. This can help identify learning gaps, which may be due to, for example, noisy or not well represented subsets of the data. For example, the Gender Shades project was a seminal work demonstrating how AI models can underperform for specific groups, likely due to under-representation of those groups in the training data. Characteristics of the data, including its representativeness, can also change over time (“data drift”). Continuously monitoring the statistics of your datasets can help understand when such a drift has occurred, potentially indicating the need to update the dataset and retrain the AI model. 

To conclude, when creating datasets for AI development it is important to:

• Plan the data collection process, considering data relevance, coverage, consent and minimisation. 
• Create and maintain accurate metadata documentation.
• Explore and monitor the dataset characteristics, updating (and retraining) when necessary.

Further resources 

• HuggingFace’s Data Measurements Tool.
• ODI’s framework for AI-ready data.
• UK Data Services’ UK Web Scraping Compliance Guide 2026.
• Towards Best Practices for Open Datasets for LLM Training.
• Towards accountability for machine learning datasets: Practices from software engineering and infrastructure.
• Advances, challenges and opportunities in creating data for trustworthy AI.

Acknowledgements

This guide was written by Stef Garasto and reviewed by Fil Garciano. 

The Carpentries is hosting a new series of GenAI + Teaching Discussions, bringing educators together to share their experiences: the challenges and opportunities they have identified, strategies they have tried in response, and what they learned in the process. Although conversations are expected to place a particular focus on the teaching of computational research skills (programming, data handling and analysis, etc), all educators are welcome at these sessions.

As part of our new series on Artificial Intelligence in research software, we have published a new guide: Getting Started with ML and AI in Research Software, written by Paul J. Wright and reviewed by Yo Yehudi.

Machine learning (ML) and artificial intelligence (AI) have quickly become part of the modern research toolkit, changing how scientific software is built and maintained. As their use grows, it is important to maintain fundamental principles of research software—reusability, reproducibility, and transparency. This guide introduces the paradigm shift from rule-based to experiment-focused software and outlines practices and tools for designing research software to support ML tasks.

Deterministic Foundations of Research Software

In traditional research software, computational methods transform data into results, from imaging and data analysis to large-scale scientific simulation. These programs are typically deterministic, meaning the same input always produces the same output. The developer explicitly defines the transformation function (for example, y=f(x)) and, because the behaviour is fully determined by the program logic, the results are repeatable, verifiable, and transparent. These are properties that standard quality assurance practices, such as unit testing, rely on.

ML departs from this paradigm. Instead of the developer manually defining the transformation function during training, the ML model approximates f(x) by estimating its parameters from example input-output pairs (x,y). The trained model is therefore shaped by the learning algorithm, the training and validation data, the optimisation process, and configuration choices. 

The accompanying figure contrasts a deterministic system with three independently trained models, each receiving the same input but producing different outputs across training runs.

This paradigm shift affects governance and reproducibility. In deterministic systems, codebase version control is generally sufficient, since the program logic fully determines the outcome. In ML systems, the code specifies how training should proceed but does not capture the experimentation itself: the data used, the hyperparameters tried, and the random seeds of countless configurations along the way. It is entirely possible to produce a high-performing model with no reliable record of its construction. The artefact exists; the path back to it does not.
 

Reproducibility, Governance, and Supporting Tools

This shift moves researchers from a script that produces a result to an experiment that produces a model. Reproducibility now relies on capturing all conditions under which training occurred.

Fortunately, established ML Engineering and MLOps practices offer ways to manage this complexity. MLOps extends software engineering principles to ML, focusing on automation, reproducibility, and governance throughout the entire ML model lifecycle. Standard workflows typically feature dataset versioning, training pipelines, experiment tracking, model registries, continuous integration and deployment, production monitoring, and environment reproducibility. Collectively, these practices reduce ad hoc work, making the experimental record a first-class artefact alongside the model itself. In practice, this means being deliberate about three key areas, with supporting tools discussed at the end of this section:

The first is data. Reliable ML systems depend on clear data provenance and reproducible preprocessing. Best practices include capturing raw data hashes, version-controlling data preprocessing pipelines, and validating that incoming data conforms to expected schemas and distributions. Generating distinct training, validation, and test data is essential to prevent data leakage, with the test set only used for final evaluation. The dataset should be tagged with its version. Any data augmentation during training must be tracked and repeatable (applied offline and saved, or with a chosen random seed). Without these safeguards, a model's provenance becomes opaque: even perfectly versioned code cannot reveal which data actually shaped the model.

The second is the training process itself. Model behaviour is highly sensitive to choices such as architecture, optimisation strategy, random initialisation, and evaluation metrics. These sources of variability must be systematically captured through experiment tracking. Without such tracking, a model may be impossible to reproduce or explain, and subsequent runs may behave unpredictably.

Third is model evaluation. Offline metrics alone are insufficient; models should be validated against datasets that were previously excluded from the training process. Take care not to exhaust this resource through model evaluation. When data arrives continuously, production monitoring provides an additional source of out-of-sample evaluation and is essential for detecting data drift and model degradation over time.

Several tools support these directly, spanning orchestration, versioning, tracking, and model management:

• Pipeline orchestration tools, such as Prefect, manage and schedule workflows, while purpose-built ML pipeline frameworks (for example, ZenML) provide abstractions tailored to ML workloads.
• Data versioning tools, such as Data Version Control (DVC), extend version control to datasets and data preprocessing steps, linking them directly to model outputs. This helps ensure experiments can be reliably reproduced as datasets evolve over time.
• Experiment tracking platforms, such as MLflow and Weights & Biases, record configurations, hyperparameters, metrics, and artefacts across training runs, enabling results to be traced.
• Model registries and sharing platforms, such as Hugging Face, serve as structured repositories for storing trained models with metadata, documentation, and evaluation results. This supports repeatability, reproducibility, and model reuse within the research community.

Combined, they allow ML systems to be developed with the same rigour and traceability expected of any other piece of scientific software.

Further reading

Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022)

Acknowledgements

This guide was written by Paul J. Wright and reviewed by Yo Yehudi. 

• Paul J. Wright's ORCID: https://orcid.org/0000-0001-9021-611X
• Yo Yehudi's ORCID: https://orcid.org/0000-0003-2705-1724 

The Software Sustainability Institute and Lingnan University, Hong Kong, are proud to jointly feature a webinar on AI world models. The event will take place online on Friday 17 April between 9 and 10.30 BST.

As AI transitions from processing data to generating environments, this session asks the pivotal question: What happens when AI goes beyond simulating reality and begins actively building it?

Professor Sunil Manghani will challenge our understanding of cognition itself, asking whether AI’s capacity for “world-making” suggests reality is more computable—and more finite—than we ever imagined. 

Professor Harry Yang will unveil World Action Models (WAMs), a breakthrough framework that allows robots to “imagine” the physical consequences of their actions before moving, paving the way for a new generation of embodied physical intelligence. Don’t miss this opportunity to explore the philosophical implications and technical breakthroughs defining how AI moves from seeing to making worlds.

The event will be moderated by SSI Fellow Gerui Wang.