We are pleased to announce the guide, Integrating Security and Ethics into AI-Driven Research Software, by Abdulhamid A. Ardo, reviewed by Chandan Bose.

The Institute for Research Software is running a series on Good practice in research software development in the age of AI.

This series provides researchers with accessible, practical training and resources to build understanding of artificial intelligence and develop skills to use AI tools effectively and responsibly in research. 

Taking place from 24 November to 3 December, this series will feature a variety of formats, including presentations from experts in the field, a panel discussion, training courses and blog posts, offering diverse opportunities to explore AI, its applications in research software, and good practice.

The training courses cater for a range of experience, from beginners with no coding background to researchers looking to integrate AI into established software development workflows. Through hands-on activities, participants will explore AI concepts, use AI to write, debug and understand code, and apply AI tools to plan and develop research software, while considering important issues around reliability, security, ethics and good software development practice.

Registration opens on 26 October via Eventbrite. 

AI-driven research software influences scientific discovery, policy decisions, and public understanding. However, many research teams focus primarily on improving model performance while paying less attention to security and ethical robustness.

Integrating security and ethical considerations early in the development process helps research teams:

• Improve trust in research outputs
• Reduce long-term technical and ethical risks
• Contribute to responsible AI practices 

As AI-driven research software is increasingly used to generate insights, automate analysis, and support decision-making, it can introduce security vulnerabilities, ethical risks, and reproducibility challenges without careful design. This guide provides practical steps for research software engineers (RSEs) and project leads to integrate security and ethics into the development process.

Understanding the Risks in AI-Driven Research Software 

AI-driven research software introduces risks that may not be present in traditional software systems. Because AI models rely heavily on data, security and ethical issues may emerge throughout the development lifecycle. Thus, understanding these risks helps research teams design safeguards early and avoid problems that may affect research integrity and trust.

Security Risks

AI-driven research software often relies on large datasets, shared computing infrastructure, and multiple external software libraries. Without appropriate safeguards, these systems can introduce several security vulnerabilities, including:

• Data leakage from training datasets, where sensitive research data is accidentally exposed or accessed without authorization
• Model theft or adversarial attacks, where attackers attempt to copy a trained model or manipulate inputs to produce incorrect outputs
• Dependency vulnerabilities in machine learning libraries that may introduce security weaknesses into the system

For example, a research lab training a medical diagnostic model using identifiable patient data without proper access controls could unintentionally expose sensitive health information. This could lead to regulatory violations, ethical concerns, and damage to the credibility of the research project.

Ethical Risks

AI systems can also introduce ethical challenges that affect the fairness, transparency, and reliability of research outputs. Common issues include:

• Bias in datasets
• Lack of explainability
• Opaque decision-making processes
• Unclear accountability for AI outputs

In research settings, this can lead to findings that are difficult to validate, reproduce, or interpret. Recognising these risks early lets research teams adopt responsible practices, such as dataset review, model documentation, and transparency checks, during the design, development, and deployment of AI-driven research software.

Integrating Security and Ethics Across the AI Lifecycle

Security and ethical considerations should be integrated throughout the AI development lifecycle rather than added after a model has been built. 

Stage 1: Problem Framing: At the start of a project, research teams should clarify the purpose of the AI system. Key considerations include:

• Defining system intended use 
• Identifying who may be affected by the model’s outputs
• Considering potential harms if the model produces incorrect or biased results

Stage 2: Data Collection and Preparation: Datasets used to train AI models should be carefully reviewed before development begins. Research software engineers should:

• Check the provenance and licensing of datasets
• Verify that appropriate permissions or consent exist for data use
• Assess whether the dataset is representative for the research task

Stage 3: Model Development: During model development, teams should introduce security and ethics checks to improve transparency and reliability:

• Test datasets and model outputs for potential bias
• Document model assumptions, and design choices
• Track software dependencies used in the machine learning pipeline

Stage 4: Deployment and Maintenance: Security and ethical responsibilities continue after the model is deployed. Research teams should:

• Monitor model behaviour and performance over time
• Document known limitations and appropriate use cases
• Review systems periodically for bias, model drift, or security vulnerabilities

Practical Checklist for Research Teams 

A simple checklist helps early RSEs embed security and ethics into AI research software:

• Problem Framing: Define AI purpose, stakeholders, and potential harms
• Data: Verify consent, licensing, and representativeness
• Model Development: Track assumptions, document design choices, and use version control
• Deployment: Monitor performance, document limitations, review for vulnerabilities, and plan updates

Conclusion 

Integrating security and ethics into AI-driven research software is essential for building trustworthy, reproducible, and responsible systems. Considering risks early, from problem framing to deployment, reduces the chance of bias, data breaches, or opaque decision-making. For early RSEs, using structured practices, documentation, and simple checkpoints makes the process manageable and repeatable. By following the lifecycle guidance and practical checklist, research teams can produce AI systems that perform well while upholding ethical standards, protecting sensitive data, and maintaining credibility. 

Acknowledgements

This guide was written by Abdulhamid A. Ardo, Assistant Professor at British University of Bahrain, and reviewed by Chandan Bose, University of Birmingham.

Contact Abdulhamid:

• LinkedIn

• ORCID

We are pleased to announce a new guide on the Responsible use of artificial intelligence in research software, written by Scott C. Edmunds.

The rapid uptake of AI-assisted and AI-generated (or “vibe”) coding may be one of the biggest shifts in research methodology since the advent of the computer. While this may speed up development, it comes with concerns about code quality, plagiarism, and risks of deskilling. Advances in this technology have been so rapid they are outpacing the development of ethical and legal policies, increasing risks of falling foul of copyright and plagiarism regulations. 

What are the risks in using AI-coding?

While these new AI-coding assistants have amazing potential, they should not be seen as taking over coding from human developers but rather augmenting their capabilities. As with the usage of AI tools in writing text, when using AI to write code the responsibility has become less clear. While most researchers may have sufficient English knowledge to understand, edit, and take responsibility for AI-generated text, research programmers who may understand their discipline well, may not have sufficient knowledge to understand exactly what their AI-generated code does. This "debugging cliff" is potentially even more problematic as AI-generated code has a tendency to produce more errors, an analysis of real-world open source pull requests showing AI-generated code introducing 1.7x more defects. There are also uncertainties about researchers’ responsibility and accountability for AI-generated code. Particularly in a research software context where software and computational research are submitted for publication in scientific journals, and the reviewers and editors may have similar difficulties assessing AI-generated code. 

Further uncertainty has surrounded licensing and attribution as fully autonomous AI outputs have no human author behind them. This leaves questions on whether you can patent or copyright AI-generated software, and also complicates open-source licensing. With the ability to translate code from one language to another using AI it is also unclear how translation relates to licensing. As with generative AI trained on text and images, there is a lack of transparency regarding the sources that were used for training coding assistants. With lawsuits filed against GitHub claiming that their Copilot coding assistant was trained on public projects without giving permission or appropriate credit to the original creators. With risks that these coding tools could be plagiarizing the efforts of others without the users’ knowledge, and inadvertently leading to data security issues through the accidental release of encrypted keys and breaches of data privacy laws. Training and running the LLMs behind these coding tools has required massive energy consumption, also raising sustainability concerns (see the related module on green computing).

How can you use AI-coding safely and responsibly?

While specific ethical guidelines for publishing and peer-review of AI-coded research are yet to be published at the time of drafting, by following the principles and general guidelines for usage of AI in research it is possible to responsibly use these tools while minimizing the legal and ethical risks. The rise of these tools has shifted the cognitive load for developers from creation to verification, making validation crucial to check if code is doing what it says it is doing, and mitigating overfilling and bias. Developers are adopting best practices that include using additional tests, and reserving vibe coding and less reliable outputs more for personal or low stakes projects. If you decide to share AI-generated code publicly, then this should follow ethical best practice guidelines developed by COPE and WAME for AI usage in scholarly publications. Authors who use AI tools must be transparent in disclosing their use and are the ones responsible for the content of the software and manuscript. This includes any parts produced by an AI tool, and authors are liable for any legal or ethical breaches the AI may inadvertently cause. The black-box nature of most AI-technologies makes transparency and reproducibility a challenge, but explainable AI techniques and integrating the usage of AI standards will assist in the validation and checking process, and improve trust and transparency to better facilitate re-use. As with any software, you must remember to provide good documentation to help others to use and maintain it in the future. For ML research there are many standards facilitating discoverability, access, and re-use such as the Crossant-ML metadata format. There are also a number of checklists and tools (e.g. REFORMS and DOME-ML) that can help focus on the essential information AI-driven research needs to capture, to facilitate sharing and more sustainable re-use of models. 

One final challenge is the risk of deskilling, with increasing dependence on these tools making it harder for young developers to understand and learn how to code and troubleshoot independently, but organisations such as The Carpentries are working on training materials tackling this issue.

Conclusions

For safe and ethical use of AI-coding the main things to take note of are scrutiny, disclosure, and accountability. 

AI-generated code needs to be carefully reviewed, edited, and tested by humans before sharing. Actively monitoring progress to catch problems early, alongside critically reviewing all generated code to ensure it meets scientific standards.

Disclose any LLM use alongside your code in your README and at the top of relevant modules. Upon submitting software and resulting work to a journal you should disclose how the Generative AI tools were used in their development, and be clear about what human review was performed on these outputs. This can be outlined in the covering letter and relevant sections in the paper.

Ultimately you as the human author take responsibility for the quality, correctness, and provenance of all code and the results it produces, regardless of how it was generated. If you are concerned about reliability, unclear provenance and licensing, you may want to reserve this code for personal projects rather than make it public.

For further information see related guides and Software Carpentry modules on Coding with AI.

Further Reading

Katz D, Hosseini M, Edmunds SC. Guest Post – Code Plagiarism and AI Create New Challenges for Publishing Integrity. Scholarly Kitchen.  https://scholarlykitchen.sspnet.org/2025/08/28/guest-post-code-plagiarism-and-ai-create-new-challenges-for-publishing-integrity/  

COPE Council. COPE position - Authorship and AI. https://doi.org/10.24318/cCVRZBms

WAME Recommendations on Chatbots and Generative Artificial Intelligence in Relation to Scholarly Publications. WAME. https://wame.org/page3.php?id=106  

Bridgeford EW, Campbell I, Chen Z, Lin Z, Ritz H, Vandekerckhove J, Poldrack RA.  Ten Simple Rules for AI-Assisted Coding in Science. aRxiv 2025.  https://doi.org/10.48550/arXiv.2510.22254 

Edmunds SC et al., (2026) ‘Integrating Machine Learning Standards in Disseminating Machine Learning Research’, Data Science Journal, 25(1), p. 1. Available at: https://doi.org/10.5334/dsj-2026-001 

Ray PP. A Review on Vibe Coding: Fundamentals, State-of-the-art, Challenges and Future Directions. TechRxiv. May 09, 2025. doi:36227/techrxiv.174681482.27435614/v1

Coding with AI https://southampton-rsg-training.github.io/coding-with-ai 

The research software community is nothing if not diverse. It brings together practitioners loosely connected by their relationship with the broad and sometimes nebulous idea of 'research software'. The day-to-day experience of, say, an engineer developing and maintaining computational fluid dynamics simulations is very different from that of a data architect supporting clinical data pipelines. It is unsurprising, then, that this diversity of experience gives rise to a wide range of hopes, concerns, and uncertainties - particularly within the context of Generative Artificial Intelligence (GenAI).

Research software’s rich diversity of experience is explicitly recognised in the Alliance for Data Science and AI’s (ADSA) and US-RSE’s recent community position statement on GenAI in the RSE workplace:

“The conversation around software engineering in the age of GenAI has […] left relatively little space for nuanced discussions of […] the complexities of developing research software in a future where GenAI is widely available.”

The nuance and complexity identified by ADSA make the development of GenAI guidance and policy particularly challenging. In the context of a highly diverse, reactive, and evolving discipline like research software engineering, there is no “one-size-fits-all” approach to GenAI governance. Rather, we must develop systems that capture the nuanced and shifting concerns of practitioners, leaders, funders and other stakeholders; systems that can also act as an evolving reference for best practice. The Responsible AI Risk Register for RSE (RAIRR) is intended as one such system.

RAIRR is a living register designed to support open discussion of how GenAI adoption is affecting research software practice. It allows contributors to document risks identified in their day-to-day work, alongside possible mitigations, examples, and shared responsibility for responding to them. We expect the register to become a useful resource to inform the development of AI policies that are grounded in the actual experience of RSE practitioners, rather than hype, promises, expectations or unfounded fears. RAIRR is open for anyone to use and contribute to, being particularly relevant to practitioners, leaders and policymakers.

Pilot and technical implementation

We introduced RAIRR at the Collaboration’s Workshop 2026 as an opportunity to test the idea in practice. Working in small groups, attendees were asked to identify the ways in which GenAI adoption might lead to harm, failure, or undesirable outcomes in their professional work. Crucially, they were also asked to consider valid approaches to mitigation of these risks, and suggest examples of best practices they have seen, either in institutional guidance or individual practice. This provides a pragmatic and actionable account of practitioners' perception of AI risk in research software. 

An example of one such risk is shown below (contributed by Paddy McCann’s group), which highlights the potential for the accidental reuse of licensed code in software with incompatible licenses (e.g. the use of copyleft-licensed code within more permissively licensed software). Given the provenance of AI-generated code is not visible to developers using GenAI (i.e. there is no way of knowing whether the generated code is mostly or entirely based on code with a restrictive license), there is a real risk that such code will be unintentionally reused in violation of licenses.

Risks can be added to and updated in the register through issue templates provided in the repository. As well as adding new risks or updating existing ones, the templates also allow users to add relevant or interesting resources. Since each risk is directly associated with a GitHub issue, users wishing to discuss a risk further can easily do so by adding a comment to the issue. This facilitates open discussion and allows users to add additional context or framing to existing risks. Of course, some users may wish to submit more sensitive risks – in such cases, an alternate route for anonymous contributions via an online form is also available.

What's next?

The register is open to new contributions describing risks around AI adoption within research software practices. We also welcome discussion of existing risks via their associated GitHub Issues, and encourage people to add examples of how risks can be mitigated, and to point to examples of best practice. We will be running a further workshop at RSECon26 in Sheffield, so please keep an eye out for opportunities for collaborative contribution sessions. 

Contributing to RAIRR

You can view the full register and contribute to it here. Please make sure to read the README.md and CONTRIBUTING.md for more information about what makes a good contribution to the register.

The register is maintained by the SSI Responsible AI Study Group – a recently established group interested in understanding ways in which AI may be adopted safely, fairly, and constructively within research software practices. The group is open to new members from both within the SSI and externally – if you’re interested in joining, then please contact joseph.shingleton[at]glasgow.ac.uk or oscar.seip[at]manchester.ac.uk.

We are pleased to announce the Creating and Evaluating Datasets in AI guide, written by Stef Garasto, which offers a practical introduction to responsible data collection, evaluation and documentation for Artificial Intelligence algorithms.

In response to the growing number of reports about the use of generative AI chatbots by learners at workshops, The Carpentries has launched a bonus module to Instructor Training, GenAI in Workshops, in which instructors will explore what widespread genAI use means for the way they teach.

AI models - algorithms that learn statistical patterns from data to perform a given task - need large datasets to infer the right patterns between their input (what is known) and their outputs (the task’s targets). AI development often focuses on designing novel, larger or more complex models to improve task performance. However, the data used for AI development is also of crucial importance, not only because it can impact the quality of the results obtained, but also for transparency and accountability purposes. 

Getting the best out of a dataset involves proceeding carefully at all stages of data design, starting from planning. It is important to consider not only how the data will be collected, but also what data will be collected. Here, researchers may want to think about how to balance data minimisation with data coverage - can we achieve the same goal with less data? For example, if working with texts: do they need to include different writing styles (coverage)? Do they contain unnecessary, and therefore removable, personal data (minimisation)? Indeed, minimising the amount of personal data collected is also explicitly a GDPR (General Data Protection Regulation) requirement. That said, data minimisation can be relevant more broadly, given that both collecting and storing data can be expensive in terms of computational resources, funding and time. This is also why the collection process is typically iterative, starting from a pilot dataset, then refining the requirements and expanding, or reducing, the dataset.

There are multiple data collection methods that may be employed, including:

1. Web scraping, or, generally, automated collection from web sources. This practice is common yet debated, for example because of the risk of ingesting personally identifiable information, and therefore best approached with care.
2. Simulations and sensors, where data is generated from computer programmes (for example, weather forecast models) or instruments (for example, microscopy data). 
3. Data collection from participants, whether targeted (directly recruiting participants) or crowdsourced (soliciting contributions from volunteers). 

Each different method can support different goals for why we may want to collect a dataset. For example, training a model typically necessitates a larger dataset size, and therefore tends to involve some element of automated collection from web sources, or simulations.  Data collection from participants is often used for smaller or benchmark datasets, for example to evaluate existing models. However, this distinction is not absolute: Mozilla Common Voice is an example of an open-source dataset created via crowdsourcing from volunteer contributors, yet able to support model training too. 

Irrespective of the collection method, a recommended best practice for responsible data collection is that of maintaining extensive and rigorous metadata documentation. It can be helpful to use documentation standards like Datasheets for Datasets or Data Cards. Metadata themselves can be stored in a machine readable format.

Documentation includes keeping track of the provenance of data sources and, where relevant, the licences and terms under which they were released. Different portions of the data could also come from sources with varied licensing requirements - for example, some might allow commercial use and others do not, increasing the need for maintaining precise documentation.  Furthermore, even permissible licenses or terms of services may still not imply there is sufficient awareness or consent from the people represented in the data (for datasets involving human subjects). Implementing opt-in or, depending on the circumstances, opt-out mechanisms can help increase the level of transparency and accountability in the data collection process (the BigCode project is an example of opt-out data governance). 

Metadata documentation can also encompass describing why the dataset was created, and what it should and should not be used for. Similarly, it can incorporate a specific training/test split that dataset users should apply. Training/test split means creating a non-overlapping partition of the data: the model learns the desired patterns from the training subset and its generalisation capabilities are evaluated on the test. The test subset should be as indicative as possible of the model’s performance on new data, otherwise we might obtain an optimistic, rather than realistic, measure of generalisation capability. For example, if a medical dataset contains multiple samples from the same patient, the training and test sets could be split by patient, ensuring that data from a single patient does not appear in both subsets. 

After creating the dataset, exploring it is necessary to better understand its content and structure. Here, important steps include calculating descriptive statistics, checking for duplicates and outliers, detecting (and removing) potentially harmful or illegal content, and identifying potential gaps in representativeness and coverage. Personally identifiable information included in the dataset should be removed. 

Exploring the distribution of the target labels or values is also crucial to check for gaps, imbalances or errors. This is because labels, as the desired output of the AI algorithm, are typically the main signal from which the model learns how to perform a task. Datasets may not always include labels when collected. Various data points might therefore need to be annotated with such labels, for example by domain experts or volunteers - a process that is complex but crucial, and therefore worth resourcing appropriately. When annotations are performed by multiple labellers, inter-annotator agreement measures can indicate how reliably each sample can be labelled. Disagreements between annotators may reflect not just labelling errors, but also intrinsic uncertainty in what the right label should be. Indeed, the categories that we use to classify data points rely on human discretion - both to determine which category should be assigned to a data point, but also in deciding which categories should be used in the first place. This can lead to bias entering the dataset and then being propagated further by an algorithm trained to replicate the same biased labels.

The data exploration work can also continue after a (partial) model is trained, for example by measuring which data points a model is more or less familiar with. This can help identify learning gaps, which may be due to, for example, noisy or not well represented subsets of the data. For example, the Gender Shades project was a seminal work demonstrating how AI models can underperform for specific groups, likely due to under-representation of those groups in the training data. Characteristics of the data, including its representativeness, can also change over time (“data drift”). Continuously monitoring the statistics of your datasets can help understand when such a drift has occurred, potentially indicating the need to update the dataset and retrain the AI model. 

To conclude, when creating datasets for AI development it is important to:

• Plan the data collection process, considering data relevance, coverage, consent and minimisation. 
• Create and maintain accurate metadata documentation.
• Explore and monitor the dataset characteristics, updating (and retraining) when necessary.

Further resources 

• HuggingFace’s Data Measurements Tool.
• ODI’s framework for AI-ready data.
• UK Data Services’ UK Web Scraping Compliance Guide 2026.
• Towards Best Practices for Open Datasets for LLM Training.
• Towards accountability for machine learning datasets: Practices from software engineering and infrastructure.
• Advances, challenges and opportunities in creating data for trustworthy AI.

Acknowledgements

This guide was written by Stef Garasto and reviewed by Fil Garciano. 

The Carpentries is hosting a new series of GenAI + Teaching Discussions, bringing educators together to share their experiences: the challenges and opportunities they have identified, strategies they have tried in response, and what they learned in the process. Although conversations are expected to place a particular focus on the teaching of computational research skills (programming, data handling and analysis, etc), all educators are welcome at these sessions.