This blog post is the first in the UNIVERSE-HPC Running a Training Pilot series. These posts contain excerpts from the Pilot Operations Guide published on Zenodo and developed by UNIVERSE-HPC.

In his treatise The Art of War, Sun Tzu said that "every battle is won before it's even fought", and so it is with training pilots - early preparation is key. That's certainly not to say that a pilot is a battle against learners, or if it is a battle, it's a battle with learners against barriers to knowledge, both from the perspective of the learners wanting to improve their skills, and the training team wanting to understand how the course needs to improve.

But perhaps a better way to see a training pilot is as a collaboration between the instructors and the learners, where decisions related to planning aim to maximise the exchange of knowledge in both directions: from the training materials and training team to the learners (i.e. what will be taught), and from the learners in terms of feedback to the training team (and by extension, the training materials).

Based on this aim, the goals of a training pilot are typically twofold:

• For the learner: deliver a positive learning experience for attendees that is measurable against the learning objectives for the training.
• For the training team: gather actionable feedback to improve the organisation, delivery, and training materials for future events (and sometimes, the same event whilst it's running).

This post will introduce the key areas in which decisions need to be made during the early planning stages. During planning it's a good idea to capture decisions within a pilot brief, which is a short, living document that evolves during the pre-workshop stages and forms a single agreed point of reference for those involved. Future posts in this series will delve deeper into these areas, as well as provide guidance on the delivery of pilots and handling post-training feedback. Another important aspect to consider early is the preparation of training materials themselves, so we'll also cover the development of training materials in this post.

What will be Piloted?

When pilots are being planned there is often an understandable focus on the learning content and training materials that are being piloted, but it's a good idea to ensure that how it will be organised, delivered, and assessed receives equal attention.

For a group developing an entirely new course for the first time, this would obviously involve piloting many of these aspects for the first time. However for more established groups, or projects that aim to deliver multiple training events, some aspects may be reused alongside established ones, so it's a good idea to discuss and explicitly record in the pilot brief how each aspect will be handled. Examples may include some or all of the following:

• For content, it could be a whole course of new modules, new versions of existing modules, or perhaps piloting a selection of new modules that will form part of a larger course in the future. If it's the latter, it makes sense to pilot these in logical groups and orderings as much as possible, so there is a consistent learning narrative throughout the pilot.
• You might be trying out new ways of delivering an existing course, perhaps piloting an online version of one that was previously conducted in person.
• Perhaps you're aiming to increase course attendance, so you're piloting new ways of promoting events to your target audience.

Alongside any new aspect to be piloted, ensure there are ways to assess to what extent it has been successful and to capture how to improve it for next time. We'll cover some approaches to this throughout the series.

Method and Medium of Delivery

There are two delivery models, both organised around a scheduled event, that we'll consider in this guide series:

• Instructor-led (pedagogical), where learners follow the lead of an instructor in a shared learning environment, using techniques such as presentations or live coding.
• Supported self-learning (andragogical), where the responsibility and pace of learning are passed to the learner, although within an environment shared with other learners.

In both cases, helpers (or other instructors) are on hand to answer questions and help solve problems.

At the far end of the self-learning spectrum, there is also asynchronous self-learning, where learning is conducted purely in the learners' own environments and typically without direct support if problems are encountered.

The rise of teleconference platforms, together with the pandemic, has also seen online training events becoming increasingly popular. How does this compare with in-person training, for both instructor-led and supported self-learning?

As we'll see in the next post, there are many benefits and drawbacks to these approaches.

How to Develop Training Materials?

The development of training materials - and the pedagogy behind it - is a volume of guides in itself, but there are key aspects to note that help with running pilots. An established approach to lesson development and one recommended by the Carpentries, is backwards design (also known as reverse instructional design). Using this approach, instead of starting with the topics to teach and writing content, the process starts by considering the intended learning outcomes as a set of goals (and in the case of pilots, these goals should stem from the overall goals of the pilot). The three fundamental steps to backwards design are:

• Identify what learners should be able to do by the end of the pilot as a set of intended learning outcomes - what should the learners be able to do at the end of the pilot?
• Develop assessments to determine whether learning has taken place, derived directly from the intended learning outcomes, instead of solely as part of the organisation of a pilot which is fairly typical.
• Plan lesson content that prepares learners to complete the assessment successfully, which keeps lesson content relevant, since it's directly derived from what will be assessed and by extension from the learning outcomes.

Another advantage of this approach is that the learning outcomes are usable directly to set clear course expectations with learners, in terms of what will be taught, when advertising the workshop and introducing the course on the day.

Another key aspect is the types of learning material to use, with many workshops making use of multiple types:

Once the materials are in a functional state, and ideally long before their pilot, invite colleagues external to the training materials development team to review them. A fresh perspective on the material will help spot potentially fundamental errors and obstacles to learning that are missed during development due to expert awareness gaps or being "too close" to the material. Where possible, invite reviews from two different types of reviewer:

• Those with similar prerequisite skills to the target learners, i.e. what are the learning obstacles, what needs clarification, what works well and what needs improvement from a learner perspective
• Those with knowledge and experience of the training topics, i.e. suggest ways to simplify the taught techniques, identify missed and valuable learning opportunities that should be included, and prioritise parts to be removed based on their practical value (which is useful if the material is known to be too long)

What's Next?

Over the next few months, the three remaining posts in this series will look into the following:

• Building on this post, the decisions around where and how the pilot will be delivered
• How to collect and use feedback to improve your training pilot, from the perspectives of both the learner and training team
• Ways to promote your event and how to actually deliver the pilot, in terms of staffing, scheduling and running it

Alternatively, if you would prefer to have all the guidance at once, take a look at the full guide on Zenodo.