Collecting and Using Feedback to Improve Your Training Pilot

This blog post is the third in the UNIVERSE-HPC Running a Training Pilot series. These posts contain excerpts from the Pilot Operations Guide published on Zenodo and developed by UNIVERSE-HPC.

In the previous two posts, we looked at key areas to include in planning your pilot, the development of training materials, and covered the various options for delivery. In this post, we'll look at feedback, an integral tool to ensure the success and continuous improvement of your training workshop. We'll take a brief look at helpful approaches and the types of feedback you can obtain to help you improve your course in the future, from both the learners and the training team.

Gathering Feedback

It's a good idea to align feedback with the goals of your pilot, so it’s important to clearly define what you want the pilot to achieve beforehand so that you can tailor your feedback collection to measure these specific goals. In particular, ensure you collect feedback on any aspects of your workshop that have never been trialled before.

As well as collecting feedback on the learning experience and training materials, consider collecting feedback on other aspects of the workshop which affect the instructors’ and learners’ experiences including the method of delivery and workshop organisation. It can also be helpful to collect feedback on how participants heard about the workshop and what attracted them to it, as this can help refine your marketing strategy for future events.

It's easy to focus on learner experiences and neglect the perspective of the instructors or helpers, but observations and suggestions from the teaching team may also provide valuable insights from the pilot workshop. One method of collecting feedback from instructors is to provide in-situ instructor notes, via a platform such as GoogleDocs, for instructors to capture experiences and suggestions. For a more structured method of collecting instructor feedback, you could use a specialised training platform such as Oxford's Gutenberg infrastructure, which allows both instructors and learners to add annotations/comments directly to learning material to highlight obstacles to learning or errors.

Surveys are a popular method of feedback collection. Feedback surveys can be used before the workshop to gauge participants' expectations and prior knowledge, at the end of each day to monitor ongoing satisfaction and address issues promptly, and after the workshop to assess overall effectiveness and areas for future improvement. Ensure these surveys are concise to avoid survey fatigue and encourage honest and thorough responses.

A particularly useful tool to determine the learning effectiveness of the workshop is to establish prior knowledge and post-workshop knowledge, by asking the same technical ability questions (based on the expected learning outcomes) in the pre-workshop and post-workshop surveys. For example, for a session teaching the Bash shell, if a learning outcome is to be able to write a Bash script to execute an operation over a number of files, an example question might be "Could you write a script to loop over all the files in a directory and output the first five lines of each file?", and adding this to both surveys. Developing similar key questions for each session or topic presents a number of benefits:

The pre-workshop survey establishes a prior knowledge baseline across the cohort, allowing instructors to tailor the pilot according to cohort ability.
Comparing these questions' results from pre- and post-workshop surveys indicates the extent to which learning took place across the pilot, and helps to highlight training material areas that need improvement.

The idea is to ensure that the questions are sufficiently representative of the learning outcome but to not have too many of them.

Additionally, at the end of each day or section it can be helpful to hold wrap-up sessions to facilitate immediate and open feedback. These sessions allow participants and instructors to voice their thoughts while they are still fresh.

By implementing a comprehensive and organised feedback collection process, you can ensure you gather as much useful information as possible from your pilot workshop.

Things to Consider when your Pilot has Finished

It's worth remembering that, particularly for a pilot, the workshop isn't complete as soon as it's been delivered. In order to get the most out of a pilot workshop, aim to collect and action as much useful feedback as possible. This can be done in a number of ways.

Firstly, in terms of the surveys:

Don't neglect to ask any attendees who dropped out early to fill in the survey, since otherwise, your survey analysis is subject to survivorship bias, and it's important to understand why the training didn't connect with anyone who did not see it through to the end.
Analyse results from post-workshop survey ability questions against the same results in the pre-workshop survey, which will help to understand the extent to which learning took place across the pilot, and helps to highlight training material areas that need improvement. For example, if learning effectiveness for a particular training episode was low, this may highlight issues with the content, its delivery, or its portrayal in the material. This analysis also provides evidence on the impact of these workshops which can be used to support funding of these activities in the future.
Collate and analyse survey text responses for areas of success and improvement. This helps direct future improvement work and any positive comments provide qualitative support for the activities. Consider requesting longer testimonials from attendees who particularly liked the training, to publish in blog posts or future promotional material, for example.
Consider a follow-up with attendees, perhaps between 3-6 months after the pilot, to ask how useful they have found the training in their work and any areas that were particularly valuable.

Having feedback from attendees is only one side of the feedback coin, so it's also important to obtain feedback from the training team. One way to do this is to hold a debrief (also sometimes referred to as a "retrospective" or "post-mortem"), where aspects of organisation, delivery and materials can be discussed to identify successes and areas of improvement from the perspective of the trainers. It also provides an opportunity to quickly prioritise issues and potentially assign them to individuals and schedule the remedial work.

What's Next?

The final post in this series will look at ways to promote your event and how to actually deliver the pilot, in terms of staffing, scheduling and running it.

If you would prefer to have all the guidance at once, take a look at the full guide on Zenodo.