How to Choose the Right Pilot Group for Training

If you need one helicopter, order two.

That’s a lesson we learned at Talance when creating an online course on Incident Command System. ICS is a language that all emergency responders know, including the police, firefighters, and Coast Guard. That way when there’s a disaster, everyone knows what to do, no matter who they work for.

The lesson is to be prepared—really prepared—for when it counts. What if something goes wrong with the first helicopter? You have another at the ready.

We always order a second helicopter by putting extra energy into planning, testing, and building in redundancies. We want the courses we write and the learning management system we use to have the best chance of success.

When we begin planning for a new project with a client, we always stress the importance of testing the course with a pilot group before officially launching it. This is a process that many e-learning projects skip, or don’t follow completely. The risk with creating online training is that the people writing the courses are rarely the same people taking the course. That means the course can be irrelevant or miss critical information that learners need.

Planning to launch a virtual training program? Consider Virtual Training: Top 10 Questions From Program Managers and Directors

Save costs and time by conducting a course pilot.

If you do this early in the process, you can save development costs by avoiding learning gaps and fixing issues early in the process. Programs that subscribe to our courses also pilot their choices. That way, they know if they’re making the right selection for the right audience.

There’s no substitute for gathering together a pilot group and gathering feedback. We show examples from our past work of how piloting has helped create better courses in ways no one could have guessed.

Here are some best practices we follow with our partners and clients so they can pick the right pilot group and maximize its value.

Who to invite to your pilot group.

The first step is often the hardest. Choosing who to involve in the pilot planning process can be complex, and it’s tempting to involve as many stakeholders as possible. This is a mistake. When programs invite dozens of people, especially decision-makers, subject-matter experts, coworkers and community partners, they’ll likely receive a different opinion from each person.

That’s too many people, and they probably don’t necessarily have the right opinions. If you’re a program manager in this position, ask yourself: “Will my boss be taking this course when it’s done?” If the answer is “no,” then they’re not the most important person to involve in a pilot phase.

We advise recruiting a selection of people from three groups:

  1. learners who are similarly motivated as the intended audience but have no experience with the material or with online training;
  2. learners who have some experience with the program’s previous training modules;
  3. one or two subject-matter experts.

This mix gives programs the best chance to understand what learners need to know and identifying gaps in the training material. It’s helpful to invite some people who have absolutely no experience with online training. They can help identify issues that everyone else ignores. Common problems include defective menus, missing instructions that would make completing the training easier, or usability issues.

Beware inviting coworkers and friends to a pilot program.

Sometimes Talance’s clients suggest this as a way to involve stakeholders in the course development process. This is a problem because …

  • they’re not the intended audience, so they can’t give the correct feedback;
  • if they’re a stakeholder who should have been guiding the project, they should be involved at the beginning, not at the end. This can lead to missed deadlines and an extra expense of rewrites when the writing phase is over.

Assembling the wrong pilot group can give you false information.

One of our clients had such an experience prior to working with us. The program managers did organize a pilot (bonus points!), but they made two other mistakes.

  1. Their pilot group was too small–only about five people.
  2. Although the training program was meant for people new to the material, they invited people who were veterans in their role with up to 25 years’ experience.

When they reviewed feedback from the participants, they all complained that the information was too basic. This feedback dominated the evaluation forms because they only had a handful of participants.

What’s the perfect pilot group size?

A too-small group is as much of a problem as a too-big group. It makes sense to look for a group that’s similar to the audience you intend to be taking the course when it’s complete. You can also include a mix of experts.

We aim for a group of around 20-30, because that tends to be the final class size for most of the training courses we develop.

Choosing the ideal pilot group participants.

It’s best to begin any pilot by clarifying who the intended audience is. It can be helpful to create one or two personas for recruiting. Every person who participates in the pilot program should match the persona.

The learner persona can include how much experience they have and how much time they need to dedicate to testing. It’s also helpful to include what their motivations are for completing the training. The subject-matter expert persona can include familiarity with online vs. traditional learning and particular areas of expertise.

Once you define who your ideal learner is and set your group size, you’ll be in a great position to find out if your training program is working as intended. From this step, making tweaks is much easier and less costly. A little preparation, careful planning—and even an extra helicopter–means better projects that accomplish your program training goals the way they were intended.