Your Source for Learning
Technology, Strategy, and News
ARTICLES       RSS feed RSS feed

Case Study: How a 3-Year Project Led Us To Scenario-Based Course Design

by Mike Dickinson

July 26, 2010

Feature

by Mike Dickinson

July 26, 2010

“Writers should make scenarios compelling by using suspense and, if at all possible, real-life dilemmas that aren't trivial. In fiction, that is done through exaggeration and conflict. With real-world training you can't necessarily do that, but you can uncover the tough dilemmas that give people a great deal of trouble.”

We are now in the third year of an intensive annual compliance-training program for employees of our company, which manufactures and distributes electric wheelchairs, mobility scooters, wheelchairs and motorized chairs. Most of the training has been online, and sometimes in the form of a game show using third-party software to make instructor-led delivery much more fun and engaging. An experienced Medicare compliance consultant labeled our online training the best he's ever seen. I think he was very generous, but, hey, we'll take the compliment!

We have faced several challenges, including quickly evaluating and mastering authoring systems and tools, meeting prescribed learner-contact times (three hours per year), and tracking and reporting completion, just to name a few.

In our most recent round we used a more scenario-based approach, and as a result we have learned some lessons about scenario design, independent of the actual authoring system. Eventually we were able to devise a template to elicit scenario-worthy content from SMEs.

We are a company of about 2,500 employees with annual revenue expected to be $450 million in 2010. Geographically, our home office is located in south-central Texas where we have our sales call center and corporate offices – about two-thirds of our employees. The rest of our employees are spread around the country, operating out of their homes or local distribution centers or retail stores.

Many of our customers are of Medicare age (64+), so we are considered a Medicare provider. Thus, Medicare compliance has always been an integral part of our policies and training, including a Compliance Department that conducts continual internal audits and review boards. In 2007 our compliance program was enhanced significantly, as was the associated training.

Year 1: Approach

The initial requirement was to design, develop, and deliver compliance training to all employees within 120 days due to the newly enhanced compliance program. Employees were required to take two to four hours of training depending on their job. We chose online training as the prudent way to reach all employees nationwide in a relatively short time.

Tools

Before this requirement arose, I had been disenchanted with the authoring system we were using at the time. It was robust and had lots of flexibility, but it was difficult to learn and took a lot of effort to build engaging interactions. So I had been keeping a continuous watch for an authoring tool that would do more of the heavy lifting, enabling us to focus on design while it provided more out-of-the box interactions. Enter Articulate Presenter.

We had just purchased the Articulate Presenter suite when this concerted compliance-training project began, and it was just the tool I needed! (I was an e-Learning shop of one at the time.) Articulate was based on PowerPoint so I could build screens quickly. It had just enough built-in interactions with its learning games and synchronized animations to add visual and aural interest. In addition, Articulate made narration very easy to do. This was important because we elected to use narration and forced navigation as the way to ensure each learner spent the minimum required time in the training. I don’t like doing that, but we saw no other viable solution.

Content development and delivery

We broke “compliance” up into sub-topics that included federal laws and guidelines, details about our enhanced compliance program including the code of conduct and internal policies and procedures, and other Medicare compliance resources.

Articulate Presenter comes with a few learning games that we used at frequent intervals to add some fun while reinforcing key points. A favorite was the word game that is similar to hangman. (See Figure 1.) The learner clicks letters to fill in the missing word, and the author can set a time limit for each question.

 

Clue: Aswer, question and answer reveal Tile game

Fig. 1: Sample Articulate word game slide

 

The learner gets audio applause if they answer correctly and a “raspberry” sound if they miss a question. I was stopped in the hallways many times by employees telling me how much they enjoyed that applause! For new hires we sometimes ran the course in a group setting, rotating drivers of the mouse among classroom participants. I can tell you that the applause along with the desire to not get the raspberry sound or let the time run out were strong motivators to answer each question quickly and correctly!

Course structure

One of the decisions we dealt with at the beginning of this project was how to package the online training. Should we have a couple of two-hour courses, or break it up into smaller pieces? At the time, our LMS wasn’t bookmarking reliably enough to have confidence in it, so we created eight 30-minute courses. This gave learners a sense of forward movement, but it also created some confusion, and it made tracking and reporting a huge chore.

How did it go?

The heightened awareness of compliance was noticeable, and course critiques (Kirkpatrick Level 1) were very favorable. Thank goodness, because I felt those 120 days of development drew on all the skill and cunning I had developed as an instructional designer over the years!

One interesting item about training evaluation: What would be a good Kirkpatrick Level 3 or Kirkpatrick Level 4 measure for this compliance training? Would you expect fewer compliance incidents, or more? In our case, the number of calls to the internal compliance hotline increased noticeably in the weeks right after the training. We viewed that as a positive response. The training stressed the importance for the company of being a self-correcting organization, it clearly described the process for using the hotline, and it reiterated the policies of anonymity and non-retaliation for identifying potential issues to the hotline. Employees took that to heart and became much more proactive than they had been before, to include self-reporting if they realized after the fact they may have committed a compliance error.

Year 2

We ran some refresher training near the end of Year 1. At our corporate headquarters, we conducted several standup sessions in a large classroom, delivered in person by our Corporate Compliance Officer (CCO). While employees valued the chance to hear his insights and ask questions firsthand, we also saw some drawbacks inherent in the live presentation format: learners could be passive and the time commitment became a burden for our CCO.

Content development and delivery

For Year 2, we wanted to try something different for certain groups. We ran online training for those in the field and for certain groups at the corporate office, much like our initial training, but with updated content. However, our largest group of employees is also a group with daily exposure to compliance risk: our inside sales and support staff. They spend all day on the computer and phone, and they tend to be extroverted and competitive by nature. We wanted to appeal to these traits while reducing the burden of a two-hour chunk of training, and at the same time we wanted to help them internalize the training.

Tools

After searching the internet again, I found a wonderful program called Game Show Presenter by Tom Bodine (no connection to Articulate Presenter). Game Show Presenter, which you can see in Figure 2, can be purchased to run in a group setting and/or online. In order to run the game show we needed not only the program, but also a set of recommended buzzers, a laptop computer, and speakers.

 

grid with learning topics and points

Figure 2: Sample game show round

 

We had a fortunate mix of circumstances: Our sales teams have ten people each, and we had a conference room very close to the Sales work area that would hold four teams. We came up with a “passport” scheme called Destination: Compliance. It consisted of three parts: a one-hour game show competition, a facilitated discussion within each sales team, and an online quiz. We scheduled four teams at a time for the live game show and gave the winning team a commemorative tee shirt. You wouldn’t believe the spirit this generated! You could hear the game show from way down the hall.

Course structure

By its nature, the game show consisted of standalone questions. With help from our Compliance Department, we developed these questions carefully to include the feedback and any elaboration. (You can display a debrief screen after each question if you like.) In addition to factual questions, many were mini-scenarios, eliciting what an employee should do in a given situation.

Each team rotated their players through the buzzer hot seat for full and equal audience participation. Teams were encouraged to coach their hot-seat person and that sometimes got very lively with hardly anyone staying in their seat! We used the online version of Game Show Presenter for make-ups.

The game show was a big hit, and again, a piece of software helped make us look like heroes. (We were now an e-Learning shop of two, plus one intern who subsequently became an employee.) Other departments have used Game Show Presenter as a fun way to do refresher training, and we’ve started using it in one of our entry-level courses. You can always tell when it’s in use by the noise filtering out of the room and down the hall.

How did it go?

We went into the Year 2 round of refresher training wanting to reinforce numerous details about compliance and our enhanced compliance program. So we used the same questions for the game show and the online quiz (with answers randomized – remember that this is a “Jeopardy”-type game).

As teams arrived for the game show we gave each person a note-taker that had all the questions and responses, without the correct answers marked. Looking at the group from the front of the room during the game, everyone was not only engaged but intensely so, note-taker and pen in hand. They made sure they knew the correct answer to each question! Scores on the online test were very high, which was our goal. We weren’t out to stump people; rather, we wanted to be sure they knew the material well.

After employees finished all three parts of their Destination: Compliance journey we asked what they thought. Figure 3 shows their responses.

 

horizontal bar chart

Figure 3: Participant responses to Level 1 evaluation of Destination: Compliance

 

We tried one other thing in Year 2. For a company-wide piece of the refresher training that was online, we wrapped a scenario around the online training, creating some suspense early in the training, alluding to it throughout, and not resolving it until the end. This became a precursor to the approach we took in Year 3, based on the feedback shown in Figure 4.

 

horizontal bar chart

Figure 4: Evaluation of blended approach

Year 3

In the past three years our compliance training has followed a logical progression:

  • Year 1: Initial orientation to our enhanced compliance program including the laws and official guidelines to which we must adhere, and our associated internal policies and procedures.

  • Year 2: Refresher on Year 1 topics with additional emphasis on our disclosure program, its value to the company, and its protective elements such as non-attribution and non-retaliation.

  • Year 3: Quick refresher of basics, then focus on job-specific compliance risk areas that need clarification or emphasis.

For Year 3, we wanted to portray situations employees could find themselves in and help them diagnose and avoid those risk areas. Thus, the job-specific part of the training would be scenario-based.

This raised a few strategic questions:

  1. Which scenarios should we include in each functional area’s course?

  2. What structure should we use for the scenarios? How could we get scenario-worthy content?

  3. How many scenarios would we need?

  4. How could we get scenario-worthy content from each of several functional areas?

Which scenarios should we include in each functional area’s course?

We met with upper management in each functional area to ask which if any compliance issues kept them awake at night. What did they feel needed to be addressed in the training? We also met with our Compliance Officer and his staff, with mid-level management, and with SMEs who had shown especially high commitment to compliance. These discussions helped us identify the key risk areas to address for each functional area.

What structure should we use for the scenarios?

Based on other scenarios we had seen and ones we’d developed ourselves, we wanted each scenario to have four parts:

  • Brief narrative story line (one or two paragraphs) that leads up to a moment-of-truth decision point.

  • A question asking what was/is the right thing to do at that decision point.

  • Analysis of the scenario to include practical application of pertinent laws and guidelines, and company policies and procedures.

    • Note: Rather than use an end-of-course quiz, our strategy was to use embedded questions throughout each scenario that the learner must answer correctly in order to proceed. Together with the scenario “wrapper,” we felt this would provide a painless way to deliver content while holding the learner’s interest, much like the evening news that asks, “Will it rain tomorrow? We’ll get to that in a few minutes, but first, this…”

  • Brief recap to include identification of traps and warning signs along with the right thing to do.

How many scenarios would we need?

We estimated that a two-hour narrated course needed a script of about 16,000 words. This estimate was based on samples from several previously narrated courses.

We found that our average narrator spoke at the rate of 2.3 words per second. That works out to about two minutes per typed page at font size 12, double spaced, normal margins. We then wrote a few complete scenarios and found they averaged about ten minutes apiece (five typed pages). After allowing for a front end that would be common to all the departments’ courses, and learner reflection and reaction time, we concluded we needed about eight scenarios per course.

Part of our concern was whether we could produce the needed content and meet the deadline with this approach. By changing the context but otherwise reusing some of the scenarios, and by writing some scenarios that addressed certain interaction between departments, we were able to leverage some of the content and gain efficiency.

How could we get scenario-worthy content?

When we initially asked for compliance inputs from various SMEs, we received input ranging from some pretty good situations to lists of topics. So how could we get the detail we needed for the scenarios? Translation: how will we ever get this project done on time and what will it even look like?

Starting with the four-part structure given above, we developed a template for the way each scenario would unfold. In addition to providing the structure for the training itself, this template served as a useful authoring and interview tool for eliciting content. Table 1 is the original version.

 

Table 1: Scenario Template – short version

(Topic or risk area and brief description of the situation

 

One paragraph that describes the situation up to the pivotal decision. Write this like you were telling a friend a story.

Who, what, when, where, how, maybe some of the why (choice between two compelling courses of action)?

 

Did ____ do anything wrong?

 

What options could ____ have used instead?

 

Warning Signs

 

 

Pressures or hindrances?

 

 

Correct and incorrect alternatives

 

What risks do the wrong choices create, for the company, individual, and customer?

 

 

We had mixed success with the template, from eliciting wonderful content to only lists of topics. Here is the reaction of one instructional designer:

“My experience in using a scenario-based template was overall a positive one. I think the “formula” approach really worked in this case. There is so much material to cover that setting learners up with clear expectations makes it all easier to digest because they already have a structure in mind.

“…I’d say the first scenario took at least 24 man-hours to write including all the re-work. The eighth scenario was probably done in about eight man-hours. So the iterative process definitely got more efficient.

“…I might think twice about this approach [on future projects] just because it would need to be worth the effort.”

Email from Marilyn Sibblies dated 3/25/2010

 

Some, especially those with no knowledge of the content, found the process to be daunting even with the help of the template. Our productivity got a boost when we divided the labor according to each teammate’s strength:

  • One person gathered the content.

  • One person did the online authoring.

  • One person recorded and edited the audio and performed initial testing of finished courses. (For audio recording, we prefer Audacity; we then import those files into Articulate or our other authoring system, ToolBook.)

In order to meet our deadline, we identified some other efficiencies. By expanding the initial scenario template (Table 2), we were able to create online course templates even as we were still developing content. Once we had the paper template populated for each scenario, it was a rather simple matter of copying and pasting into the online skeleton.

 

Table 2: Scenario Template – long version

1. Scenario:

Description of the situation as told in linear sequence from principal characters’ viewpoints.

The storyline will go to the point where an infraction was discovered or averted.

Try to write actual point of discovery as it might occur in real life: who, what, when, where, how.

 

A multiple choice question at the point of infraction. Did an infraction occur? Or, did the person choose to do the right thing? Each choice has remediation feedback.

 

Analyze situation with “Compliance Advisor” as guide:

  • Who committed or avoided the infraction?
  • What was the ethical dilemma or error?
  • What risk area does this scenario represent?

 

From a compliance perspective, what is the risk to:

  • The Customer
  • The Employee-Owner
  • The Company

 

Have any recent Compliance publications addressed this situation? If so, reinforce them here and show their applicability.

 

Remind learners of the applicable Medicare P&Ps, Code of Conduct, etc.

 

What action should the person or persons take now?

What should we expect to happen as a result of this action?

 

What were some warning signs? How could you avoid making this mistake going forward?

 

Concise summary of the scenario and lessons learned

 

 

Another efficiency measure: Some risk areas pertain to several departments but the context is slightly different. So we were able to reuse several scenarios by changing the setting but leaving the underlying instruction relatively unchanged.

How did it go?

We met the overarching requirement: Everyone who was required to take this round of refresher training did take it, except for bona fide absences such as FMLA (Family and Medical Leave). (Using the LMS to create and manage target audiences is beyond the scope of this article.)

As far as Level 1 evaluation, the scenario-intensive format was not quite as well received as previous compliance training, although a higher percentage of learners felt the objectives were met. Here is a comparison across the past three years:

  • Percent who said the training was worthwhile/valuable

    • 2010 scenario-based: 82% (n=623)

    • 2009 group game show: 92% (n=154)

    • 2007-8 presentation format with interactivity: 93% (n=175)

  • Percent who said the course objectives were met:

    • 2010 scenario-based: 96% agreed

    • 2009 group game show: 84% agreed

    • 2007-8 presentation format with interactivity: didn’t ask (this was our mistake)

Written and verbal comments suggest that the Year 3 scenario-intensive training created a more bi-modal reaction than previous approaches. Some employees got a lot out of the scenarios whereas others felt the scenario structure became too repetitive. Written comments did suggest that some employees felt more empowered to take the correct action after seeing it played out in a scenario. Finally, I think the fact remains that some people like “edutainment” (the learning games and game show that we did in previous years).

What about our time estimate?

Some learners took considerably longer than the required – and intended – two hours. Some of these slowdowns were the result of technical issues with computers or networks. Others were the result of variability. A slight increase in average screen load time when multiplied by a few hundred screens per course could add 30 minutes or more to the course length. So could learner response time; for example, the difference between a person who blasts through the course vs. one who takes time to reflect on several screens.

Lessons learned/conclusions

Possibly the most important lessons learned had to do with developing scenarios. I have seen some scenarios done very well by third party vendors. Trying it as an internal developer, I discovered some key points.

First, using principles of fiction writing, writers should make scenarios compelling by using suspense and, if at all possible, real-life dilemmas that aren't trivial. In fiction, that is done through exaggeration and conflict. With real-world training you can't necessarily do that, but you can uncover the tough dilemmas that give people a great deal of trouble. You can also find out the situations for which the employees keep asking for clarification and yet don't quite get the detail they need. One example for us was a compliance and marketing no-no: cold calling. Although I can't take credit for bringing it about in this case (I checked and my name never came up in the discussions), we were pressing for clarification on some of the “gray area” situations people encountered when a very well-written policy came out of our Compliance Department that described exactly what an employee can and cannot do in those situations, with good examples to accompany the rules.

If you are writing management or interpersonal scenarios, you can include some of the real-world conflicts that have no clear answer, instead of stopping at a superficial level.

Second, we chose to use a series of still photographs to accompany guest narrators' talks. Even for me, as a serious photographer, this proved more challenging than I expected. It was difficult to get good pictures of each person and to make their expression coincide with what they were saying. It takes really good lighting. It takes good directing. I recently read an interesting book by Ron Schick, telling the lengths Norman Rockwell went to in directing his models to pose for the message he was trying to convey in his paintings. He would even spend considerable time getting them into the mindset of the situation so their face and body expressions were what he was after. Unfortunately I read that book after I had taken all the photos for these courses!

Third, we nearly always use internal talent because it's “free.” I usually prefer this to the obviously professional voices you get with paid actors, but at the same time it's harder to convey realistic emotion with inexperienced talent.

Our conclusion about writing compelling scenarios is that it's difficult to do if you want them to go beyond the superficial. It takes time to really know the target audience's pressures and dilemmas and to write situations that will grab people's interest while offering those “ah ha” moments of insight.

Finally, here are the “process” lessons learned.

  • Courseware is software; it needs thorough testing
    • Some learners experienced delays because we missed one navigation error that was duplicated when we reused similar scenarios.
  • The act of developing training may show the need for clarified policies and procedures. The scenario approach exacerbated this:
    • They can’t say that? Where’s the written guidance? How may we describe the line between what they can and cannot do?
  • Project and sponsor management:
    • Identify not only the designated approver but also who’s got veto power. Try hard to get their input early including key topics they want the training to address and buzzwords they want to include or avoid. When you’re using photography and audio this can save a lot of rework.
  • Audio and photos:
    • We’re a very personal and close-knit company, so employees enjoy seeing and hearing narrators that they know. We used all internal voice talent, usually the affected department’s management or staff to avoid taking front-line employees off the job.
    • This makes it relevant and engaging even if the chosen talent doesn’t have the best radio voice
    • Narrators downplayed it, but they enjoyed the star status and occasional ribbing from their constituents.
  • Tool selection:
    • The Articulate Presenter suite lets you focus on development without having to create all the interaction and basic animation yourself. I recommend it for its reliability, simplicity, and built-in interactive features, when you don’t need more complex customization.
    • Audio: We used Audacity software and Plantronics headsets (with laptops). With this combination I have recorded in offices that had a lot of extraneous noise or that were near slamming doors or people talking in corridors, and none of that background noise was picked up.

References

Kirkpatrick, Donald L. 1994. Evaluating Training Programs: The Four Levels (1st edition). San Francisco: Berrett-Koehler Publishers.


Schick, Ron. 2009. Norman Rockwell: Behind the Camera. New York: Little, Brown and Company.


Poised and Positioned for Performance Conference, Nov. 6, 2009, San Antonio, featuring Dr. Donald Kirkpatrick, sponsored by local ASTD chapter.



Topics Covered

(65)
Appreciate this!
Google Plusone Twitter LinkedIn Facebook Email Print
Comments

Login or subscribe to comment

Thank you so much for sharing this!
You're very welcome!
Just yesterday I read an article, Creating e-Learning That Makes A Difference by Ethan Edwards of Allen Interactions. This case study provided examples of what the "Makes a Difference" article recommended! Thank you for sharing what you did, how it worked and participant responses. Very encouraging!
This is a great article! It both affirmed what I've seen when developing scenario-based online training, and provided me with some things to think about for future courses. Thanks for sharing!
Excellent summary of process & impact. We used a decision-tree set of scenarios as an add-on to the ethics training provided for Dell employees. At first, there was concern about the time spent working through each scenario, but in practice, learners stayed more engaged and felt they spent less time on the scenarios than on the other sections -- altho the reverse was actually true! People will spend the time willingly if they are actively engaged in applying the lessons learned.
Excellent article. And timely too.
Really appreciate the effort that you took to put together this article. You have helped me learn in 15 minutes what you learned in 3 years! Thank you!
Related Articles