A.D.D.I.E. Meets the Kirkpatrick Four: A 3-Act Play

Evaluation

When it was time to evaluate how well participants had mastered the learning content and how effectively the training program had taught that content, the Kirkpatrick Four and ADDIE were home free. Participants rated the program as highly useful and valuable on Kirk One’s reaction sheets. All participants passed Kirk Two’s learning tests with flying colors — and since the organization wanted every employee to succeed, no one mentioned bell curves. About two months after the program, when Kirk Three asked the participants’ managers, co-workers, internal customers, subordinates, and, when appropriate, external customers about on-the-job performance, the graduates of the program received top ratings.

When the evaluators reassembled to debrief, Kirk Four text-messaged, “Busy.” So the others wrote up a project report, summarizing their work in Table 1.

Act III

[ADDIE’s office, hit by an avalanche of papers and files, covering her desk, desktop computer with attached speakers, camcorders, and earphones as well as her laptop, an electronic whiteboard, and a large wall computer screen for videoconferencing.]

As ADDIE was sorting and filing mounds of leftover project papers, CDs, DVDs, and wall charts, she received a strange visit. Kirk Four, who had led the charge of the Kirkpatrick Four into the full design process, slumped into a chair in ADDIE’s office. Gone was the commanding presence and the air of executive confidence.

“I failed,” groaned Kirk Four. “I tried and tried to show how the improved performance on the job produced needed organizational results — senior management really wanted a high return on their investment in training but I can’t prove our training program had any impact on revenue, cost controls, retention of customers, or expanded market share. Nothing.”

“But our organization’s revenue is up and we’ve expanded our number of customers!” objected ADDIE. “Can you prove our course had anything to do with that?” Kirk Four asked.

“I can’t. If, instead, revenue had tanked and we lost customers, I’d hate someone to blame our course for that. I tried to base my findings on evidence, but I can’t figure out how to separate the impact of the course from everything else.”

“Like what?” asked ADDIE?

Kirk Four ran down the laundry list: “The impact of changes in the economy, competitive pressures, and the introduction of alternative technologies. All that occurred within three months of the end of our training program.”

“Hmm,” said ADDIE thoughtfully. “Luckily, our company just happened to show an increase in revenue during the first six months after the training program ended. But I see your concern. Maybe that happened only because the economy was driving more customers to us.”

“I’m useless,” gasped Kirk Four. “Your questions were important,” ADDIE said, consoling. “If we didn’t ask about senior management’s business goals, our course would have been much less relevant to real organization needs as well as training needs. Thanks to your questions, at least our instructional designers started talking with senior management, asking what their business goals are, and building them into the training programs.” Kirk Four shrugged, but looked relieved to have contributed something.

ADDIE suggested they go try the new mocha cappuccino. As they walked to the break room, she told Kirk Four that management had doubled the training budget for next year. “I think just the fact that we listened to their business needs and built them into our training program made senior management assume they achieved a solid return on their investment in the training.” 

They sipped in silence for a while. In fact, for a while they looked as if they had fallen asleep in their office chairs. Kirk Four broke the silence. “We could just mention in our project report that within six months after our blended e-Learning and onsite training program, revenue and market share increased. Let the readers connect any dots they want to connect — but I guess that’s cheating.” Kirk Four was becoming more and more depressed. “I fool so feelish — listen! I was such a big know-it-all and I can’t even talk straight. I feel like a foolish fop. Say that one fast.”

“Not foolish — Phillips,” muttered ADDIE, talking quickly as she excavated her keyboard from a pile of files then started typing frenetically. “Phillips and Phillips.”

“That’s not much of a tongue twister,” said Kirk Four. “Try ‘Foolish fop felt foolish when flagrantly flaunting fatuous ...’ “

“No, Phillips and Phillips!” interrupted ADDIE, typing in a VoIP connection and activating her videoconferencing software. She focused a digital camcorder at herself and Kirk Four. “Patricia Pulliam Phillips and Jack J. Phillips. ROI Basics! That’s who we need!!”

Suddenly a voice answered the phone with a “Hello, Jack Phillips — Hey, is that you ADDIE? What’s up” and the face of Jack Phillips appeared on ADDIE’s conferencing screen.

“We were wrestling with a challenge when I suddenly remembered that you and Patricia tackled it in ‘05,” said ADDIE. She introduced Kirk Four and explained that they had not been able to isolate the effects of their program. “We can’t sort the R.O.I. on our e-Learning investment from all the other influences on major business outcomes. I’ll loan Kirk Four my copy of your book, Return on Investment Basics after I finish re-reading it. But I just wondered if you could give her a few ideas today.”

“Have you thought about setting up a research design to show R.O.I.?” asked Jack. “You know — like drug testing. Have a control group that doesn’t get any training as well as a training group — like patients in a treatment group. Then you can document their results in hard cold numbers before and after the training period.” 

This suggestion made Kirk Four even more depressed: “That would be great. But we’ve already run the training. Besides, wouldn’t it be unethical to deprive some employees of the training? Sure they could take the course after the research, but they’d be way behind their co-workers ... especially bad for sales reps. We can’t set up a research project like that!”

“Jack — are you talking to the infamous ADDIE?!” Patricia Pulliam Phillips appeared on screen next to her husband. “If you say you can’t set up a research project to show R.O.I. for e-Learning, we’ve used another way to evaluate business outcomes over time — chart performance trends before training and for at least a year afterward.”

But Kirk Four shot that idea down too. “Thanks, but all the same ‘other influences’ would be at work on the trend lines. We still couldn’t prove that any increase in revenue or market share was directly related to our e-Learning program, rather than to corporate strategies, competitive activity, the economy, or hundreds of other variables. It’s hopeless. Sometimes I wish I’d never been conceived!”

“You’re not all alone in this,” said Patricia Phillips. “You need a brain trust. What about getting experts to estimate the effects of the training — people who understand the business performance measures. Even participants in your e-Learning program can serve as experts. If you set it up right, their managers can be part of your expert advisors too.”

Figure 5 ADDIE tries to hide her shock

When Kirk Four asked how that would work, Jack Phillips suggested setting up a focus group of participants after the training program. “They can estimate how much the training affected their actual accomplishments and how much those accomplishments affected the overall business goals.” ADDIE was a little concerned about having to take people away from their work to participate in a focus group, even for a video conference.

Patricia Phillips thought a moment then suggested another method: “You could make up a questionnaire about how the training affected the participants’ onthe-job accomplishments. And also ask how much those accomplishments affected the overall business goals.” ADDIE liked that one because people wouldn’t have to take much time away from work to participate. Kirk Four added that “Kirk One would really like that. It’s a structured way to link reactions with performance.”

Jack Phillips added to that suggestion. “You can also put together a group of the participants’ own managers and collect data from them about how much the training affected overall business goals.”

Kirk Four was no longer hopeless. “Maybe we can do all of this — conduct a focus group with about twelve participants, send the questionnaire to the rest, and conduct structured interviews of the managers. We’ll record the results. Analyze the numbers. Quote the comments. Produce a really credible report.”

“Credibility is the key. Revenue is up this quarter so many people will be fighting for credit for the company’s good results, and completely ignoring training programs. But if the economics of our industry start going downhill, those same people will be trying to blame training and will figure out reasons to cut our budget — or our jobs.”

The Phillips’ pointed out how important it is to be clear. “When you present your results, say they are your best estimate,” said Jack. “Make it clear they are based on the input of credible, knowledgeable business people who are themselves employees accountable to the management committee,” added Patricia.

“Right,” exclaimed Kirk Four, “We can also report the hard data about the increased revenue and market share, reduced costs, or whatever business goals the organization has achieved. As long as we don’t look as if we’re taking credit for it.” ADDIE agreed, saying “We can even say to executives: based on what you’ve heard from the focus group and the managers of the participants, do you think it’s possible that the training program accounted for even a small per cent of the revenue and market share gains?” Kirk Four added, “Yes. We can let the management advisory committee pick the percent. In some companies, even one percent impact could mean a major increase.”

ADDIE exclaimed, “Thanks!”

Kirk Four, sleeping on the office swivel chair, was startled awake. “I guess I dozed off. Thanks for  what?”

The videoconferencing screen was blank, not turned on. Piles of files still covered the keyboard of the desktop computer. ADDIE, confused, wondered, “What happened?!” As she looked from Kirk Four — still half asleep — to the blank video conferencing screen on the wall, she realized she had fallen asleep too. The entire conversation had been just a dream. She didn’t even know Jack and Patricia Pulliam Phillips ... had never spoken to them in her life. But she had read — or at least skimmed — their book. She rushed to the bookshelf, found a copy of Return On Investment Basics and plunked it down on Kirk Four’s lap. “We’ve got to read this book. And then we’re going to show exactly how our e-Learning program supported crucial business goals!”

Kirk Four flipped through the book, at first skeptical, then absorbed and suddenly started text-messaging the rest of the Kirk Four. “Meet in ADDIE’s office now. ROI rocks!”

CURTAIN

References

Grafinger, Deborah J. (1988) Basics of Instructional Systems Development, INFO-LINE Issue 8803, Figure 1: “ISD Model Featuring the ADDIE Processes.” Alexandria, BA. American Society of Training and Development Kirkpatrick, D.L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA:

Berrett-Koehler Molenda, M., Pershing, J.A., and Reigeluth, C.M. (1996) “Designing Instructional Systems” in R.L. Craig (Ed.) The ASTD Training and Development Handbook, 4th Ed. (pps 266-293). New York. McGraw-Hill.

Phillips, Patricia Pulliam and Phillips, Jack J. (2005) Return on Investment (ROI) Basics. ASTD Press Winfrey, E.C. (1999). “Kirkpatrick’s Four Levels of Evaluation” in B. Hoffman (Ed.), Encyclopedia of Educational Technology. Retrieved February 11, 2007 from http://coe.sdsu.edu/eet/articles/k4levels/start.htm

Comments

Login or subscribe to comment