Mobile reinforcement can make learning stick, flatten the “forgetting curve,” and boost the effectiveness of training—all while reducing costs.
If that sounds impossible, take a look at ECHO, an app created by SwissVBS; better yet, consider their results. The creators of ECHO, Shahin Sobhani, president of SwissVBS, and Brett Smith, the director of learning solutions, wowed attendees at FocusOn Learning 2017—and walked away with the award for performance support. (Short presentations on ECHO and other DemoFest winners are available in the Best of FocusOn Learning DemoFest webinar.)
Sobhani and Smith presented the ECHO app as a case study with General Electric’s Water and Power division. GE puts new sales hires through a two-day, instructor-led training course on selling skills. The company’s practice has typically included a benchmark test at the end of the two days, which measured learners’ comprehension and gave GE some idea of the instructors’ effectiveness.
Focus on learning retention
The SwissVBS team leveraged GE’s approach with an app that took learning to a new level, focusing on reinforcing the training and making it stick. To be effective, Sobhani said, reinforcement has to be effortful, which includes what he calls “interleaving” topics; it also has to incorporate delayed feedback, and it has to be based on a spaced learning model. ECHO incorporated these pedagogical principles and used the existing GE training framework to test—and demonstrate—its effectiveness.
Following the standard post-training benchmark test, groups of employees—just under 100 in all—were divided into test and control groups. Test groups used the app; control groups did not. Learners with the app received mobile push reminders to do the reinforcement exercises three times a week for four weeks. Each spaced learning session lasted about five minutes. Learners had studied nine competency areas; the ECHO app covered all of these, interleaving questions about different areas of competency and asking that learners apply the knowledge.
“In the case of reinforcement, first of all, there is a big assumption: that learning has already happened somewhere else,” Sobhani said. “What you’re trying to do is reinforce the person so that they will retain that information. Therefore, what you do with interleaving is, if you have taught someone A, B, C, and D, you put it all in a bucket, and the brain’s got to figure out how A, B, C, and D are connected.”
The SwissVBS team created a large pool of questions and tips, as well as flashcards, microlearning modules, and coaching podcasts that learners could review on their own. “When you ask questions, flashcards, tips—they have to figure out how everything is connected. And that exercise, which makes it effortful, helps reinforcement and retention,” Sobhani said. “Interleaving is actually one of the more powerful tools for reinforcement,” he said, though it is unpopular with learners, since it makes the exercises more difficult.
Each mini-session asked the learner to answer five challenge questions across the nine competencies. Learners received feedback appropriate to their responses; the “delayed feedback” aspect meant that feedback on all five questions came at the end, not question by question. Delaying feedback increases the effort required, Sobhani said. It’s based on an illusion that we all have—we think we know more than we do, he explained. When learners get instant feedback on whether they answered each question correctly, “it’s instant gratification,” but “if the brain thinks, ‘I got this one; I got this one …’ and then they find out no, they got one out of five right, the whole perception of the brain changes.”
“Delayed doesn’t mean months, weeks, days, or hours; it could be seconds,” he said. The added effort comes when the brain finally gets the feedback and has to go back and “see what the right answers were and why that was the right answer”—seeking the solutions, rather than having the right answer instantly appear.
Dynamic testing improves engagement—and results
In addition to answering challenge questions in ECHO, learners got coaching and suggestions for further resources. The app uses a dynamic test engine that collects and tracks learners’ responses—then targets coaching, resource recommendations, and future questions to their weak areas. “The questions that get sent to the learners are exactly the questions they need, based on where their skill set and knowledge retention is at that time,” Sobhani said.
The dynamic test engine is an element that Sobhani said was critical; with it, “the messaging and tips can become much more fine-tuned to the actual learner instead of being a general tip,” he said. The engine “allows the application to learn from the learner, so, as you continue to use the app, and it sees that you’re weak—let’s say there are five competencies that the organization is trying to make you learn, and it sees you’re weak in three of them—the tips, the modules, the flashcards, the questions it starts giving to you are going to be focused on your weaknesses.”
It seems logical that this approach would improve performance. What might be less obvious is that it also increases engagement. “What we’ve seen is that adoption has gone up because of that. If you get an app which is basically a polling app, and it just sends you a whole bunch of questions, and the next day it sends you a whole bunch more questions that are on the same topic that you already know, you’re going to start not using it anymore,” Sobhani said. “But if the app starts giving you stuff that you don’t know, the chance of engagement and adoption just goes up.”
SwissVBS is constantly working to improve the mobile reinforcement app. Three reinforcement sessions a week seem to be about right; many customers choose a Monday-Wednesday-Friday reinforcement schedule. “It’s nothing to do with delayed feedback, but giving that one day rest between reinforcement—the studies have shown that that helps as well,” Sobhani said. “What happens there is, the brain starts losing a little bit—memory leak—on that topic and then immediately, the next day, it gets the topic back. That makes it effortful; there is an exercise they have to do, and it forces the brain to see how all these things are connected.” Some clients have tried using reinforcement twice weekly, which works, but not as well, he said. “One time a week? You might as well not do reinforcement.”
Four weeks after completing the in-person training, both groups of GE employees—test and control—took a second benchmark test. Control group learners, who had not used the ECHO app, scored an average of 15 percent lower on the second test than they had on the first. But those who had used the app three times a week scored, on average, 20 percent higher than they had on the first exam—a net difference of 35 percent and evidence that, rather than succumb to the forgetting curve, those who used the app had actually improved their recall and understanding over the previous month.
A typical reinforcement program lasts four weeks, and some go as long as six, Sobhani said; length is partially dependent on the amount of content, but, he cautions, “You can only do so much in a learning reinforcement plan. … You’re only reinforcing the core concepts; you’re not reinforcing everything.”
It takes a few weeks to create a reinforcement program, so planning should be concurrent with designing and developing training; SwissVBS looks at the training date and works back from that to ensure that the reinforcement is ready to go the moment learners need it.
“The most important point about reinforcement is you have to—you have to—I don’t know how much more to emphasize it—you have to do it literally the day after class is over. The studies are showing that even waiting two or three days, you’re losing so much,” Sobhani said. “Therefore, a lot of our customers are seeing reinforcement as part of the entire ecosystem of learning and the learning journey. So it’s not three days of class and you’re done; it’s three days of class and 30 days of reinforcement—and then you’re done, then you get your certificate, then you get recognition that you finished this course.”
Sobhani is so convinced of the difference a reinforcement program makes that he believes there’s no point in investing in eLearning or training without it: “If you’re going to spend $200,000 on your learning campaign, but you don’t have a reinforcement campaign? Don’t do it! Don’t waste your $200,000; 90 percent is gone within a week!”
Analytics sweeten the pot
The mobile reinforcement program demonstrably improved learners’ retention of the training. But that was not the only benefit that their managers noticed. Included with the app was a manager dashboard with significant analytics data—a key component for GE. “It was the analytics that sold them on this,” Sobhani said.
The app targeted content to each learner’s weaknesses. The analytics data tracked individual learners’ performance and progress. It also provided aggregate data on the learner cohort, far beyond whether learners completed the training or answered questions.
Comprehensive data enables training managers to improve the training by looking at behaviors and patterns. “If 80 percent of your group got this competency wrong, it’s probably not them or the app; it’s the way you taught it,” Sobhani said. “That’s a behavior. Now you can adjust.”
Improved training clearly benefits the next cohorts of learners; current learners win as well. Prior to using the ECHO app, an entire cohort of GE employees might attend follow-up training sessions that covered all nine competencies—even if there were areas where they did not need additional training. With the data from the app, managers are starting to eliminate expensive training days, replacing them with coaching and one-hour webinars targeted to learners’ weak areas.
Now, the plan is to pair learners who excel at a competency area with learners who are weak in that area for peer coaching. If larger groups of employees share a weak area, a one-hour webinar can cover only the needed skill or information, reducing the need for longer, more general—and more costly—training sessions. An additional benefit is that problem areas are identified quickly, by studying learners’ progress data, whereas previously, GE training managers would react to mistakes employees made on the job over the several months following the initial training.
Putting business intelligence to work
“The analytics is very powerful at looking for behavior change and patterns. The retail industry has been using these analytics for years,” Sobhani said. “We’re just using the power of business intelligence in the learning space.”
Sobhani is excited about the success he’s seen with the SwissVBS app and the future potential. “Mobile reinforcement is so new that people don’t even realize that they need it. Except that they’re dealing with the crisis if you don’t have it: Why are people not getting this? Why are people making mistakes? How can they be forgetting it?”“It’s a very new thing in one respect,” he said, but not everywhere. “In the safety industry, this is happening constantly. Why is it not happening everywhere else?”