Peter Drucker famously said, “What gets measured gets improved.” Yet to be fair, not everything that matters can be measured. Can we really quantify an employee’s commitment to the organization’s purpose? Do we really want to weigh every positive statement a coach says to a performer? Probably not. Instead, we have to choose what to measure. And that’s what matters.
The difficulty of measuring L&D
In many organizations the learning function is measured in terms of efficiency. You’ll see summaries of how many employees are served per learning-function personnel. Or cost per hour per seat. This information is important, but is the hour in that seat benefitting the business? That’s the first question!
There’s much made about return on investment (ROI), but I think that also can be misleading. If you save $20,000 by an investment of $1,000, you are getting a 2,000 percent ROI. However, if you save $2 million by investing $500,000, your ROI is only 400 percent. But isn’t a $2 million savings more important? It’s more about targeting the big costs in the organization—the areas where performance can be improved—figuring out what it would take to improve it, and prioritizing what’s business-critical.
The Kirkpatrick standard
Implicit in that discussion of ROI is that you’re actually measuring the impact on the business, which isn’t common. Using the old standard, Kirkpatrick, a survey from a few years ago showed almost everyone doing Level 1, but it dropped off to a third at Level 2, which is still about what they do after the learning experience, not what’s happening in the business. It dropped to one-eighth at Level 3. That’s people doing things differently in the organization as a result of the intervention, which could be courses, job aids, etc. And essentially zero (with a rounding error) when going all the way to Level 4. Granted, the survey is several years old, and I think the numbers are slightly better today, but they’re still not good.
The rationale for this isn’t completely hard to understand. It requires one or both of two things: data from the business unit, and/or data beyond course completions. And both require a change; in behavior or technology. The problem is: an LMS isn’t really a learning management system; it’s only a course management system (yes, new ones have more functionality, but that’s their core DNA). And course completions are, at best, Level 2. How do we go further?
Structuring a new model of measurement
We can (and should) be talking to the business units we serve. When they ask for a course, we should be asking, “What’s the evidence that there’s a problem, and how will we know when we’ve solved it?” We need data on what the gap is between desired and existing performance. And we can then infer a cost: Moving from 10 percent errors in manufacturing to five percent saves us five percent on returns. The annual cost of returns is $400,000, so we’re saving $20,000.
We also can have finer-grained data; we can track (via, say xAPI) what resources people touched, host that data in a learning record store (LRS), and then correlate with business intelligence software. Or we can integrate a proprietary system to connect our learning and performance interventions with that business data. We need to measure. But we need to be measuring the right thing.
Measuring impact
It’s not just about L&D efficiency. It absolutely has to be about our impact. It’s been comfortable to coast along ensuring that our costs are no worse than industry average, but eventually someone is going to want to determine whether the L&D investments are good value. And as long as L&D operates as a cost-center, an expected investment with no real data on the efficacy, it will be the first victim in cost-cutting.
Any intervention—learning and performance support resource portal, collaboration and communication platform, even courses—should have a viable business proposition about how the cost is going to result in net benefit to the company. Further, those investments need to be prioritized in terms of that benefit.
In conclusion
Measuring L&D may be hard, but it’s necessary. While instituting the discipline can be initially onerous, the long-term result is better alignment with the organization, and demonstrable contributions. And that, ultimately, is what a performance ecosystem is supposed to achieve. It’s doable: we have some very clear principles and practices, so there’s no excuse. It’s time to measure your movement towards measurement. Are you ready?