Evaluating Training ROI With a Learning Intelligence System

Alignment to individual and organizational performance

What differentiates a cross-functional learning intelligence system from most LMSs is the ability to align training with performance objectives for the entire extended enterprise, including individuals, the organization, and its business partners. The learning intelligence system can combine the course completion, certification, and assessment scores of the LMS with the evaluation and competency data in a performance management system. Likewise, the system can combine LMS data with business results from other corporate systems.

Historical training and business data provides a good starting point for developing a statistical training model that will identify the training programs that had the greatest impact on individual and organizational performance. After developing this initial model, the organization can apply the model to current data to quantify how training affected performance.

Reporting and analytical tools

Flexible reporting and intuitive user interfaces are also keys to making the data available and easily accessible. Portal applications can provide reporting tools that training professionals can use to leverage their statistical training models. This allows for more informed decisions when designing and implementing a training program. Some possible tools include:

• A training scorecard that evaluates training programs on ROI and other performance metrics
• Sales, manufacturing, distribution, customer service, and other scorecards that provide performance metrics specific to each domain, including training
• Ad hoc reporting for sophisticated and quick information retrieval to meet pressing business requirements
• A predictive analytical tool that allows organizations to allocate training resources to achieve desirable organizational performance

For example, a training scorecard application can track ROI metrics, such as sales increases as a result of course completions, assessment scores, or certifications, and it can maintain performance accountability. The training scorecard becomes a much more powerful tool to manage interdependent activities and performance if it has the following features:

• Drills down to supporting data detail for ROI scores
• Provides a breakdown of performance scores for a wide range of training attributes, including curriculum, location, instructor, delivery method, and others
• Accounts for regional economic differences, and other environmental factors beyond the organization’s direct contro

With an easy-to-use “drill-down” capability, training professionals can identify how cost and performance results contribute to a training program’s ROI score. Providing this type of analysis, however, depends on a statistically validated model of cause-and-effect.

Developing a model for learning improvement

Physicians study the signs and symptoms revealed by medical tests and prescribe medicines, diets, surgical procedures, or exercise programs to their patients. Taking the medicines in the prescribed dosages, following the recommended diet, and doing the exercises are the critical behaviors that help patients improve their health. If the medical tests were faulty, the signs and symptoms would be wrong, and the doctors would be unable to make proper recommendations.

In medicine, physicians can properly read the signs and symptoms uncovered by tests and other diagnostic tools. The tests and diagnostic tools work because they embody scientific theories of human anatomy and physiology that explain biological interdependencies. To develop the tools that will help achieve higher ROI, training professionals need a model of training for their organization similar to the diagnostic tools used by physicians. A model of training would identify the critical learning variables in all areas of interest and show how those variables affect business results.

A training model based on multivariate statistical analysis provides the necessary — and often missing — basis to reduce variation and improve training processes. (See Figure 2.) Multivariate statistical analysis is a collection of procedures which allow measurement and analysis of multiple variables simultaneously. This is one of the disciplines behind methods used in industry to improve quality, such as Six Sigma and structural equation modeling. I believe this model and these industrial methods are applicable to training.

Figure 2 Multivariate statistical analysis: Given a set of data, statistical analysis can identify patterns in the data. These patterns can quantify how one or more variable(s) in the data set impact another variable in the set.

Evaluating effectiveness

Tracking training and business performance results is critical to achieving an expected ROI. However, when an organization simply measures without an understanding of interdependent cause-and-effect relationships, it does not accurately evaluate training effectiveness. Often, people make inferences about simple causal relationships, focusing on a single cause and effect.

For example, good or bad sales may be the result of general economic conditions and other factors. A company may achieve better sales numbers following a sales training initiative even if the training itself was deficient. Tracking results does not necessarily evaluate how training modified sales staff behavior or ability. There can be a distressing disparity between training metrics and the information that actually helps an organization leverage training to improve business results.

Multivariate analyses can provide a tool to help organizations evaluate and quantify training effectiveness. Specialized techniques, such as structural equation modeling (SEM), can enhance a decision maker’s intuitive understanding of the world with more precise multivariate analyses. In this way, the decision maker is better equipped to propose and test broader models of the world. (See the References at the end of this article for some details and examples.)

This statistical understanding can become a predictive model that identifies how to invest training allocations and expenditures in order to have the greatest influence on training results or business results. Training results could equate to competency demonstrated by assessment scores or job performance evaluations. Reductions in assembly line rework and increased customer service satisfaction might be appropriate business results. Such metrics facilitate ROI calculations.

Training does not always produce the desired organizational results. Improving outcomes depends on recognizing failed training and its causes. The mathematical method I propose here not only helps ROI measurements, but it can also identify remedies for lower than expected ROI. By identifying the intervening factors that disrupt training effectiveness, it becomes possible to address the root causes for failed training.

Simplify interdependencies by reducing complexity

A complete ROI analysis depends on the causal connections between training and non-training data. Developing a robust training model that can make these connections may require significant data collection and analysis effort. Many training and non-training activities contribute to performance. Which activities does the organization attempt to manage and to what degree? Identifying which factors actually affect desired performance narrows the scope of activities that the enterprise must manage. This reduces both the complexity of the relationship and the administrative effort required to influence performance.

Statistical analysis of performance and training data and other information can help map the interdependencies of an enterprise’s training programs with other factors. The analysis can also identify the impact of those interdependencies on business performance. It is possible, for example, to produce a causal mapping that quantifies how the many-to-many relationships of a set of activities will impact overall organizational performance. Such a map or model can then help predict how modifying the training in a particular way will impact desired behavior, such as key performance indicators.

For example, suppose that an automotive manufacturer finds that different types of training yield different customer satisfaction results. One approach would be to create incentive systems and certification programs that encourage all dealerships to allocate resources and effort in those areas that will positively affect customer service.

By offering the incentives and certifications to all dealerships, the manufacturer reduces the amount of variability across the entire dealer channel. For example, the manufacturer may mandate that dealership service managers complete specialized training that maximizes customer satisfaction more than other types of training. The training model both provides a justification for service manager training and it identifies an area for improvement. The entire organization, from the manufacturer down to the individual service manager, develops a greater confidence in how training will impact performance.

Example: Analysis of dealership evaluation

One automotive dealership study contained a detailed analysis of an evaluation system that an automotive manufacturer used to measure dealer compliance to the manufacturer’s standards. The evaluation system measured more than a hundred items, including cleanliness, size of signs, whether the waiting room had fresh coffee, and a wide range of other factors. Independent of the compliance scores, the study obtained eight dealer performance measures, including unit sales, part sales, market share for two different vehicles, and customer satisfaction with the sales, parts, and service departments.

After applying statistical analysis to this data, only seventeen of the measured items in the dealer evaluations system showed consistent relationships with business results. Of those items, certified training was an item that had a positive impact on the dealer performance measures; however, non-certified training did not appear to affect dealership performance. In this case, certification could demonstrate a measurable ROI based on the cost of the certified training programs, and on the business result improvement, as quantified by the statistical analysis.

Structural equation modeling

Structural equation modeling (SEM) provides a statistical method to develop a causal network of exceptional service elements and to quantify how each relevant variable affects service. This advanced statistical technique studies the simultaneous impact of several independent variables on a specific outcome variable. Each outcome variable of significant interest would have its own model. For example, if the goal of training is improving customer service, there would be a model for the outcome variable that captures the effects of the improvement, the customer satisfaction index. Structural modeling involves four steps:

1. Set initial model according to expert judgment.
2. Collect data for all variables. Data may already exist in company databases or may be commercially available.
3. Run the statistical software to estimate the model parameters.
4. Review and revise model according to data results.

The result of this structural modeling process is a path diagram (see Figure 3) showing which variables cause changes in other variables. This diagram could represent the causal network of interrelated factors in the dealer channel that affect customer service. Not only do training professionals get an idea of the variables that affect customer service directly or indirectly, they also get coefficient values that quantify how the variables affect one another.

Figure 3 Caption: Dealership Service Training: How specialized service manager and technical infrastructure training can affect customer satisfaction.

For example, an initial model for automotive customer satisfaction (defined in the first step of the SEP process) may include a range of potentially relevant data. A large amount of data is available from third party dealer management systems, and from the evaluation systems that an automotive manufacturer’s field organization uses to score dealerships on a wide range of metrics. Statistical software tests the initial model. Some variables will have an effect on the customer satisfaction outcome and others will not. The next step is to retain the variables that appear relevant, and then to run the model parameters again.

The emerging body of knowledge on transfer of training (see the References at the end of this article) suggests a number of factors that can affect training effectiveness. For example, the transfer “climate” can have a powerful impact on the extent to which learners use their newly acquired competencies back on the job. Delays between training and actual use on the job directly relate to skill decay. In addition, social, peer, subordinate, and supervisor support all play a central role in transfer. Finally, appropriately designed intervention strategies can improve the probability of transfer. These factors would be good starting points for inclusion in the initial iteration of a structural equation model for training programs.

Conclusion: A proposal for continuous improvement

In the education and training field, it is not unusual to hear criticisms of ROI measurement as being an instrument of justification, rather than of performance improvement. However, by using a robust learning management platform and applying statistical methodologies, ROI calculations can support continuous improvement of instruction just as they do in other activities.

The question is, “How?"

In manufacturing, final inspection was the dominant model for quality control until management understood that it makes more sense to focus on the processes. They learned that it is easier to locate the true cause of a failure, or potential failure, in the process rather than in the final product.

Manufacturers found a method to improve product quality. You may have heard of it. Six Sigma methodologies measure process quality with statistical procedures in order to continually improve manufacturing processes. Six Sigma methodologies significantly improve quality and reduce variability from one unit to the next.

In the same way, by selecting those measurements that can support valid inferences about the effectiveness of programs, learning and training professionals can know where to improve and how to allocate resources and effort.

Adopting a Six Sigma-like method for training would improve every program’s influence on business results

References

Kline, R. B. (2005). The principles and practice of structural equation modeling (Second ed.). New York, NY: Guilford Press. MacCallum, Robert C. and Austin, James T. (2000). Applications of Structural Equation

Modeling in Psychological Research. Annual Review of Psychology, Vol. 51: Page 201 — 226. Salas, E., and Cannon-Bowers, J. A. (2001) The science of training: a decade of progress. Annual Review of Psychology, Volume 52: Page 471-499.

Comments

Login or subscribe to comment