The cost-effectiveness pane
In the previous section I discussed CA and EA. In this section I will be combining them to show the CEA results with the help of the Cost-Effectiveness Pane.
The Cost-Effectiveness Pane is a graphical representation of the changes in cost and effect of the e-Learning investment compared to the previous traditional training. It consists of a cost decrease dimension (y-axis) and an effectiveness increase dimension (x-axis). (See Figure 2.)
Figure 2: The four Quadrants of the Cost-Effectiveness Pane for blending or substituting an existing face-to-face training with e-Learning
The point of cost-effectiveness for the traditional training lies at the intersection of the cost and effectiveness axes. We can evaluate each alternative e-Learning investment in terms of the incremental change in both costs and effects. This incremental analysis results in the placement of the point for e-Learning cost-effectiveness into one of the four following cost-effectiveness quadrants:
- Reduced cost, increased effectiveness
- Increased cost, increased effectiveness
- Increased cost, reduced effectiveness
- Reduced cost, reduced effectiveness
The “Reduced Cost, Increased Effectiveness” Quadrant
If your e-Learning investment finds its way into this quadrant, you can increase effectiveness and reduce cost by blending or substituting your traditional course with e-Learning. Therefore, e-Learning is clearly the better alternative.
As effectiveness and monetary benefits are positively correlated with each other, we also know that in these cases the ROI of e-Learning is higher, without ever having calculated any monetary benefits.
For example, substituting e-Learning for a traditional new employee orientation in fast-growing enterprises often results in this: the huge economies of scale drive cost down. Also, utilizing multimedia can be a much more effective means of communication, as opposed to relying solely on traditional methods.
It is important that we can draw the same conclusions as above if the result is located on the borders of this quadrant as well. In these cases, either the cost stays constant and effectiveness increases, or the cost decreases and effectiveness remains the same.
I have often found that using e-Learning results in a cost decrease, while keeping effectiveness constant, Think, for example, of trainings for computer applications such as Enterprise Resource Planning (ERP) systems.
The “Increased Cost, Reduced Effectiveness” Quadrant
E-Learning investments falling into this quadrant are plainly not economically beneficial for the company. Compared to the status quo, e-Learning is worse in terms of cost, as well as in terms of effectiveness.
Even without looking into the monetary benefits of the two training modes we can conclude that an e-Learning alternative has a lower ROI.
If you are making an investment decision, you might think of simply postponing the project: e-Learning is still an industry where innovation takes place every day. As time passes, projects might make their way out of this quadrant by themselves.
The “Reduced Cost, Reduced Effectiveness” Quadrant
If you find your e-Learning project in this quadrant you are looking at a more complex situation. E-Learning may be deemed beneficial or not based on the net result of cost and effectiveness reduction. Further analysis is needed to clarify the result.
In order to get a clearer picture of your project you might decide to use a ROI analysis. Even if the ROI result is positive, there is still another issue to deal with in this quadrant. Whereas a CFO will not likely have a problem with a cost decrease, getting trainees to commit may be more difficult, especially if certain expectations regarding effectiveness already exist. If this is the case, one can use internal marketing strategies to help gain support for the course and better promote its advantages, thus protecting the investment.
Another way is to increase e-Learning effectiveness to a change of zero or greater in order to make e-Learning the beneficial strategy. Sometimes this is easily attainable: I was involved in a situation where an e-Learning project management course was far less effective than the course given by an internal trainer from the IT department. Upon further analysis, we found that it was the face-to-face trainer himself that raised effectiveness to an incredibly high level. Since he was such an exceptionally good trainer, the solution to this problem was to blend the e-Learning course with his face-to-face training. This blended solution successfully moved the training course into the “Reduced Cost Increased Effectiveness Quadrant”.
The “Increased Cost, Increased Effectiveness” Quadrant
If both cost and effectiveness are increased by e-Learning, further analysis is needed to justify the transition. If this is your first e-Learning investment it may be advisable to consider another one with more obvious benefits.
The question is: Is the additional effectiveness worth the increased cost?
The pressing issue with this situation is the cost. One way to make this an easier investment decision is by finding ways to lower the change in cost to zero. One could, for instance, use cheaper technology. Another way to balance the cost would be to increase your user base in order to make use of greater economies of scale. Increasing the user base is not only possible internally: if your project is to be tailored specifically for you, you may discuss opportunities with your provider to share the cost with other customers of this vendor. You may also choose to use an already-existing course instead of a completely customized one. Unfortunately, all of the aforementioned ideas, the last in particular, bear risks of problems with effectiveness.
If you determine that a course in this quadrant is indeed beneficial for the company, it may be more difficult to ensure commitment from the CFO than from the end-users.
Dealing with multiple measures of effectiveness
Originally, CEA was intended to compare several alternatives based on a single measure of effectiveness. Of course, many training programs produce several outcomes, requiring more than one effectiveness measure. In this chapter we will look at ways to deal with such cases.
An excellent way to deal with these cases is to conduct a separate CEA for each measure of effectiveness. The Cost-effectiveness Pane would then be extended by these additional measures of effectiveness. If all of the points for each alternative are located in the same quadrant then the interpretation remains the same. It becomes more difficult to analyze when this is not the case. Fortunately, the evaluator is still able to clearly present the results and describe the related trade-offs.
Another method is to combine the different measures of effectiveness, weighted by their relative importance, into one single measure of utility. This approach is called Cost Utility Analysis (CUA). The problem with this approach is that it can easily become very complex, even convoluted.
Finally, there remains Cost-Benefit Analysis: in this approach, all outcomes are translated into monetary values. The popular training ROI is actually one form of Cost Benefit Analysis. Jack Phillips refers to this as an additional fifth level of Kirkpatrick’s four-level training evaluation framework. As mentioned before, the big disadvantage of Cost Benefit Analysis is the difficulty of determining exact financial training benefits.
Summary
In this article I have attempted to provide a comprehensive approach to economically evaluate and communicate e-Learning investments by using Cost-effectiveness Analysis.
We have seen that it is important to base the total cost calculations on variable costs, fixed costs, and the number of participants. We have also seen how to select an appropriate measure of effectiveness and how to present the results using the cost effectiveness pane.
I hope that the CEA based training evaluation outlined in this article will help practitioners of e-Learning and those considering e-Learning. I also hope to stimulate discussion about the financial evaluation of e-Learning investments, which is presently dominated by arguments for or against ROI.
Bibliography
ASTD’s 2010 Learning trends report, http://www.astd.org/LC/0110_trends.htm.
Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning Vol. 27, pp 341–347.
Phillips, J. (1996). How Much is the Training Worth? Training & Development, Vol. 50 No. 4, pp. 20-24.
Kirkpatrick, D. (1959). Techniques for evaluating training programs. Part 1 - Reaction. Journal of the American Society for Training and Development, Vol. 13 No. 11, pp. 3-9.
Kirkpatrick, D. (1959). Techniques for evaluating training programs. Part 2 - Learning. Journal of the American Society for Training and Development, Vol. 13 No. 12, pp 21-26.
Kirkpatrick, D. (1960). Techniques for evaluating training programs. Part 3 - Behavior. Journal of the American Society for Training and Development, Vol. 14 No.1, pp. 13-18.
Kirkpatrick, D. (1960). Techniques for evaluating training programs. Part 4 - Results. Journal of the American Society for Training and Development, Vol. 14 No. 2, pp. 28-32.
Kirkpatick, D. (1996). Evaluating Training Programs. 3rd editionm, Berrett-Koehler Publishers.
Levin, H. M., McEwan, P. J. (2001). Cost-Effectivness Analysis, 2nd Edition, Methods and Applications, Sage Publications.
