Enterprises can receive great benefit from e-Learning and training managers have realized this. According to ASTD’s 2010 Learning Trends report, over 18% of enterprises spent more than half of their entire training budgets in 2009 for blending or substituting traditional training with e-Learning.

Nevertheless, financially evaluating corporate e-Learning investment decisions remains difficult: the popular approach in this field, Return On Investment (ROI) analysis, requires a comparison of the cost of a program with its monetary benefits. Problems arise mainly from the latter.

In this article I introduce the less complex Cost-effectiveness Analysis (CEA) for identifying economically beneficial e-Learning investments. In contrast to ROI analysis, CEA does not require the translation of training outcomes into monetary training benefits in order to produce insightful results.

I will focus on decisions regarding blending or replacing traditional training with e-Learning, where CEA is especially feasible. After highlighting key issues when assessing training cost and effectiveness, I will introduce the Cost-effectiveness Pane as a powerful tool to communicate CEA results by providing a clear visual representation of the changes in cost and effects. I close the article with advice on how to deal with multiple outcomes and a quick review of key concepts.

Costs and effectiveness

The two major economic changes for an enterprise that is substituting or blending traditional training with e-Learning are:

  1. Training cost changes
  2. Changes in training effectiveness

Training cost and effectiveness are the chief components of CEA, and being clear about these points is essential to attaining meaningful results. Here is a brief introduction to Cost Analysis (CA) and Effectiveness Analysis (EA).

Cost analysis

Depending on the nature of the face-to-face or e-Learning training, different costs may occur. Whether, for example, staff travel expenses appear in the CA of a face-to-face training depends on where the training takes place. Travel costs occur only when the course is not delivered in-house. The analysis only has to consider development for an e-Learning course if the course is specially developed for one’s own organization. If the organization bought a standard course, user license costs appear instead.

It is important to base the cost formula on fixed costs, variable costs, and the total number of participants.

The cost formula

Costs are either fixed or variable in relation to the number of participants. Fixed costs, such as course development costs, occur independently of the number of participants. This means that the total fixed cost remains the same, no matter how many people join the course. Thus, the fixed cost per person actually decreases as the number of participants increases. Economists call this phenomenon “economy of scale”. Variable costs, such as travel expenses, occur for each participant individually. This means that total variable costs increase along with the number of participants but the variable costs per person stay constant.

Therefore, we can use the following formula to calculate the total cost of both face-to-face training and e-Learning:

Total Cost  =  Fixed Cost  +  Variable Cost  x  No. of Participants

Irrelevant cost

We need not include in our cost calculation those costs that are the same for both traditional and e-Learning solutions (e.g. the salary of the training manager). The reason for this is that we are comparing the changes in cost of substituting or blending an existing face-to-face training with e-Learning. Including these costs for both alternatives will yield the same result but will create more work than is necessary.

When deciding whether to blend or substitute an existing face-to-face training with e-Learning, you should not include the costs which have already been paid for during the face-to-face training (sunk cost). For example, you have developed a face-to-face training for which now there are only variable costs (i.e. the trainer’s salary). These development costs are considered sunk cost and can thus be ignored. For the new e-learning alternative, however, you should include these costs.

A CA example

Let us analyze the cost of substituting an existing tailor-developed face-to-face course with that of a similarly tailor-developed e-Learning course. Figure 1 shows the total cost of both alternatives in relation to the number of participants.

Formulaic chart comparing cost between students, red versus green

Figure 1: Cost comparison of a personalized face-to-face training versus a personalized e-Learning course, dependent on the number of participants

 

Both courses have fixed development cost. The development cost of face-to-face training is clearly lower than that of e-Learning. On the other hand, once developed, the e-Learning course does not incur any further costs. Therefore, the total cost line for e-Learning is horizontal. The traditional course, however, contains such variable costs as the cost of the trainer. Consequently, the total cost line of the face-to-face course continues rising after development. When a particular number of participants is reached at N1, the total cost line of the face-to-face training intersects with that of the e-Learning training. So after this number is reached, the total cost of e-Learning is less than that of a traditional training. From N2 on, the additional cost of e-Learning is less than the additional cost of continuing the face-to-face training.

Effectiveness analysis

CEA compares the cost and effectiveness of different alternatives with the same goal. In the previous section we analyzed the costs of blending or substituting a face-to-face training with e-Learning. This section deals with the Effectiveness Analysis (EA) of the two alternatives.

Effectiveness is the extent to which an activity fulfils its intended objective. Finding the appropriate measure(s) of effectiveness is essential for a successful EA.

Levin and McEwan (2001) emphasize that evaluators need to consider two general concepts when choosing among measures of effectiveness: reliability and validity. A reliable measure is one that yields the same results when applied repeatedly to the same individual. A measure is said to be valid if it reflects a close correspondence to the main objective of the alternatives. The evaluator's goal is to employ the most reliable and valid measures available.

Donald Kirkpatrick is a pioneer in the science of corporate training effectiveness evaluation (Kirkpatrick 1959, 1960) and his model (Kirkpatrick 2006) is still by far the most popular one used by organizations today (Bates, 2004). It consists of four levels of training outcomes:

  1. Reaction: measures learners’ satisfaction with the program
  2. Learning: measures changes in knowledge, skills and attitudes
  3. Application: measures changes in job related behavior
  4. Results: measures changes in business impact variables

CEA can be done most easily if we focus on exactly one measure. It should be the most reliable and valid measure available, revealing to what extent both face-to-face and e-Learning fulfill the intended main training objective. For new employee induction training you could use Kirkpatrick’s Level 2 test results as an effectiveness measure.

In the next section, I will assume that there is only one such effectiveness measure. Later, I will discuss how to deal with more than one effectiveness measure.

The cost-effectiveness pane

In the previous section I discussed CA and EA. In this section I will be combining them to show the CEA results with the help of the Cost-Effectiveness Pane.

The Cost-Effectiveness Pane is a graphical representation of the changes in cost and effect of the e-Learning investment compared to the previous traditional training. It consists of a cost decrease dimension (y-axis) and an effectiveness increase dimension (x-axis). (See Figure 2.)

 

four sectioned square with a round moniker in the middle saying face to face training

Figure 2: The four Quadrants of the Cost-Effectiveness Pane for blending or substituting an existing face-to-face training with e-Learning

 

The point of cost-effectiveness for the traditional training lies at the intersection of the cost and effectiveness axes. We can evaluate each alternative e-Learning investment in terms of the incremental change in both costs and effects. This incremental analysis results in the placement of the point for e-Learning cost-effectiveness into one of the four following cost-effectiveness quadrants:

  1. Reduced cost, increased effectiveness
  2. Increased cost, increased effectiveness
  3. Increased cost, reduced effectiveness
  4. Reduced cost, reduced effectiveness

The “Reduced Cost, Increased Effectiveness” Quadrant

If your e-Learning investment finds its way into this quadrant, you can increase effectiveness and reduce cost by blending or substituting your traditional course with e-Learning. Therefore, e-Learning is clearly the better alternative.

As effectiveness and monetary benefits are positively correlated with each other, we also know that in these cases the ROI of e-Learning is higher, without ever having calculated any monetary benefits.

For example, substituting e-Learning for a traditional new employee orientation in fast-growing enterprises often results in this: the huge economies of scale drive cost down. Also, utilizing multimedia can be a much more effective means of communication, as opposed to relying solely on traditional methods. 

It is important that we can draw the same conclusions as above if the result is located on the borders of this quadrant as well. In these cases, either the cost stays constant and effectiveness increases, or the cost decreases and effectiveness remains the same.

I have often found that using e-Learning results in a cost decrease, while keeping effectiveness constant, Think, for example, of trainings for computer applications such as Enterprise Resource Planning (ERP) systems.

The “Increased Cost, Reduced Effectiveness” Quadrant

E-Learning investments falling into this quadrant are plainly not economically beneficial for the company. Compared to the status quo, e-Learning is worse in terms of cost, as well as in terms of effectiveness.

Even without looking into the monetary benefits of the two training modes we can conclude that an e-Learning alternative has a lower ROI.

If you are making an investment decision, you might think of simply postponing the project: e-Learning is still an industry where innovation takes place every day. As time passes, projects might make their way out of this quadrant by themselves.

The “Reduced Cost, Reduced Effectiveness” Quadrant

If you find your e-Learning project in this quadrant you are looking at a more complex situation. E-Learning may be deemed beneficial or not based on the net result of cost and effectiveness reduction. Further analysis is needed to clarify the result.

In order to get a clearer picture of your project you might decide to use a ROI analysis. Even if the ROI result is positive, there is still another issue to deal with in this quadrant. Whereas a CFO will not likely have a problem with a cost decrease, getting trainees to commit may be more difficult, especially if certain expectations regarding effectiveness already exist.  If this is the case, one can use internal marketing strategies to help gain support for the course and better promote its advantages, thus protecting the investment.

Another way is to increase e-Learning effectiveness to a change of zero or greater in order to make e-Learning the beneficial strategy. Sometimes this is easily attainable: I was involved in a situation where an e-Learning project management course was far less effective than the course given by an internal trainer from the IT department. Upon further analysis, we found that it was the face-to-face trainer himself that raised effectiveness to an incredibly high level. Since he was such an exceptionally good trainer, the solution to this problem was to blend the e-Learning course with his face-to-face training. This blended solution successfully moved the training course into the “Reduced Cost Increased Effectiveness Quadrant”.

The “Increased Cost, Increased Effectiveness” Quadrant

If both cost and effectiveness are increased by e-Learning, further analysis is needed to justify the transition. If this is your first e-Learning investment it may be advisable to consider another one with more obvious benefits.

The question is: Is the additional effectiveness worth the increased cost?

The pressing issue with this situation is the cost. One way to make this an easier investment decision is by finding ways to lower the change in cost to zero. One could, for instance, use cheaper technology. Another way to balance the cost would be to increase your user base in order to make use of greater economies of scale. Increasing the user base is not only possible internally: if your project is to be tailored specifically for you, you may discuss opportunities with your provider to share the cost with other customers of this vendor. You may also choose to use an already-existing course instead of a completely customized one. Unfortunately, all of the aforementioned ideas, the last in particular, bear risks of problems with effectiveness.

If you determine that a course in this quadrant is indeed beneficial for the company, it may be more difficult to ensure commitment from the CFO than from the end-users.

Dealing with multiple measures of effectiveness

Originally, CEA was intended to compare several alternatives based on a single measure of effectiveness. Of course, many training programs produce several outcomes, requiring more than one effectiveness measure. In this chapter we will look at ways to deal with such cases.

An excellent way to deal with these cases is to conduct a separate CEA for each measure of effectiveness. The Cost-effectiveness Pane would then be extended by these additional measures of effectiveness. If all of the points for each alternative are located in the same quadrant then the interpretation remains the same. It becomes more difficult to analyze when this is not the case. Fortunately, the evaluator is still able to clearly present the results and describe the related trade-offs.

Another method is to combine the different measures of effectiveness, weighted by their relative importance, into one single measure of utility. This approach is called Cost Utility Analysis (CUA). The problem with this approach is that it can easily become very complex, even convoluted.

Finally, there remains Cost-Benefit Analysis: in this approach, all outcomes are translated into monetary values. The popular training ROI is actually one form of Cost Benefit Analysis. Jack Phillips refers to this as an additional fifth level of Kirkpatrick’s four-level training evaluation framework. As mentioned before, the big disadvantage of Cost Benefit Analysis is the difficulty of determining exact financial training benefits.

Summary

In this article I have attempted to provide a comprehensive approach to economically evaluate and communicate e-Learning investments by using Cost-effectiveness Analysis.

We have seen that it is important to base the total cost calculations on variable costs, fixed costs, and the number of participants.  We have also seen how to select an appropriate measure of effectiveness and how to present the results using the cost effectiveness pane.

I hope that the CEA based training evaluation outlined in this article will help practitioners of e-Learning and those considering e-Learning. I also hope to stimulate discussion about the financial evaluation of e-Learning investments, which is presently dominated by arguments for or against ROI.

Bibliography

ASTD’s 2010 Learning trends report, http://www.astd.org/LC/0110_trends.htm.


Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning Vol. 27, pp 341–347.


Phillips, J. (1996). How Much is the Training Worth? Training & Development, Vol. 50 No. 4, pp. 20-24.


Kirkpatrick, D. (1959). Techniques for evaluating training programs. Part 1 - Reaction. Journal of the American Society for Training and Development, Vol. 13 No. 11, pp. 3-9.


Kirkpatrick, D. (1959). Techniques for evaluating training programs. Part 2 - Learning. Journal of the American Society for Training and Development, Vol. 13 No. 12, pp 21-26.


Kirkpatrick, D. (1960). Techniques for evaluating training programs. Part 3 - Behavior. Journal of the American Society for Training and Development, Vol. 14 No.1, pp. 13-18.


Kirkpatrick, D. (1960). Techniques for evaluating training programs. Part 4 - Results. Journal of the American Society for Training and Development, Vol. 14 No. 2, pp. 28-32.


Kirkpatick, D. (1996). Evaluating Training Programs. 3rd editionm, Berrett-Koehler Publishers.


Levin, H. M., McEwan, P. J. (2001). Cost-Effectivness Analysis, 2nd Edition, Methods and Applications, Sage Publications.