Why reinvent the wheel? If you do not have to develop custom e-Learning, why would you? Creating customized training, and specifically e-Learning, can be expensive. Customized e-Learning can take months to develop and requires specific, in-demand instructional design resources to create an effective e-Learning product. So, why would you build a custom training product?
Much of the content that we use to train employees these days (such as project management and ethics content) is common to many organizations. So if the content and knowledge that your learners’ need is common to other organizations, then an e-Learning course is probably available from a vendor to fulfill your needs. Off-the-shelf products can save you money, and, unlike customized training, are ready for your learners today.
The problem is selecting the best off-the-shelf product for the needs of your organization and learners. Unfortunately, many off-the-shelf e-Learning products are mediocre (or worse). Anyone who has developed any distance education products knows that not all creation of e-Learning is equal. Different e-Learning products can have profoundly different educational impacts on learners depending on the design of the experience and the quality of the content. However, many outside our industry, and most learners, see all e-Learning as being equal. If buyers do not know that products can differ in quality, they tend not to evaluate each before purchase and thus can end up with a mediocre product.
Consequently, using mediocre e-Learning has short and long-term consequences. In the short term, poor e-Learning can be a boring experience, which in turn can affect learners’ ability to achieve their educational goals. Learners then perceive the experience as a waste of time. This leads to a bleak long-term problem. These bad experiences will lead to resistance to e-Learning in the future. Think of the money organizations have spent on Learning Management Systems (LMSs) to manage their training, and particularly their e-Learning. Now think of learners rejecting the e-Learning in the LMS. This puts the entire LMS at risk.
One might argue that the real advantage of e-Learning is that you can track it from an LMS. This point of view moves away from the needs of the learner and looks at just the needs of the organization. I would argue that you must take both the learner and the organization into consideration. But, if your learners are taking inferior e-Learning products, what exactly are you tracking in the LMS?
The Three-Pass Evaluation Approach
Organizations need to systematically evaluate off-the-shelf solutions to identify the best product to help them reach their training objectives.
Doing full-scale evaluations of e-Learning products is necessary, but it takes time. This article explains a three-pass approach that will help you identify the best e-Learning experience for your learners in the least amount of time (see Figure 1). This methodology saves time by quickly eliminating inferior products in the first and second passes, so you do detailed analysis of just a few potential products in the third pass.
Figure 1 The Three-Pass Approach
If you do not have the time or the resources to do three full passes, this article also offers guidelines and ideas to help you identify quality e-Learning products for your organization’s needs.
First Pass – Build a list of potential e-Learning products
The goal of the first pass is to find all the potential e-Learning products on the market that meet your educational objectives. Once you identify potential products, add them to a list to review in the second pass. Start the process by identifying your educational objectives. Then, search for products, mostly on vendor Web sites, that will help your learners reach those objectives.
Be flexible. Do not assume that vendors have documented their products’ learning objectives as well as you have. Sometimes vendors only write short descriptions for each of their courses. In many cases, you will have to parse objectives out of these descriptions. If these descriptions come close to your intended goals, add the product to the list of products to review in the second pass. Also, don’t expect vendors to have one or even multiple products that cover all your objectives. If you find a product that covers a few of your objectives add it to your list.
At this point, do not eliminate a product based on cost. Upon detailed examination in the second and third pass, you may discover an obvious reason for the extra expense. You may also find that the more expensive product meets your needs better than the less expensive product.
Second Pass – Do quick online review of the e-Learning products
Take the course in the second (and third pass). The problem is that many organizations stop after the first pass. They base their decision on what the vendor Web site says about the products; they never actually log in and look at the e-Learning product. When this occurs, organizations make their purchasing decisions based on price, which often leads to an inferior learning experience for their learners.
As the saying goes, “You can’t tell a book by its cover,” and, by the same token, you can’t tell good e-Learning from the vendor’s Web site or a vendor demo. To assess whether an e-Learning course meets the goals of your organization and learners, someone needs to log in and take it as a learner would. Then after trying several similar courses, someone needs to compare them. This sound obvious, but few organizations do it. If vendors are proud of the quality of their products, they will want to give you sample logins for you to evaluate their e-Learning products. If you need to review a synchronous training product, ask the vendor to review a recorded session.
Why don’t more organizations have someone log in to and review potential e-Learning products to evaluate them? Maybe they see it as too time consuming to evaluate products well. It does take time, but it is worth it if you want to find the best e-Learning for your organization. Besides, what does it say if the training department expects learners to take an e-Learning product, when they themselves will not take the time to evaluate the product?
Here are a few rules for you to follow in the second pass.
1) Find an e-Learner to do the review
Would you have someone who has never driven a car, or only driven one type of car, test-drive a car for you? Of course not, and e-Learning is no different. To evaluate e-Learning, you need someone who is an experienced e-Learner – that is, someone who has taken a variety of e-Learning products in several different formats and quality levels. Experienced e-Learners will give you an objective view of the e-Learning experience from a learner’s perspective.
Ideally, the evaluator also has some instructional design experience to evaluate pedagogy and the educational effectiveness of a product.
2) Do a quick review
The second pass is not an extensive or time-consuming review of each product. In the second pass, no one has to take the course from start to finish, but they do need to log in and review it online. An experienced e-Learner can tell a great deal by spending 15 to 20 minutes in a product. The experienced e-Learner can quickly look at the design of the product and identify which course will educate and engage learners and which will not. The person can also determine if the content is vastly different from your objectives, and thus eliminate the course from going to the third pass. (If the evaluator is not sure if the content meets the needs, review it in the third pass.)
3) Ask yourself, “Is it better than a book?”
When I do a second-pass review of an e-Learning product, I ask myself “Is the e-Learning better than a book?” This may sound strange; after all, e-Learning is on a computer and the Internet. Computers and the Internet are the technological marvels of our age, while books involve technology that Guttenberg used in the fifteenth century. Therefore, e-Learning must be better than a book. Right?
Not so fast. Let’s take a closer look. E-Learning has lots of text. Books have lots of text at a much finer resolution (600+ DPI for a book, versus about 60 to 100 DPI on a computer screen). E-Learning has graphics. Books have graphics.
So where does e-Learning excel over a book? E-Learning that uses its technology to customize an experience for the learner’s needs is superior to a book. E-Learning can also use multimedia to make the content come alive in useful ways that a book cannot. If an e-Learning product does neither of these things, ask yourself, “Is it better than a book?” The reality is that a lot of e-Learning does not take advantage of these capabilities, and, compared to a book, delivers an inferior learning experience.
Here are few specific questions I ask myself when reviewing an e-Learning product to determine if it is better than a book.
- Does the product use interactivity to deliver a customized and engaging learning experience to learners? Does it use interactivity to make the learners think and challenge their own assumptions? Does it use interactivity to deliver helpful and constructive feedback, or does it deliver “giving-away-the-answer” feedback?
- Does the e-Learning product use multimedia features, such as video, animation, or sound? If so, are these features used in a way that enhances learning, or are they used gratuitously? For example, does the product use animation just for animation’s sake – in a way that doesn’t really add anything to the educational experience?
- Is there social interaction, such as a chat room or threaded discussion, with instructors and/or other learners? Is social interaction integrated into the product, or does it exist outside of the product? Is the social interaction beneficial? (Ask vendors for actual chat room transcripts.)
If the answer is “no” to these questions, particularly related to the use of interactivity, then the product is
4) Look for insufficient content
The other problem with many e-Learning products is insufficient content in the product. Organizations base many e-Learning products on PowerPoint slides – some with only one or two sentences per page. A 100-page e-Learning product may have less content than a 10-page book! Ask yourself if so little content justifies the price that you pay for these products. I would eliminate such e-Learning products from going on to the third pass.
5) Beware the vendor demo
Someone from your organization needs to go into and evaluate these products. Do not depend on vendor demonstrations to evaluate products. A vendor demo is nothing like the learner’s experience in an e-Learning product. Demos done by vendor sales reps are fast and hit the highlights of the product. The learner’s experience is very different. It is slower, because the learner has to be deliberate. This deliberation is what you want to emulate in order to evaluate a product. Vendors can give you a skewed view of the product. For example, if only three percent of the product’s content has animation, you can be sure that the vendor will show you those animations, and lead you to believe that the majority of the product uses animation to illustrate its content.
Third Pass – In-Depth Reviews
Ideally, at this point you have several educationally effective e-Learning products. Now it is time to find the best one for your needs. Do this in two steps that you can do simultaneously:
- Do an in-depth review of each product by taking it as a learner would.
- Seek the opinions of others to evaluate and compare specific aspects of each product.
In-Depth Review
The only way to understand the learner’s experience is to take it as a learner would. To find the best product for your learners, you need to do an in-depth review of each product.
The goal of an in-depth review is to determine if a product will really help the learner reach the educational objectives. Specifically, this step needs to answer the following questions:
- Does the product have a sound pedagogical design? Will learners learn and retain the content that it covers?
- Will the audience accept the product or not?
- Does the product teach the content in a practical context? That is, does the content help the learner transfer skills and knowledge back to the job? Or is it taught in the abstract?
- Will the product keep the target audience engaged?
The only way to answer these questions is for the evaluator (preferably the same experienced e-Learner who did the second pass) to take each course from start to finish. For the third pass, it is not enough to take a quick tour. That would be like getting a movie recommendation from someone who only saw the trailer and watched the first 10 minutes of the movie.
Have different evaluators evaluate different aspects of each product
An e-Learning product is a complex experience that you can look at from different perspectives. To review a product thoroughly, you need to answer these questions, each from a different perspective:
- Is the content correct and appropriate for our audience?
- Is the content written well?
- What will the learners’ reaction be to this course?
To answer these questions, it is best to find people with specific skill sets. You might want to recruit the following people to help you evaluate these specific aspects of each product.
- A Content Expert to review and compare the content quality
- A Professional Editor or Writer to evaluate the writing quality
- Learners from the target audience to understand the learner’s experience
Sometimes it is not realistic to line up three different people. In that case, the evaluator should make several passes through the product and assume the role of a different specialist for each pass.
It is unrealistic to expect content experts and editors to evaluate the entire course, or even to log on to it. So included in each section below are some strategies that will help you get valuable input, while minimizing time commitments.
Content Expert
It does not matter if an e-Learning product gets passing marks from an evaluator or editor if it doesn’t cover the right content. That is, content that is accurate, up to date, and meets your learner’s needs. Your evaluator, probably an instructional designer, is most likely not an expert, and will not be able to do a detailed content comparison of these products. That is why a content expert needs to review these products.
Content experts need to answer these questions for each product:
- Is the quality of the content appropriate? Is it accurate? Is it up-to-date? Does it meet the learners’ objectives?
- Is the quantity of content appropriate to achieve learning success? Does it go into enough detail for the needs of our learners? Does it have too little detail?
- Is the level of the content appropriate for your learners? Is it too complex? Too rudimentary?
- Is the content specific enough to your industry and organization?
To make efficient use of content experts’ time, set up a time to sit down with the expert to demo each third-pass product to them. If you cannot find a convenient time, print out content samples covering similar material from these e-Learning products, and have the content expert review and compare them independently.
Be careful; in my experience, content experts can be overly critical of anything they did not create. So try to separate minor complaints from major issues. An example of a major issue would be that the content is not up-to-date. Use major issues to weed out courses from the third pass, and take the minor comments in stride.
Professional editor or writer
I used to think that the multimedia capabilities of e-Learning would allow us to communicate with visuals and sound, and not the written word. I thought that with all this interactivity, video, and animation we could educate people with less dependence on text. I was wrong.
Today text is the cornerstone communication medium of most asynchronous e-Learning products. In addition, much of the content in corporate America, like changes to accounting law, is not inherently visual, and therefore any product built to teach this content will be highly dependent on text.
How is the quality of writing in e-Learning products? This leads us back to the question, “Is this e-Learning better than a book?” In general, books excel over e-Learning products in the quality of their writing. To be published, a book goes through a rigorous editorial process in a publishing culture where excellent writing and content knowledge are expected. Experts who can write, experts with patient and talented editors, or professional writers who have access to experts, write books. That is not the case with e-Learning. The process for e-Learning is much less rigorous. Instructional designers or technical writers who generally do not know the content in detail, and who have only limited time to work with busy experts, usually write e-Learning. This often leads to some dry and not well-organized writing.
Mediocre writing can, at best, slow the learner down, and, at its worse, confuse learners. Writing that is not concise, and is unclear, takes a toll on the reader. It can be drudgery to get through, and make for a very unpleasant experience.
Here are some questions that can help to guide a professional editor:
- Which products will be the most straightforward and clear for learners to read?
- Which products are at the appropriate conceptual level for the audience?
Like content experts, editors are busy people. So, print out example pages from all the third pass e-Learning products for them to review independently.
Sample Learner Group
Last, but not least, is getting input from actual learners. It is always a good idea to have a group of learners from your target audience test potential products. This feedback can be invaluable, but it is often overlooked. It can also catch technical issues before they occur when the product goes live.
Create a test group of three to four members from the target audience to take each third-pass course. Like the evaluators, this test group needs to be encouraged to take each third-pass course from start to finish.
The first course they take will be the one from which they learn the content. The fact that they are only evaluating the second and third courses they test, and not learning from them, can bias their input. To limit this bias in your feedback, have learners test the courses in different orders. Thus, each learner will start with a different product. In addition, have them test the products in separate rooms (or in different places) so they do not influence each other’s views of the product.
Collect information from these learners in two steps: a post-product survey and a focus group.
1. Post-Product Survey – After they test each product, give the learners a survey to complete. Use this survey to collect general qualitative information about each course. For example:
- Which product would you most recommend? Why?
- Which product would you least recommend? Why?
- Which product did you find most engaging? Why?
- Do you feel that you learned the [educational objectives]? Why, or why not? Explain.
- Was the course easy to log into and start? Explain. Did you have technical problems accessing or getting through the course? If so what were the problems?
2. Focus Group – After all learners have evaluated each product, conduct a focus group session (or a conference call) with all participants in the sample learner group to discuss each product. These learners have spent time in these products, and have developed opinions about them. The focus group will bring out those opinions and comparisons that would not come out in the surveys. This qualitative data can be valuable in making a decision.
Making a Decision
Now it is time to make a decision on which e-Learning product to chose. From my experience, the best product will become apparent. After you have followed the three-pass process, and find two or three products that meet your needs, then you can make your decision based on price.
Below are some considerations, depending on the outcome of the three passes.
What if no clear good e-Learning products emerge?
This is a rigorous process, so you may not find any product that makes it through all three passes. Does that mean the process was futile? No! You just prevented your organization from spending money on an inferior and ineffective product, and you have not wasted the time of your learners. However, it does mean that you will need to come up with a custom solution or a vendor classroom course to reach this particular performance goal.
Since we have asked the question “Is the e-Learning better than a book?” you may want to consider printed materials as a solution to help you reach your organization’s learning goals. If you do not find any useful off-the-shelf e-Learning products, but you know of informative books and/or articles, encourage people to read a book, chapters in a few books, or a few articles. If you need to track learners’ progress, use a test in the LMS to evaluate their understanding of the content.
What if you find a good e-Learning product that only covers some of your objectives?
Most likely, vendors do not organize content in the same way your organization does, so the content of off-the-shelf products will not normally fit perfectly with your needs. Many organizations use this as an excuse not to use an e-Learning product, and create a completely customized solution. If you find a good off-the-shelf product that covers some of your needs, use it, and then build some custom training around it to cover the rest of your objectives.
E-Learning going forward
Poor e-Learning has ramifications for the future of all e-Learning. Learners with bad experiences with previous e-Learning will resist using e-Learning products in the future, thus denying themselves valuable learning experiences from quality products. Learner rejection of e-Learning products will eventually lead to organizational rejection of these products, as the organization sees no one taking or completing them. This resistance hurts your organization’s ability to use this cost-effective technology in the future. It also hurts all of us in the e-Learning industry, as rejection and resistance will slow demand for e-Learning in the future.
Organizations can raise the quality of e-Learning if they are more discriminating about the products they buy and give to their learners. If organizations evaluate potential products closely by using a systematic process, like the three-pass approach, they will start to select higher quality products. Then, in turn, vendors will produce higher quality e-Learning. Everyone – learners, organizations, and e-Learning professionals – will benefit in the end.
