Recently this thought hit me: “Who cares how employees learn it, so long as they can demonstrate that they do know it – on a test?” This notion may be my learner-control subconscious rebelling after a series of mandated minimum-student-contact-time online courses my team had to develop. In order to ensure that each learner met the minimum prescribed contact time, we felt compelled to force them to see most of the screens as well as listen to the entire audio on each of those screens. (We did try to make it interesting by making the courses scenario-based.)
Editor’s Note: Parts of this article may not format well on smartphones and smaller mobile devices. We recommend viewing on larger screens.
So, on courses that don’t have such mandates, can we go to the other extreme and give learners total control over what content they view, including the option of whether they even view the content we provide at all? Can they just prepare on their own and test out?
Perhaps … it seems that it depends on a few things:
-
Are the tests valid? Do they measure the right knowledge and skills?
-
Are the tests sufficient? Do they cover enough of the material so there’s a high correlation between test success and job performance?
-
Do employees really want to follow self-directed learning or constructivism approaches, or do they really want someone to guide them?
-
What about mastery learning, a practice shown to be among the most successful learning strategies of the past several decades?
There is an abundance of research on the above theories in academic settings, but not so much in workplace settings.
In my own experience, employees like it somewhere in the middle. In preparing this article, I researched a variety of terms while looking for one that described the concept I was after. I believe the concept of “mastery learning” comes closest. With mastery learning, the learners receive the resource materials but otherwise expect to study on their own, progressing to each successive level after passing the prescribed test(s).
Many schools use this practice. I learned of two examples of mastery learning in the workplace. In one, a new employee had to take a comprehensive test after one year. If she passed it, she received a sizable pay raise. If she failed, she lost that job. In another example, the setup was the same but after the employee passed the test the company had no money for the pay raise, so the employee left on his own. I plan to do more research on the use of this practice in the workplace.
I suspect mastery learning holds the most promise for enlightened workplace learning, but no matter where we are on this spectrum, there is a need not just for valid online end-of-course tests, but for tests with slightly different purposes, as well. That’s what we’ll cover in this article.
Tests: macro view
Where do test questions come from? If you immediately said, “the learning objectives,” then you get an A+. Before we look at construction of questions themselves, let’s look at tests from a macro viewpoint.
The role of tests
What role do tests play in online workplace learning?
“…our first concern with assessment is knowing what students know.” (sic) [Stoner and Smith, 2007 – see References at the end of this article]. But I have come to view tests as having much broader application than merely assessing knowledge, especially in Web-based training (WBT) where there’s no instructor to moderate the learner’s learning path, conduct role plays, offer encouragement, or debrief a test. We can use well-designed test questions diagnostically to moderate the learner’s path, reinforce learning, and help the learner gain new insights; and, together with appropriate feedback, they can even encourage and motivate the learner. Online assessments can play an economic role as well, by saving employers time, money, and effort. [Greenstein 2010]
Before we go on, let me set the scope of this article. The goal of workplace learning is job and organizational performance. Usually such performance involves some levels of knowledge and skill that we can’t teach or measure on a computer. We often use online learning (or Web-based training) for lower-level cognitive skills, but we’ll see how you can use it at higher levels of Bloom’s taxonomy, too. Other types of learning such as authentic assessment must and do take place in the workplace, and we may measure these in other ways. But although some of the principles mentioned here apply equally to paper and pencil tests, the thrust of this article is on using online tests, or automated testing as some would call it. [Brooks 1997]. We will also examine how to enhance instruction by what we do after the learner takes a test, and we’ll look at some authoring and learning management system capabilities and limitations.
Tests can measure knowledge at the outset of a course or program (pre-tests), within it (embedded), and at the end (post-tests). Tests can be summative (scored) or formative (used to adjust the learning path or approach). Properly set up and combined with the right tone of feedback or remediation, a test can encourage the learner and build self-confidence. We frequently embed self-check questions in our online courses that look and feel like games, and our employees love them! So much so that we even use them in some instructor-led group settings to add some variety and fun. I believe there is a key difference between tests used in a school setting and those used in the workplace. In school, the student’s goal is a high test-score, yet the instructor often wants to force a bell curve. Some would likely accuse an instructor whose students all get A’s of teaching the test. Conversely, in the workplace the goal is mastery – by everyone. Why? Because we want employees to know how to do their job. They may not all achieve 100%, but we hope to skew the curve as far towards 100% as feasible – learner ability, not intentionally difficult test design, being the variable.
Test types
Now let’s look at the pros and cons of using the various test types. Throughout this discussion I’ll use the word “course” to signify any designated content no matter how structured or unstructured.
Pre-tests
We give pre-tests before the learner embarks on a course. Here are some reasons commonly cited for using pre-tests [Berry, 2008, and Berman, 2001]:
-
To find out what the learner(s) already know about the subject. This may be useful to:
-
Establish a baseline to judge the effectiveness of the training or, at least, the amount of new knowledge the learners gained
-
Divide a group into learning groups (pertains more to instructor-led, but there is a parallel for WBT as we’ll see later)
-
Help the learner herself know which material to zero in on. [Stoner and Smith, 2007]
-
-
Provide a preview of the terminology and content as well as the depth of coverage.
-
Motivate the learner by virtue of the preview.
-
Set the learner’s expectations, especially if you plan to use similar or identical questions on the post-test.
The above reasons notwithstanding, in my opinion there is a single compelling time to use pre-tests, and that is when you expect a significant amount of prior knowledge among the learners and you will adjust the learning path to accommodate that knowledge.
One big downside of pre-tests is that they are, after all, a test. A significant percentage of learners already suffer test anxiety, so why exacerbate that fear. That can hardly be motivating.
I think there are more personal, congenial ways to motivate or give a preview than through a test. Borrow from fiction and establish suspense at the outset. Here is an example of an opening line you could use in a sexual harassment prevention course:
What if you overheard your neighbor at work making a lewd comment to another co-worker. Should you keep it to yourself, or tell someone? If you tell someone, are you putting yourself in jeopardy of retaliation?
Post-tests
The main reason for using a post-test is simply to measure the learning outcomes. Can the learner demonstrate mastery of the subject matter as defined by the course objectives? The post-test must evolve from solid and thorough task analysis in the early stages of instructional design. In fact, many instructional designers develop and finalize the learning objectives and then go straight to designing the post-test. Only then do they develop the actual instructional material.
I think a key to a successful post-test is what you do with the results. For one thing, the test review itself, even online, should serve as the capstone to that chunk of learning. And you should use item analyses to assess the effectiveness of the instruction and the test.
Embedded tests
If pre-tests are used for tailoring the learning path and post-tests are used solely to measure knowledge, then what is the role of embedded tests? They can do the same things the other two test types can do, and more. Embedded tests can:
-
Perform formative assessment, or diagnostics, throughout the course, tailoring the delivery to the learner’s existing knowledge on a topic-by-topic basis instead all at once up front.
-
Perform summative assessment, but at intervals throughout the course instead of all at the end. An embedded test can provide an alternative to a final exam. While discussing compliance training at a conference workshop a few years ago, most of the participants agreed it was sometimes more advisable to force learners to get each embedded question correct before they could move on than it was to use a scored final exam. The rationale? In effect, every learner has to get 100% in order to get credit for the course.
-
Provide practice for and/or a preview of the final test. Such a preview can reduce anxiety for the final test and can thus build the learner’s confidence, especially with the use of conversational, reassuring feedback.
-
Encourage and motivate. You can use short embedded tests along the way to reassure the learner in building-block fashion that he or she is acquiring and applying the knowledge at an acceptable level.
-
Provide a scenario-based learning experience.
-
In a course on sexual harassment prevention, we used multiple-choice questions in each scenario to simulate a conversation. [Dickinson, 2009] It wasn’t as open ended as a live roleplay, but it engaged the learner more than a straight presentation would have. It also enabled us to address the “gray area.” For example, an employee has options on whom to approach with a harassment complaint – the instigator, their manager, or HR – and there are pros and cons to each option. So we allowed multiple correct answers (thus letting the learner proceed in the course) while the feedback acknowledged the points to consider with each choice and why one option may be the preferred choice in a given situation.
-
