Blended Assessment in Small to Medium-sized Classes

"As blended learning takes hold, educators will need to continue to look at different ways of assessing learning. Blended and balanced assessment is a necessity. ... Many courses feature e-Learning, classroom- based learning, blogs, wikis, Podcasts, discussion groups, and many other technologies. ... Educators need to plan for changes in the way we assess learners in this changing blended learning landscape. New blended assessment techniques, tools, and methodologies may be necessary."

Assessment is the process of establishing, usually in evidence-based and measurable terms, knowledge, skills, attitudes, and beliefs. While there are many different methods of assessment available to educators, they often choose to rely on simple methods for testing (such as an instrument composed of multiple choice questions) or by traditional methods (written examinations).

But this, too, engenders discussion as educators have for a long time debated the merits of multiple-choice type assessments versus those of traditional “pencil-and-paper” assessments. The dilemma for educators is often deciding which method is best suited to a particular situation. Very often, the number of learners to assess dictates the chosen assessment method. When a large number of learners is to be appraised, a multiple-choice test is used even though it takes a lot of time to create one. After all, such tests require little time to assess and mark, especially if online assessment tools are used. When the assessment involves a smaller number of learners, the more traditional approach of setting essay-type questions works best for many educators, regardless of the fact that such questions require extended time for marking.

The dilemma between item types is even more acute in the online environment. Rarely are learners asked to provide long typed answers on a computer in a formal examination. Mostly, it just does not make sense to do so when there is a simpler and cheaper option available: hand-written exams. Consequently, most educators in the online environment rely on the tried-and-trusted, though limited, multiple-choice type of questioning. Typed answers are rarely more than simple fill-in-the-blank or short-answer questions, making them relatively easy to mark. In recent years, an explosion of new learning technologies for assessment is changing the educational landscape. Curtis J. Bonk refers to the Perfect E-Storm, “... where technology, the art of teaching, and the needs of learners are converging.” (See “References” at the end of this article.) Bonk discusses 30 emerging technologies that are generating waves of new opportunities in online learning environments. Combined with the traditional classroom environment, teachers and learners now face powerful teaching and learning methods that provide a “blended learning.” Marc J. Rosenberg in his book Beyond e-Learning describes “true” blended learning as a “... combination of training (formal) and non-training (informal) approaches that support the smart enterprise (such as knowledge management, performance support, and coaching) in ways that improve the effectiveness and efficiency of learning.” In this blended learning environment, then, it is logical to expand assessment by blending this too. In this article, we review the advantages and disadvantages of each type of assessment, investigate new ways of blending assessment, and explore the implications for emerging learning technologies.

Multiple-choice questioning vs. “pencil-and-paper” assessment

Written examinations have been the central component of assessment for centuries. Ancient China seems to have been the first civilization to develope an examination system, during the Sui Dynasty (589-618). The examination for entry into the Chinese Civil Ser-vice was known quaintly as “Han Lin,” the forest of pencils. These exams were of essay type. Choice response exams, with multiple-choice questions (MCQ) have been around for a relatively short time, only about 100 years or so. Frederick J. Kelly is credited with creating multiple choice questions in 1914 at the University

of Kansas. One of the early large-scale uses of multiple choice questions was to assess the capabilities of World War I army recruits, according to an entry in Wikipedia (see http://en.wikipedia.org/wiki/Multiple_choice).

As is widely recognized, MCQ is a type of objective stimuli where questions are designed to have a largely indisputably correct answer, or “best” answer. This type of test mostly tests broad knowledge and comprehension. In contrast, a subjective test, like an essay, has no indisputably correct answer; rather, in these appraisals the examinee responds to a written scenario (often called a “prompt”) and develops a topic sentence with supporting arguments.

With careful planning and design, MCQ tests can also assess higher-order learning outcomes, such as the application of knowledge and analytical skills. But writing suitable multiple-choice questions — with a leading stem and appropriate distracters — is difficult, and many educators are not skilled in this specialized kind of writing. In addition, doing this work involves a considerable investment of time and can dramatically increase teacher workload. However, the advantage is that the resultant tests can be marked very easily and quickly. And, when using an online system, the results of the appraisal can be given to learners immediately upon conclusion of the test. To this end, the teacher’s workload reduces considerably. Other advantages of online delivery are that tests can now include a variety of media (audio, video, and graphics), and supporting feedback can be given automatically (during or after the test). Also, randomized selections from large question banks can be made, thereby improving test security.

Many learning content management systems (LCMS) now provide tools to make the creation of online tests relatively easy. These tools also relieve the administration burden for educators associated with assessment. The number of question types is limited, and varies according to the type of LCMS in use. Two popular systems in use in third level education are Blackboard and Moodle. At the time of writing, Blackboard currently allows for fifteen different question types, while Moodle allows for nine.

However, if one wants to encourage an examinee’s appreciation of underlying concepts and foster deeply critical analysis, MCQ has major drawbacks. Knowing that the end-of-term test will be composed of MCQs which often tend to address superficial facts may encourage learning of surface detail rather than a deeper admiration of the underlying concepts. Even using carefully worded questions, assessors cannot be sure that a learner who answers correctly not only knows the correct answer, but also understands the subject being examined. Then, with MCQ, guessing is another obvious limitation. After all, a learner who guesses the correct answer gets the same marks as one who fully understands the subject. Almost certainly, this would not happen in pencil-and-paper tests. And, learners can select a correct answer for superficial reasons, such as when they vaguely remember reading something about the subject, or by selecting the answer through a process of elimination.

While it is difficult for learners to achieve high overall marks in an MCQ test with limited knowledge, they can get lucky and pass a test by merely guessing, looking for patterns, and reducing the number of possible correct answers by eliminating some of the wrong options. Negative marking (or formula scoring), sometimes used to counter-balance guessing, is disadvantageous in that it might discourage a learner’s proper response based on good partial knowledge.

Educators, therefore, cannot be fully satisfied with MCQ tests, regardless of whether they are paper- or -computer-based. Throw in authentications and security issues in the online environment and there are many reasons why MCQ is not popular with some educators. Equally unpopular for essay exams is the effort of marking page after page of written scripts.

A blend of assessment methods

A few enlightened educators have been using blends of assessment for many years. Blended assessment is a “... combination of a variety of assessment modes, such as paper and pencil tasks, online assessment tasks, peer-assessment, overall assessment ...” (as Filip Dochy defined it). At its simplest, it is common for examiners to set different types of questions in an exam paper. Typically, one section contains a series of short or multiple-choice questions, followed by essay-type questions in another section. However, final assessments are usually carried out in traditional exam halls, as providing online facilities is frequently impractical. For online tests, it is rare to include an essay component.

The numbers of learners taking an exam will still influence the choice of assessment methodology; however, there are opportunities for small- to medium sized groups to make use of the benefits of both MCQ and written assessment together in a single “blended” exam. Consider for example the multiple choice question from a course (“Introduction to Java Programming”) shown in Figure 1.

 

 Figure 1 There are problems in determining whether the learner who chooses the correct answer to this question really understands why there is an error in the code.

 

As most Java programmers (and the diligent examinee in the course) will notice, there is a syntax error in option “C.” The full stop will generate an error in the Java compiler. Options “A,” “B,” and “D” will not generate an error. As the question stands, a learner selecting option “C” will get full marks regardless of whether they are proficient in detecting Java syntax errors or not. Perhaps option “C” simply looks wrong because it is a bit different from the others. No examiner can be certain that a learner who selects option “C” really understands why there is an error in the code, after all, the learner was not required to write out a particular “for” statement which, if done correctly, would demonstrate a deeper understanding of syntax, as well as coding. One can also say that learners who instead select options “A,” “B,” or “D” may strongly feel that their selection does in fact contain an error, but they have no opportunity to explain why or justify their choice.

Suppose, now, the question is given in two parts — one part composed of MCQ and the other requiring a written reply. The first part would be exactly the same question as above and is answerable in the same way. The second part could be a follow-on question based on the first part. At its simplest, the follow-on question could be something like “Explain why you made your selection.” A more complex version could be something like, “Identify the syntax error(s) in your selection and write a correct version.” Short written answers are all that is required for the follow-on part of the question. Learners should get extra marks for these explanations. These written responses will have to be marked separately.

Suddenly, but usefully, the question becomes a tougher one for learners to get full marks on. The more able learner who can correctly identify the right option, explain fully what the syntax error is in this case, and suggest an acceptable alternative, will receive the top marks. A learner who selects the wrong answer and gives an invalid written answer will score lowest, per- haps even zero for the full question. A learner who guesses the correct answer, but does not know why the error occurs and cannot give a reasonable explanation, will still get some marks for the correct option, but will score poorly in the follow-on question. What about the learner who makes an incorrect selection, but writes an explanation as to what they believe the error is, and even writes an alternative, acceptable, version of the code? This learner will still receive zero marks for the MCQ part, but perhaps their written answer should warrant some extra, even meritorious, marks. Overall, this blended approach can encourage learners to think a little deeper into the answer as they will have to provide an explanation for their selection. It is also encouraging to know that even if a learner gets a multiple-choice question wrong, there is still the possibility to get some marks in the follow-on. Learners are not discouraged from guessing, as negative marking does not apply. They may even be encouraged to guess and to attempt an explanation in the hope of gaining some marks.

The absolutism of MCQs being either correct or incorrect can disadvantage learners. Supposing in the question on Java code the learner is confident that he or she can eliminate two incorrect options out of the four choices available, but is undecided as to which of the remaining two options to select. The learner did evidence that he or she can at least partially answer the question correctly by eliminating two options, but if the poor soul then selects the incorrect option from the remaining two, zero marks ensue, despite displaying some knowledge.

While a learner may have demonstrated partial knowledge, they do not get any credit for this. Negative marking, if applied, is therefore sometimes unpopular with many educators. There is a debate as to whether negative marking makes a difference to overall test scores and has lead to much research in this area. For example, William Angoff and William Schrader show in a major study that “... formula scores are essentially invariant under different testing directions ...” though Mark Albanese questions these findings. (See “References” at the end of this article.)

Our approach will give the assessor some insight into how learners, and the class as a whole, are demonstrating true knowledge. If learners are getting MCQs correct, but are unable to provide explanations for their selections, then guessing may be a factor. The assessor will have to re-visit such questions to ensure that they are carefully written so as not to hint at the correct option that is easy to guess. Assessors will also be able to give credit for partial knowledge in the cases where follow-on questions indicate this knowledge on behalf of the learner.

In contrast, for pencil-and-paper assessment, learners can write their explanations at length and will most likely gain at least some credit for even partial knowledge. Here, they are less likely to get zero marks in written responses, especially if the response illustrates some knowledge of the subject area. The examiner’s judgment is now a factor in assessing a response.

Of course, use of follow-on questions for each MCQ defeats the purpose of using MCQs in the first place. Such a strategy would be impractical, and would increase workload. Instead, one should achieve an appropriate balance for the number of follow-on questions compared to the total number of MCQs. Our practice is to set two to three follow-on questions per assessment. Even one such question in an MCQ test can be useful. This number is small enough to mark quickly, but at the same time get a clearer picture of learners’ levels of understanding.

It is important to note that one must take greater care in phrasing both parts of this type of MCQ/follow- on combination. Meticulous wording of questions gives learners some opportunity to gain marks, even if they make a wrong MCQ selection. Although difficult, if the instructor can prepare questions that target inductive or deductive reasoning, he or she can gain even further insight into examinee’s responses.

So how do learners perform under the blended assessment format described here? While this article is part of a wider and on-going study on blended assessment, preliminary results show that learners can benefit if appraisals employ blended assessment. Two experiments compared learners’ overall scores for blended assessment (both featuring ten MCQs and three follow-on questions) with scores that exclude the marks for the follow-on questions. Figure 2 shows the results where overall class marks were poor for a class of 22 learners.

 

Figure 2 A comparison of marks for full blended assessment (blue line) with marks excluding follow-on questions (red line).

 

In this assessment, 16 out of 22 learners had their marks increased by between 0.4% and 11.2% when marks for the follow-on questions are included. The remaining six learners show a decrease ranging from -1.9% to -8.1%.

Figure 3 shows the results where overall class marks were good for a class of 19 learners.

 

Figure 3 A comparison of marks for full blended assessment (blue line) with marks excluding follow-on questions (red line).

 

In this assessment, 14 out of 19 learners had their marks increased by between 0.5% and 9.06% when marks for the follow-on questions are included. The remaining five learners show a decrease ranging from -0.7% to -7.1%.

While in both experiments the results closely matched, they do show that the majority of learners gained extra marks in the follow-on questions and benefited overall from this method of assessment. Both of the above two tests where overall scores were poor in one assessment and good in the other assessment demonstrate this. Currently, we are conducting more research into these kinds of assessments to understand them more fully.

Future of blended assessment

The drawbacks for using some types of blended assessment are that there is a still a written component to be marked, and there is a lack of tools usable in the online environment. In order to succeed in the online environment, question authoring tools would have to be created to allow follow-on questions to be inserted at the author’s choosing. In the experiments described above, we authored and delivered the MCQ questions in Moodle, while learners wrote their answers to the follow-on questions on paper handed up at the end of the assessment. Learners don’t get their overall score until the follow-on questions are marked. This mixture of online and hand written responses can be confusing to learners, so instructions must be very clear. Where learners are not familiar with this type of assessment practice tests may be necessary.

As blended learning takes hold, educators will need to continue to look at different ways of assessing learning. Blended and balanced assessment is a necessity. Filip Dochy believes that learners might find “ ... new modes of assessment motivating and Interesting not certainly because of the characteristics of such new modes, but perhaps because they severely dislike and are de-motivated by traditional testing methods.” Many courses feature e-Learning, classroom- based learning, blogs, wikis, Podcasts, discussion groups, and many other technologies. Will Richardson says, “The coming years will be marked by a flood of new innovation and ideas in teaching.” Educators need to plan for changes in the way we assess learners in this changing blended learning landscape. New blended assessment techniques, tools, and methodologies may be necessary to respond to this change.

References

Albanese, M. A. 1986. The Correction for Guessing: A further analysis of Angoff and Schrader. Journal of Educational Measurement 23 (3), 225-235.

Angoff, W. H. & Schrader, W. B. 1984. A Study of Hypotheses Basic to the Use of Rights and Formula Scores. Journal of Educational Measurement, 21 (1), 1-17.

Bonk, C. J. 2004. “The Perfect E-Storm: emerging technology, enormous learner demand, enhanced pedagogy, and erased budgets.” http://www.publicationshare.com/part1.pdf (Accessed November 14, 2006)

Dochy, F. 2005. “‘Learning lasting for life’ and ‘assessment’: How far did we progress?” Presidential address, EARLI 2005 at the 20th anniversary of the European Association for Research on Learning and Instruction. http://www.kuleuven.be/ ~u0015308/ Publications/EARLI2005%20presi- dential%20address%20FINAL.pdf (accessed November 14, 2006).

Osterlind, S. J. (2006). Modern Measurement: Theory, Principles, and Applications of Mental Appraisal. Upper Saddle River, New Jersey: Pearson, Merrill Prentice-Hall Publishers. ISBN 0-13-025590-4 Richardson, W. (2006). Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms. Corwin Press. Thousand Oaks.

Rosenberg, M. J. (2006). Beyond e-Learning: Approaches and Technologies to Enhance Organizational Knowledge, Learning and Performance. San Francisco, CA: Pfeiffer.

Comments

Login or subscribe to comment

Related Articles