Tool Overview: Questionmark Perception

This is the first of a series of articles on specific tools that are in use in the e-Learning industry. Why a series? Why not just borrow an adage from The Highlander and declare, “There can be only one,” or perhaps paraphrase from The Lord of the Rings, “One tool to rule them all.” In other words, isn't there one tool that is simply better than all the others, one that is more feature-rich, easy to use, and reasonably priced?

The simple answer is no. The more interesting answer is there probably can never be just one. Consider the tools you may already know. Some are powerful and feature-rich. They tend to be time-consuming to learn because they have many features. Or think of those that are simple to use; they tend to offer fewer options and do not allow as much customization as their more powerful cousins. Designers and subject matter experts without a programming background may feel overwhelmed by powerful tools. Those who love to program computers find themselves frustrated by the lack of scripting in simpler tools.

In truth, most e-Learning developers come to realize that only by having a combination of tools, and taking advantage of the strengths of each, can we truly deliver world-class learning. When we force ourselves to use one tool only, we tend to create less than stellar e-Learning.

Along with what are commonly called authoring tools are assessment tools. Practitioners develop e-Learning lessons with the former, and quizzes, tests, exams and surveys, which we will collectively call assessments, with the latter. In addition, assessment tools usually have robust data collection capabilities and typically support detailed reports that allow for much easier analysis of the effectiveness of the learning that is taking place. In short, is the learner really learning? Assessment tools help to achieve that goal by noting where there are weaknesses in the learner’s knowledge, even after completing an e-Learning course. This can help you to improve the next version of your e-Learning lessons.

It should go without saying that it is difficult to know whether e-Learning works without assessing the progress of learners. Many e-Learning tools allow developers to build assessments as well, but usually to a lesser degree than is possible with a dedicated assessment tool. In this article, I will focus on a popular assessment tool called Questionmark Perception.

The Questionmark tools

Questionmark offers three tools that work together. (See Table 1.) From what I have seen, the tools complement each other well.

Perception Authoring Manager

Perception Authoring Manager is an application that runs in Microsoft Windows. It allows for the creation of assessments and provides a wealth of tools and question types. Authors can organize questions in a question bank structure and reuse questions as needed. Users can incorporate media and other elements into the question stems, choices, and feedback.

Learners may access tests directly from a Perception Delivery System or from a link they click on their Learning Management System (LMS). The SCORM- or AICC-compliant results of the tests can be sent back to the LMS if one is in use.

I have played with the Authoring Manager enough to say it is rather intuitive, though there are places where you may have to think things through a little before you start banging on the keyboard. For instance, you will want to create a set of questions before you create an assessment. You will also want to think about how you want to organize your questions. As in many cases in life, a little forethought can go a long way toward saving you rework.

Let’s start with some basic information you need to know to get the lay of the land. I always find it helpful to get a view of the forest before I start looking at the trees.

Assessment types

Perception Authoring Manager supports four assessment types. (See Table 2.)

Questions

You can see in Figure 1 and in Table 3 that the types of questions that the Authoring Manager allows you to build are much more extensive than what you will find in a typical authoring tool. I was quite impressed with the variety of questions that are possible.

Figure 1: The Question Wizard provides an extensive menu of question types for use.

Questions are always organized into topic folders. Topics can represent subject matter, such as Automotive Maintenance, or they may be more specific, perhaps Changing the Timing Belt. You may also choose to save questions in categories such as Easy, Medium, and Difficult.

You can assign metatags to questions, so that rather than create topics for difficulty levels, you can attach a metatag to each question in order to indicate its difficulty level. As an example, you may have 300 questions stored in a topic called Heart Bypass Procedures and assign a difficulty level of Easy to 50 of them. Do you find the idea of metatags scary? Don’t worry, they’re actually quite simple to conceptualize and use.

When you create a question, you can then save it in multiple topics. The question itself is stored only once so that if you change any part of the question, the changes will be reflected everywhere it is used. There is no need to edit each instance of the question. I really like this: not only does it save on storage space; it allows me to save a lot of time as I need make changes only once.

Media

You can use many different media types in your questions, which include those in Table 4.

When you create an assessment, you choose whether it is an exam, a test, a quiz or a survey. You then choose an assessment template. Templates are sometimes thought of as limiting factors. That is not true here. They provide the structure to set up questions.

Assessment templates

Templates control the look and feel of assessments as well as determine how questions are presented. For instance, you may choose to present one question at a time, the more traditional approach. However, you can also choose a template that puts several questions on the screen and lets the user scroll through the test to answer the questions, similar to how you take a paper test in school.

There are over 30 templates available in Perception and you can modify elements of a template. If you modify a template, every assessment that uses that template will reflect the changes. This is good if you are going to update your corporate colors or logo, for instance. You can also make a copy of a template and make changes only to the copy.

Selecting questions

Once you’ve chosen your template, you can then start to select questions to use in your assessment.

Figure 2:  Perception offers three different ways to insert questions from a topic into an assessment.

As you can see in Figure 2, you can choose to insert all questions that are in a topic into your assessment, or a random number of questions – from one chosen randomly all the way up to the number that are in your topic, or a single question of your choosing.

You organize the order of the questions and you’re done.

In practice, I have found this to be straightforward though it takes a little getting used to the power of choosing questions randomly because you don’t necessarily know which questions will be posted. No matter – that’s the beauty of randomization. As long as you have set up your questions in the topic the right way, then each learner will get a set of questions that is comparable to every other user, though they may not be the same exact questions.

Questionmark Live

Figure 3: When a SME uses Questionmark Live to enter and edit questions, this menu offers a subset of the question types available in Perception.

As indicated earlier, Questionmark Live is principally used by SMEs to enter and edit questions. When a SME sets up a new question, the choices in Figure 3 will appear. These are a subset of those question types available in Perception.

Questionmark Live is browser-based, so those using it do not need to install software on their systems, something I consider a big plus because it makes working with SMEs easier.

In addition, you can access Questionmark Live from a browser running either on Windows or on Mac OS X. Finally, the interface is stripped down and much easier for those who need to use it to enter and edit questions. There are WYSIWYG elements for entering text and media, for instance.

One other feature I consider very important is that you can work with shared question sets, so that multiple SMEs can be working on the same questions! I like having this option available. I like it a lot, because I have found in all my years of creating e-Learning that it’s very important to make it easy for SMEs to work on a project. They are busy people, and if you make their life difficult they will not be as likely to help. Having an easy-to-use tool like Live does a lot to remedy the situation.

Perception Browser-Based Authoring

You also access this feature from the browser and it gives you access to some features that Questionmark Live does not. Think of it as an interim between installing Perception on your system and using Questionmark Live. Whereas Questionmark Live is limited to working with questions, Browser-Based Authoring also lets you create topics and organize questions into assessments. While those who already have Perception installed on their systems do not normally need this, for those who are only going to be on a project for a little while, it may perhaps make sense to give them access to this browser-based product.

Delivering assessments and generating reports

In the introduction to this article, I spoke of the importance of being able to see the results of the assessments that learners take. This is crucial to determine how well learners are progressing and also to determine how effective tests and exams are. Of course, you also use reports to see the results of surveys you have created.

You can generate reports at an aggregate level, meaning groups of students who have taken an assessment, or for individual students.

There are twelve built-in reports available. For my money, reports answer the following questions.

• Are the test questions measuring what they’re supposed to measure?
• What exactly did learners learn?
• How do different groups of test takers compare?
• As a coach, how can I help learners who are behind to improve?
• What do learners think of the learning that took place?

Figure 4 is an example of an item analysis report.

Figure 4: Typical item analysis report from Questionmark Perception

Delivery of important exams can be problematic in life because of the possibility of cheating. Learners can sometimes browse the Web, or perform other actions to try to find answers. Perception remedies this by allowing for secure browser delivery, which means that learners cannot switch applications using Alt-Tab or use Instant Messenger. It will even darken external monitors to ensure that the exam taker is not seeing extraneous material. Of course, this is normally reserved for when you are delivering high-stakes exams, not for simple quizzes.

I must note that, when all is said and done, assessments run in a browser. You can also run them in many handheld devices such as the iPhone or the Blackberry.

Conclusion

I have worked with a variety of assessment tools over lo these many years, and have even helped to design an assessment tool in the past. I can see how far we’ve come since those days. Questionmark Perception provides a lot of options that I have found makes creating questions, setting up assessments and seeing the results, very useful in my working with the product. Is it a perfect product? No, of course not. That’s why there are always new versions. However, I can honestly say it never crashed once (a big plus), it is rather intuitive to use, and dare I say it can even be fun. (See the USDA case study.)

There are some new features in the latest version that I haven’t even touched upon here. One of them, the ability to deliver to multiple languages, is near and dear to my heart.

Overall, I see Questionmark Perception as a very powerful tool that is relatively easy to use and has a place in any enterprise that is in need of creating and delivering a number of assessments.

Comments

Login or subscribe to comment