It has been a while since diagnostic assessments have been at the forefront of learning and development discussions. However, they have the potential to address emerging issues in your workplace. Let’s revisit the purpose of diagnostics and the impact they can make in online learning.
First, how would you define a diagnostic assessment? It is often described as a set of questions that evaluates a learner’s prior knowledge in relation to specified learning objectives. While this definition appears to be straightforward, if you ask two online learning professionals why they use diagnostics, their answers will likely be different. In reality, the results of a diagnostic can be used in a wide variety of ways to achieve many different objectives. Diagnostics can extend beyond typical knowledge-based questions to include items that assess a learner’s experience, confidence level, motivations, interests, or personal needs.
In our experience, many learning professionals have not been using diagnostics to their full potential. Before reading on, ask yourself these questions:
- Do I currently use diagnostics in my online learning?
- What value are my diagnostics bringing to learners? To the organization?
- Could I use diagnostics more effectively?
In this article, we will explore the opportunities that diagnostics present to learning professionals. We will discuss the benefits of diagnostics in asynchronous online learning and offer practical steps and ideas to designing a diagnostic that meets your business goals.
Benefits of a diagnostic
Diagnostics offer a broad range of benefits when incorporated into asynchronous online courses. Consider the following:
- Personalized learning paths: Each learner is unique and may benefit from personalized learning paths based on diagnostic results. For example, you may provide advanced learners with a recommendation report that outlines their identified areas of weakness. The report will help them to focus on priority content and move through the course more efficiently.
- The right learners in the right courses: A diagnostic that aligns with a final test can help you determine which learners should enroll in a course, an effort that allows your organization to manage employee time more effectively.
- Measurable evidence of learning: Diagnostics can provide learning managers with numerical evidence to demonstrate the value of a course. You can measure the progress of learners by comparing their first-attempt results from the diagnostic to those of the final test.
- Global learning needs: You can aggregate diagnostic results for multiple learners or even multiple courses to identify where to allocate training resources based on high-priority learning gaps. In this way, diagnostics can be part of a more comprehensive approach to performance management.
While there are many benefits, diagnostics may not be appropriate for all situations. When would they be ideal? If you can answer yes to any of the following questions, a diagnostic might be right for you:
- Do my learners have a degree of prior knowledge in the subject area?
- Is my audience made up of both novice and advanced learners?
- Is my course a reasonable enough length to provide custom recommendations?
- Can some learners benefit from approaching the course in a non-linear fashion?
- Do I want to give learners more control over their learning experience?
- Is my department required to show tangible evidence of learning?
This is by no means a comprehensive list, but it can help point you in the right direction.
Steps for designing a diagnostic
Now that we have discussed why diagnostics are useful and when they are suitable, let’s get started on how to design an effective diagnostic assessment. We have come up with five simple steps:
- Define your goal
- Identify impact on course design
- Assess learning objectives
- Determine question format
- Develop a message to learners
Step 1: Define your goal
Your first step is to clearly define your goal for using a diagnostic. If you have multiple stakeholders, we recommend validating the goal to ensure the solution aligns with each stakeholder’s expectations. You may be tempted to create a diagnostic to serve multiple and/or dissimilar needs, but keep in mind that conflicting priorities may ultimately dilute the effectiveness of your diagnostic.
Example goal:
The diagnostic in Accounting 101 will generate study recommendations for learners based on their personal strengths and weaknesses, allowing them to plan their study schedule accordingly. This solution responds to the different learning gaps among learners who come from varied educational backgrounds and experiences.
Step 2: Identify impact on course design
Next, you need to determine whether or not the results of the diagnostic will change the course design and in what way. If you choose to change the course experience for the learner, Table 1 provides some suggestions. The options are not mutually exclusive, so you may want to choose a blended solution.
Table 1: Suggestions for course adaptation based on context
Step 3: Assess learning objectives
Once you have defined your goal and know your intended impact on course design, you can begin to build a diagnostic that will meet your needs.
First, establish which topics and how many questions should be in the diagnostic. We recommend analyzing your learning objectives so that the diagnostic focuses on core learning areas rather than “nice-to-know” ideas. A common practice for instructional designers is to prioritize learning objectives by ranking them according to criticality, frequency, task difficulty, or domain size. (Note that assessing learning objectives may not be as critical if the purpose of the diagnostic is to identify which topics are of interest to the learner or how they prefer to learn—their motivation.)
With your prioritized learning objectives, you can determine the domain weightings for the diagnostic. If you are building a diagnostic that must be comparable to a final test, ensure that you break down the results of both assessments in the same way. Here are a couple of challenges we have encountered when aligning the diagnostic to the final test.
Challenge 1: The diagnostic needed to be shorter in length than the final test.
Our solution: Reduce the number of questions for the diagnostic but ensure that the number of questions are proportionate to the weightings of the final test. Table 2 shows a sample arrangement.
Table 2: Making the diagnostic proportionate to the final test
Challenge 2: We needed to deliver diagnostics at the module level but deliver the final test at the course level, making it difficult to measure the progress of learners from start to finish.
Our solution: Ensure you categorize the final test results in the same way as the diagnostic results. Display the results to learners using these categories so they can clearly see their improvements. Here is a sample report:
Step 4: Determine question format
Now it is time to design the questions themselves. Questions may evaluate any of the following including knowledge, skills, experience, confidence levels, interests, and motivations. The goal of your diagnostic should drive which type of questions are appropriate. If you mean to compare the results of your diagnostic to be compared with the final test, your question format will likely be exam-quality items mapped to taxonomy levels. On the other hand, if you mean for the diagnostic results to provide a more efficient study path, your question format may include rating scales to address more subjective measures such as confidence levels or study interests.
Challenge 3: We were concerned that learners who did not know the answer would guess correctly, which would produce misleading diagnostic results.
Our solution: Include an “I don’t know” option in the question. In the diagnostic instructions, encourage learners to select the option when they are unsure of the correct answer so that they can receive a more accurate recommendation report. From a tracking perspective, treat the “I don’t know” option as an incorrect answer.
Challenge 4: We thought that learners with high levels of experience may possibly answer knowledge-level questions incorrectly, which would produce misleading “beginner” recommendations on how to approach the course.
Our solution: Integrate questions that ask learners to rate their levels of experience. Map the scores from both the experience and knowledge questions to recommendation levels. Learners who have high experience ratings but low knowledge scores will no longer receive a “beginner” recommendation but rather an “intermediate” one. Table 3 below shows a sample mapping.
Table 3: Sample recommendation mapping
Step 5: Develop a message to learners
Before launching a diagnostic, it is important to position it properly with learners so they understand why and how they are being assessed. If a learner fails a test prior to starting a course, it could negatively impact the learner’s motivation. You can mediate his outcome through proper positioning of the diagnostic. In your message to learners, we recommend addressing the following questions:
- Why am I taking a diagnostic before starting the course?
- Do I have to take the diagnostic?
- How long will the diagnostic take to complete?
- Can I re-do the diagnostic?
- How do I use the recommendation report (if provided)?
The answers to the above questions will vary based on the goal of your diagnostic. Consider the question, “Do I have to take the assessment?” If your diagnostic is intended to provide a streamlined learning path for advanced learners, it will not be useful for novice learners and should not be mandatory. On the other hand, if your goal is to track evidence of learning, the diagnostic should be taken by all learners. In both situations, you should clearly communicate the purpose of the diagnostic to learners so they can approach it accordingly.
Summary remarks
You have explored the benefits of diagnostic assessments and when they are suitable. If you are ready to create a diagnostic, try integrating the five-step process into your work. We believe there is a great opportunity for diagnostics to resolve critical challenges in our field today. Learning professionals are looking for new solutions as the demand grows for personalized learning experiences, as organizations increasingly require accurate reporting, and as the field of knowledge management emerges. Diagnostics are powerful measurement tools that have the potential to both meet organizational needs and transform learning experiences.
How have diagnostics helped you and your organization? What challenges have you faced? We are interested in your experiences on how you have used diagnostics in your online courses.