Delivering Computerized Assessments Safely and Securely

Security is a pressing concern for organizations using computer-based assessments, but so is cost-effectiveness. It’s essential to maintain the integrity of item pools, keep test scores secure and minimize the possibility for cheating — but without an exorbitant price tag. The thoughtful use of technology, combined with careful consideration of an assessment’s purpose and consequences, makes it possible to deliver assessments safely, securely and economically.

Over-engineering low-stakes assessments can result in unnecessary costs and wasted time. Under-engineering high-stakes assessments can undermine people’s confidence, organizational processes and the face validity of an assessment. This article provides a methodology for assessing the security requirements of your computer-based testing programs. It will help you implement appropriate security measures based on the purpose and nature of assessments. It will also explore the various means and technology available to enhance the safety and security of a wide range of computerized assessments.

Differentiating between assessments

The first step in determining security levels for assessments is to distinguish between different types of assessment. One important distinction is whether you are using computerized assessments to:

a. Measure knowledge, skills, attitudes, and personality traits; or,

b. Promote learning and reduce forgetting.

Another way of classifying assessments is according to their consequences (or stakes). The potential outcomes of an assessment could affect a participant’s propensity to cheat and consequently how much security must be applied. After high-stakes assessments people might be hired, fired, promoted, demoted, graduated, not graduated, and/or authorized or certified to perform a particular job or role. Other assessments, with lower stakes, have far less significant outcomes.

Assessment purposes

The method of delivering a computerized assessment depends largely upon the use and consequences of the results. The participant’s motivation to complete the assessment is important too. This often relates to the nature of the assessment, which I’ll define with the following terms:

Formative: designed to help students turn information into knowledge by asking questions that tend to increase remembering and reduce forgetting.

Summative: designed to measure knowledge, skills and attitudes by asking questions and measuring the responses.

Diagnostic: designed to provide a diagnosis and prescription to help people reach their objectives.

Assessment levels

Assessments are often referred to as exams, tests, quizzes, or surveys depending on whether the assessment is intended to measure or promote learning. For the purposes of this article, I’ll define an assessment as any systematic method of obtaining evidence by posing questions to draw inferences about the knowledge, skills, attitudes and other characteristics of people for a specific purpose.

There are four levels of assessments, mentioned above. Here are the definitions as they are used in this article:

Exam: A high-stakes summative assessment used to measure and document a student’s current knowledge or skills. Its purpose is to measure knowledge. Consequences for passing or failing an exam might extend to being hired, fired, promoted, demoted, released from custody, authorized or certified, graduating or not graduating. People who take exams are highly motivated to complete them successfully.

Test: A medium-stakes, diagnostic assessment to measure a student’s knowledge or skills. A test is designed to inform the student or their tutor of the student’s current level of knowledge or skill. Its purpose is to measure knowledge in order to promote learning. The consequences for passing or failing an individual test are not high — however, consistent failures might extend to being hired, fired, promoted, demoted, graduating or not graduating. People who take tests have a medium-level incentive to complete them successfully.

Quiz: A low-stakes formative assessment used to measure a student’s knowledge or skills and provide feedback. It is designed to provide feedback to inform the student of their current level of knowledge or skill. Its purpose is to promote learning. There are few if any consequences for passing or failing a quiz. Quizzes are normally formative assessments used to promote learning and reduce forgetting rather than forming a summative judgment. The motivation to complete a quiz is medium.

Survey: A low-stakes assessment to measure the opinions, attitudes and sometimes knowledge and skills of a group. Its purpose is to guide thinking about how to improve learning activities and job aids for individuals, the group, or future groups. There should never be direct consequences for a survey respondent as you are looking to understand true opinions and attitudes. However, as results are tabulated for the group, training courses, job aids, and other interventions may be planned. Sometimes the opinion might have consequences for others: an instructor in the case of a course evaluation or managers in the case of an employee attitude survey. Despite the low stakes of surveys, the sensitive nature of the information they collect can call for considerable security. (See Case Study #1.) It goes almost without saying that the motivation to complete a survey is generally low. Indeed, the challenge is to get people to respond!

This article does not attempt to describe all of the consequences, legal liabilities, validity, reliability and planning issues that are involved with writing an assessment that is suitable for its purpose. Many other publications detail the processes and procedures used to produce good and fair assessments. However, as you can see from Figure 1 all of these issues are related to the stakes of an assessment.

FIGURE 1 Low, medium and high stakes

Delivery issues

Good decisions about assessment environments begin with considering the distinct set of delivery issues that pertain to each type of assessment. The higher the value of the assessment, the more you will want to control the delivery environment and the more supervision you will require. Figure 2 details some of the issues that you must consider when deploying computerized assessments:

FIGURE 2 Delivery issues

Delivery options

Now that you’ve considered the nature, purpose and consequences of your assessments, you need to consider your options regarding assessment environments. High-stakes assessments must offer a consistent environment that is free from distractions together with some form of monitoring by an invigilator or proctor. Low-stakes assessments do not require such rigor. The greater the rigor required, the more expensive an assessment will be to deliver. The sponsoring organization that owns an assessment stipulates the physical environment, technology, monitoring standards and authentication methods for delivering it. (See Figure 3.)

FIGURE 3 Delivery method

Where should assessments be delivered?

You need to consider a wide range of delivery methods when planning assessments. An assessment center offer controlled environments where groups of users can take assessments under conditions laid out by the organization that is running it. This means that candidates can be sure of fair conditions and equal treatment. Assessment centers can be used for a wide variety of assessments, depending on the way they are set up and administered.

Physical Environment. Assessment centers should be enclosed rooms without through-traffic. They should be quiet and free from distractions, well lit and with a comfortable temperature throughout the year. Computers should be separated from one another by acoustic partitions. The equipment should offer similar operation, accessibility options and performance. Candidates should not be able to print or capture the assessment content; they should be prevented from accessing resources, such as Web pages, cell phones, digital phones and digital cameras that might unfairly help them during the assessment process.

Technology environment. Assessments should be delivered on consistent and comparable technology. Screen sizes, processor speeds, and network connections do not have to be the latest and greatest, but they do need to be comparable.

Monitoring. You can monitor (or proctor/ invigilate) assessment centers in a number of ways. Candidates should be in an environment that permits constant visual monitoring. This can be achieved directly by a person in the room or via video surveillance equipment.

Authentication. It is important to confirm that the person taking the test or exam is truly the person authorized to do so. This can be done by confirming a candidate’s identity via a photographic I.D. provided by the government, academic institution, or, in some cases, by their employer or simply by a signature.

I use the following grading system to indicate the suitability of different delivery methods for particular assessment types:

1. the method is very suitable;
2. the method is suitable;
3. suitability will vary according to circumstance
4. the method is unsuitable;
5. the method is definitely not suitable.

Which type of center suits which type of assessment?

Computerized assessment centers come in various guises. Some of the ways in which they can be defined are as follows:

Professionally controlled centers. Professionally controlled centers provide a very consistent and highly controlled environment where a candidate could expect to receive exactly the same treatment and experience from one assessment center to another. This consistency would start with their reception and continue through the assessment process to the time they leave the premises. These centers are used for very high-stakes tests such as nursing and medical exams. They are expensive to use and maintain. Professionally controlled centers provide the consistency and rigor required for very high-stakes, government regulated certification exams. Proctors at these centers will ensure that candidates do not enter the testing room with banned devices such as cell phones, photo phones and digital cameras.

These centers are rarely used to deliver non-regulatory exams due to the high costs associated with these centers. Here’s how they score for the various types of assessments we have discussed:

Franchised centers. Franchised centers are similar to professionally controlled centers, but they cannot enforce the rigor required for very high-stakes assessments. However, these centers are less expensive and provide a great place to offer high-stakes tests in a reasonably consistent environment. These centers are far more prolific than professionally controlled centers. They deliver government regulated exams as well as non-government regulated exams. There are approximately 5,000 franchised centers around the world. How do they rate for different assessment types?

Higher Education centers. Many higher education institutions offer their testing facilities to local companies and are sometimes part of a franchised assessment network. These centers are similar to professionally controlled centers, and they enforce the rigor required by their institution. These centers are less expensive and provide a great place to offer high-stakes tests in a reasonably consistent environment. These centers are ideal, not only for the institution’s use but also for local companies that need to administer high-stakes assessments in conjunction with a local institution. Here’s a report card for them:

Corporate centers. Increasingly, large organizations are establishing their own assessment centers to provide a consistent, confidential, and cost effective method to deliver high-stakes assessments. These centers might be testing potential employees, partners or existing employees on safety or government regulation related issues. These centers are equipped in a similar fashion to professionally controlled or franchised centers. Their grades for different types of assessments look like this:

Non-dedicated assessment centers

Most assessments, computerized or otherwise, are taken within non-dedicated assessment centers. These facilities are used for other purposes but can be used effectively to administer quizzes, tests, exams and surveys. The rapidly improving features, quality, ingenuity and authentication capabilities of assessment software make it possible to run many higher-stakes assessments in these centers. Non-dedicated assessment centers can be one of the following:

Training rooms doubling as assessment centers. Training rooms may not offer as tightly controlled an environment as a dedicated assessment center. For example, a training room is unlikely to offer screening between computers. But you can overcome such limitations with a little thought and ingenuity. Professional assessment software now offers technology that can turn training rooms into assessment centers. Professional software packages can prevent printing, capturing assessment content, exiting inappropriately and task switching. They can also enable you to randomize the distracters in multiple choice and response questions or even create an assessment from randomly-drawn questions, thereby making it difficult to cheat. Note the particular suitability of training rooms for surveys and tests:

Closely supervised in the work place. Of all the issues that affect the delivery of computerized assessments, monitoring (invigilating or proctoring) remains critical for delivering high-stakes certification exams. Local supervisors or managers can monitor widely dispersed candidates. This makes supervised assessments in the work place a practical, valid and cost effective way to provide high-stakes assessments. Beyond that, the advances in assessment software mean that employees can take even very valuable assessments at their own desks with limited supervision. Assessments can open at a specified time, making sure that the candidate cannot get a “sneak preview.” Although you wouldn’t want to offer very high-stakes exams in a supervised work place, you can certainly consider this for other purposes:

Remotely supervised in the work place. It is now possible to monitor candidates remotely using video cameras. Although not suitable for very high stakes assessments, this can provide a valid way to administer exams remotely.

Unsupervised at the work place. Low and medium-stakes unsupervised assessments at a participant’s desk or other work venue are becoming increasingly important. Organizations are striving to discover how effective their training is, to identify the people who need training and to provide new ways to promote learning. This form of assessment doesn’t require the same monitoring, but it does require assessment software that prevents people from printing or capturing confidential information.

Supervised at home. Some distance learning institutions and small-scale certification authorities are now administering higher-stakes assessments at home. However, the candidate must provide the organization with a professional in the field of study (e.g. a doctor, lawyer or accountant) to supervise the assessment. Once again, the success of this kind of delivery lies in the assessment software and how monitors are selected.

Unsupervised at home. Most commonly used for delivering low-stakes self assessments aimed at promoting learning and reducing forgetting.

Unsupervised in a public place (library).

Most commonly used for delivering low stakes self-assessments aimed at promoting learning and reducing forgetting.

Whatever choice you make about assessment delivery, using technology judiciously can improve the safety and security of assessment content and answers, at low cost.

Creating the right assessment environment

Delivering assessments securely is becoming easier as the quality of networks and assessment software improve. Secure assessment players and various monitoring

Solutions are making it possible to deliver many types of assessments safely and securely, either remotely at the desk or in a training room. When close monitoring and a consistent environment are essential for geographically dispersed candidates, there are many excellent assessment centers to choose from. But many organizations are now being empowered by assessment software to create and run high-stakes assessments within their own environments. As the quality of software, monitoring and network solutions grow, this trend will certainly continue.

A combination of software settings and proctor guidelines helped a consortium of communities address the security concerns presented by a distance learning program for surgical technicians. (See Case Study #2 in the sidebar.) And even with certifications, creative thinking and planning can do away with the need for proctors. (See Case Study #3.)

Creating the physical environment

The ideal assessment environment would include the following characteristics:

• Comfortable, clean and smoke-free
• Adequate lighting, ventilation, comfortable seating and work surfaces for the candidate during the assessment session
• Free from external distractions
• No opportunity to use equipment such as printers, fax machines, copiers or telephones while taking an assessment
• Provide a viewing window and/or video surveillance system
• If a monitor is present they should have an unobstructed view of each candidate within the assessment room.
• IT equipment should be consistent and robust and comply with the local accessibility rules and regulations.

Deploying software to deliver secure assessments

Although many issues can be addressed within a physical environment, assessment software can help ensure that assessments are delivered in a secure way. These include:

Randomization and shuffling

There are many safeguards that can be taken during the assessment authoring process to help protect the security of the assessment. One of the most popular is to shuffle the order of the choices. You can also deliver the questions themselves in a random order. Both of these features help prevent cribbing when users are sitting in non-screened assessment centers.

Encrypted communications

Most computerized assessments are now delivered via the Internet or an intranet. It is important to encrypt communications between the browser and server. This means that if someone ‘sniffs’ the network they would not be able to see the information sent from server to browser and vice versa. This can be done with Secure Sockets Layer (SSL), a protocol that allows the browser and Web server to encrypt their communication. This means that anyone intercepting the communication will not be able to read it.

Scheduling assessments

Some software enables users to set start times of assessments, the length of assessments, and the number of times that assessments may be taken. You can also enable user names and password protect your assessments.

Monitoring assessments

A “monitor” option ensures that a participant cannot start an assessment until a proctor or invigilator has logged on after the candidate. The monitor can also be limited to a range of IP addresses to ensure that a certain physical location is used to administer the assessment.

Secure browsers

You can now ‘lock down’ computers that are being used in assessments to keep users from accessing other applications and Websites while engaged within a medium- or high-stakes assessment. This lockdown prevents candidates from printing, capturing screens, accidentally exiting the assessment viewing source, task switching, etc. The server must effectively authenticate the secure browser to prevent spoofing by technically competent cheats.

Conclusion

Sidebar 2 summarizes software features and their benefits as applied to each level of assessment.

Organizations have a wide choice of assessment environments, authoring practices and technologies that make it possible for them to deliver assessments of various types safely, securely and economically. It’s essential to consider the nature and purpose of assessments and use appropriate delivery environments in order to balance security requirements with the need for cost-effectiveness.

Comments

Login or subscribe to comment