It’s ninety percent complete, this on-line course you’ve been working on for months. The ones you love are sick and tired of hearing about all the interesting things you have learned about your subject matter and you’re not sure if there is anything left to learn about it. Your eyes are bleary from looking into the unending depths of your computer screen. Your brain is crammed so full of the course content, images, navigation, and assessments that you don’t care if you ever see this course or anything that has to do with your topic again for as long as you live. If you are lucky, your course has nothing to do with chocolate or ice cream.
At this point in the development process you’re ready to take your course out on the town, parade it around and get a little feedback. That’s right, it’s Beta Testing time. Pulses quicken, faces flush, hearts race at the fun times ahead. The excitement of the imminent vetting of your flawless work makes you giddy and high-spirited.
This is true unless, of course, you are in the normal, non-geeked-out, majority. You know, the rest of us who got into course development for reasons other than running tests, surveys, and data collection. Believe it or not, there are people who don’t get thrilled at the prospects of Beta Tests. People who are not even really sure how to run a Beta Test. People who get a little queasy about Beta Testing their baby in front of a bunch of strangers who have no comprehension of the amount of work it takes to put such a course together. Normal people.
For us normal folks, let’s take a look at Beta Testing and see if we can make it a bit easier to understand, perform, and use this process.
Beta means “not perfect”
Many developers believe that Beta Tests should only be conducted when the course reaches 100% completion. Our group at the California Department of Transportation (CalTrans) practices otherwise. For us, beta means not perfect. When we reach about 90% completion, we start talking Beta Test. This practice may have developed from compressed schedules, eagerness to capture feedback, or a desire to proceed with the process while the Web developers, Flash geniuses, and graphic artists were fine-tuning the last bits and pieces of the course. In our experience, early Beta Testing — before the product has reached 100% — has not significantly hindered our results and may give us other benefits with schedule and participant feedback.
When it comes to showing their work in progress, course developers are often like artists, no peeking until it’s finished. Most developers are hesitant to release an unfinished product for testing. This is a reasonable and normal concern. Nobody likes to be opened up to critical comments, no matter how helpful, when they are not even finished with the product. However, when it comes to Beta Testing, in our group unfinished is just fine and even preferred. There is still time to make a few tweaks, and developers can continue working on the project while the Beta Test is in process. In my experience, testers are more willing to make comments when they can see that the work is in progress and not a finished product waiting for their blanket stamp of approval.
Beta Tests for on-line courses are not meant to review the content. A subject matter expert (SME) provided the content, and probably double- and triple-checked it for accuracy. The purpose of the Beta Test for an on-line course is simply to check the effectiveness, usability, and functionality of the course from a typical user perspective. Is it easy to understand the navigation? Are the assessments intuitive or well explained? Do users get bogged down anywhere? Do the learners learn? Do the colors and layout work, or does neon green text and 80 images on the page actually hurt the user’s brain?
It’s important to make this distinction when you are preparing your users for their experience in the test. Let them know that the test is not about content. Tell them that they are helping you refine the functionality of the on-line experience. Explain that the questions afterward will be about the ease of use and functionality of the course, and not about the accuracy of the content taught in the course. When you make these distinctions before the testers begin your results will be more helpful and complete. These statements will also help ease the nerves of some of your testers who feel like they need to be experts in the subject matter of the course they are reviewing.
Picking the Beta Test group members
One of the key elements required when running a Beta Test is a group of willing victims (make that participants). It can be a real challenge for some developers to identify the right people to participate in the test. Many developers wonder how they can find 15 to 20 experts to review their on-line course. However, from the previous discussion you know that the test participants shouldn’t be experts in the subject matter.
Remember that we are simply testing the presentation of the material in the on-line environment. Since we are not testing the content we don’t need or want subject matter experts for the Beta Test. Test participants should possess about the same level of knowledge as the average future user of the on-line course. Since we typically invest in and build on-line learning for large audiences, this level of knowledge should be very easy to find in a willing group of 15 to 20 people.
If you work for a large organization you will probably have no problem identifying potential Beta Testers. If you are a smaller shop or a one man show you may have more of a challenge here. Smaller course development businesses should look to their client or potential clients for possible Beta Testers. You can always rely on family and friends for help if they possess the knowledge level of your typical target learner. However, if you’re building a course on concrete pavement design or high intensity LED optical testing, you’re going to have to go farther than the family reunion for a quality test group.
Once a participant pool is found it’s important to realize that there is a difference between identifying Beta Testers and getting them to participate in a test. Even in a larger organization it can be frustrating to find a group of people that will take the time to help you. Here at CalTrans we first began gathering groups by pulling people out of their cubes for a few hours, or sending out e-mails to different work groups asking for a big favor. Response to this method was less than stellar.
One solution to this challenge that we have adopted here in our big organization can be very effective for the small shop as well. Avoid the hunt all together and shift the burden to someone else. This timeless work-management technique is called “passing the buck,” and can be very effective.
Through experience, some developers have learned to include a paragraph in the project charter or contract that alerts the sponsor of the future Beta Test, and requires the sponsor to provide a group of 15 to 20 participants. A good clause might identify the participants as individuals at similar knowledge levels as the potential course users and provide an estimate of time required to complete the test.
This approach has worked wonders for the sanity of course developers who can become easily frustrated with busy people unable to find the time to complete the Beta Test. When the sponsor shoulders the burden to provide the Beta Test participants, the message to complete the on-line course is less of a favor and more of a mandate. When the sponsor is aware of this requirement early it is hardly an issue of concern.
Of course, this works well when you are building a course with a client or sponsor and a contract or charter. If you’re building a course on spec in hopes of selling it once it’s complete, you’ll have to find your Beta Test participants on your own. But their glowing comments in the feedback will be great marketing tools to help you sell your excellent product, right?
Test on-line, or in person?
So you have your group of testers. They are all eager to be the guinea pigs for your on-line course. Now you get to decide how to conduct your test. In a traditional classroom course there is really only one way to run a test; find a room, fill it with people, and teach your course. One of the many benefits of on-line training is the freedom to choose your testing methodology. You can use the traditional approach with a few adjustments, or you can go high-tech with on-line participation in multiple locations.
Classroom test
It’s only logical that the first Beta Tests for on-line courses that most of us run are hybrids of a traditional classroom method. We naturally extend our previous experience and knowledge into practice with the new technology. When we have taught or built courses for the classroom, and conducted Beta Tests in a classroom setting, we feel very comfortable with a live classroom test of our on-line course. This is not a problem. There are serious advantages to this method and some disadvantages that we’ll look at as well.
Process
When Beta Testing in this fashion, the project team finds a classroom full of computers, rounds up 15 to 20 people, all of whom will make their way through the on-line course individually. Observers hover around the room answering questions and taking notes. After most users are finished, a facilitator conducts an informal focus group, which is a fancy way of saying the facilitator asks specific questions, and has a guided conversation with participants about their experience in the on-line course environment. The facilitator and observers take lots more notes, and look for common concerns and threads of issues that will help refine the course.
Advantages
This is an excellent way of conducting a Beta Test. Advantages of this approach include timeliness and richness of information. When conducted in this manner, a Beta Test can be completed in one day. Your test takes up only one box on the calendar. The other method discussed below often takes one to two weeks. When you’re under a tight schedule, a one-day Beta Test is a huge benefit.
The quality feedback you get from this approach is also a plus. It’s been my experience that the feedback from these tests is rich in amount and detail. Because the facilitator conducts a guided conversation in a group format, the comments of participants prompt others to share more than they might in a one-on-one interview or survey situation. The focus group model allows the facilitator to dive deeper into any areas that present more concern or interest. It allows more freedom to immediately follow up on issues or concerns that rise in the course of the conversations, and allows the facilitator as well as every participant opportunities to seek clarity and understanding on questions and answers.
Disadvantages
While this is an effective way to perform a Beta Test, there are some disadvantages or challenges that need to be considered. This type of test is an event and comes with all the logistical issues implied of any event.
The first issue to consider is venue. Where will you conduct the Beta Test? You need a room with 10 to 20 computers. The size of the test group is limited by this constraint. If the available space only has five computers, then you can only have a test group of five or you must conduct multiple sessions. While many of us may know of computer labs at schools or businesses, the price or schedule may hinder our efforts. I work for a huge state government department with multiple computer labs and a contractual relationship with a nearby state college with computer labs and can still be challenged by scheduling issues.
Once you have a location all ready to go, you must get all your staff and Beta Testers into the same place at the same time. Make sure you provide clear instructions and a map to the location if needed. (See Figure 1.) It’s probable that two or three of your participants will not be able to make your test date or location for one reason or another. This isn’t a deal breaker, just don’t be surprised when your group of 15 turns into a group of 12.
Figure 1 This e-mailed flyer provides instruction and directions for Beta Test participants. Note the image of the cookies, a little bribe never hurts!
Set the stage for success by checking each computer, open the browser, navigate to your on-line course, ensure the mouse works, find headphones for each station if you have audio, and so forth. I also try to always bring a few plates of cookies and some napkins to make it feel more casual and friendly. How can you possibly go wrong with treats?
A traditional classroom-based test of your on-line course will have its challenges. However, the benefits are definitely there. Most of us are pretty comfortable with this approach, the feedback is rich and immediate, and since we can finish in one day, the classroom approach is a great way to go when the schedule is tight.
