Beta Testing an Online Course

On-line test 

The second option for Beta Testing is perfect for the small operation without large resources capable of providing a classroom full of computers. This approach can be done with a team or with a one-person company. It may cost very little, or absolutely nothing depending on your current resources. Conducting an on-line test is also a natural choice for an on-line course. Using this method simply brings one more step of the process into the bright and shiny world of technology. Fun!

Process

With the on-line method, the nearly finished course is hosted on-line. The course is uploaded to a Web site that the testers will be able to access from their offices or homes. Once the course is on-line and ready for testing, the test facilitator sends out a notice to the test participants that informs them that the test is up and ready for their review. The notice gives the Web address of the site, and invites them to test the course at their convenience (within the next two days, two weeks, or whatever your time frame is). I typically give the participants one to two weeks to participate in the course, and I send reminders out daily to encourage participation.

Don’t forget the reason for a Beta Test is to gather feedback. When conducting an on-line Beta Test the easiest way to collect results is to continue the method with a simple on-line survey.

Our group uses Survey Monkey, an on-line survey service that provides a free account to anyone, with some minor limitations that are easy to accommodate. The free account allows you ten questions and 100 participants per survey. As you will see, the ten-question limit is a little malleable and just right for the kind of information being sought in a Beta Test. Of course, the 100-participant limitation has no bearing on Beta Tests. As mentioned earlier, a group of 15 to 20 users is more than adequate for most Beta Tests.

The Survey Monkey system is easy to use. You can build surveys for your Beta Test quickly, and made them available to your participants with a link in an e-mail. Build your survey before you release your Beta Test, and include the survey link in your original notice to the test participants. Let them know that when they are finished with the course they need to go to the survey to share their experience with you.

Another option is to imbed a link to the survey at the end of the on-line course. This is a temporary element, of course, and will be removed in the final product. This allows for a smoother transition, and possibly less confusion for the Beta Test participants.

Remember, the point of a Beta Test is to measure the functionality and usability of the course and not the content. The questions in the survey reflect these goals and do not test for learning of content. The validity of the content is tested by the assessments built into the course; games, quizzes, activities, tests, and so forth.

Due to the nature of the on-line course approach, most surveys for on-line course Beta Tests are very similar in our organization. Table 1 is an example of a survey we recently used for an on-line course Beta Test.

This test was created on a free Survey Monkey account. While Survey Monkey only allows for ten questions on a free account, we can massage our survey to include more. In this example we actually have 15 questions that the software only sees as nine. Notice that question 3 is a matrix of seven questions. Survey Monkey views this as only one question. This is a little trick that, I am sure, Survey Monkey graciously allows us to use for our benefit.

Take a look at question number 7. Remember, the instructions stressed that Beta Tests are not about content. However, no matter how many times you have told test participants this, there will be a few who found something they have issues with in the content. If you don’t give them an outlet for this information, they will e-mail the facilitator directly, place the comment in a different field, or simply speak disparagingly to others about the course and all the content flaws they seemed to find. Even if you know that the content is perfect and the comments about content are wrong, your testers still need an outlet for their concerns. They’ll be happier, and you’ll be less harassed. Plus, they might be right.

Two concerns should be on your mind when building a Beta Test survey: question quality and question quantity. I hesitate to use too many questions in fear of brutalizing the test participants. Our team prefers to target the areas of concern with a limited amount of good-quality questions, and allow comments to gather the thoughts of the test participants who need to share more. Table 1 is one example of this approach, and more can be found on-line. As you build your own survey instrument remember what your objectives are, and target your questions accordingly.

Advantages

On-line Beta Testing is the way to go when you are looking to save costs and accommodate participant schedules. Because the process eliminates the need for a classroom full of computers, we save on these resources. This method allows users to participate in the test from their own computers at their own locations. The logistics issues of bringing everyone together in a computer-equipped classroom are non-existent. No worries about maps, parking, security clearance into a building, or even providing cookies.

This method also caters to the varied schedules of the test participants. When given an extended period of time to complete the course, like one or two weeks, your users can log into the course at a time that is convenient for them. You and your users are not locked into a set two-hour period to participate. They may even have the option of testing the course from home in their bathrobe and bunny slippers. Who doesn’t want that?

This approach is also relatively easy for a course development team to implement. Simply host the course on-line and inform the participants of a time window to complete the course. The development team is free to continue on the project and simply wait for feedback to roll in through the on-line survey. Some of my team members wait with eager anticipation and check Survey Monkey daily to monitor the feedback as it posts.

Disadvantages

On-line Beta Tests are inexpensive, accommodating to participant schedules, and easy to conduct. However, an on-line test can take much longer than a one-day classroom approach. Even when testers are mandated by their supervisor or an SME, an on-line test is not often a high priority. In real life, other work issues bombard your testers and pull them away from you and your project timeline. Patience and communication will help you persevere. Daily friendly reminders to participants, SMEs, and other stakeholders are a valuable tool to reach the end of your test period successfully.

The richness of information gathered in an on-line Beta Test may also be lacking when compared with a traditional classroom test and feedback session. When you use a survey to gather your feedback you usually limit your data by the questions you ask. In a classroom approach, a focus group feedback session allows the facilitator to follow threads of discussion and delve deeper into possible areas of concern. This is not the case with a survey. Unless your participants volunteer information, you will only get feedback on issues you address in the survey questionnaire. Identifying your concerns and developing your questions are important tasks in this approach.

Using your results

Once your test is complete, whether it’s a classroom or on-line approach, you should have some valuable feedback that will guide you as you make the final touches on your course.

The in-course assessments will help you identify problem areas in the content. If you have students who consistently miss questions in specific knowledge areas you may need to reassess the content or the questions.

The feedback from the classroom focus group or the on-line survey will provide you with information on ways to improve the usability and functionality of the course. It may be challenging to sift through all the feedback, and identify the improvements you will make as well as the ones you won’t.

Since budget and schedule limit all of us, we must be frugal with the changes made at this point. Simple corrections and adjustments are fast, inexpensive, and easy to integrate. Large-scale system-wide changes are expensive, time-consuming, and, at this point, should be unnecessary. If the course design is sound and the content is correct, there should be very few changes needed.

Feedback that asks for color changes or changes in the instructional methodology is often ignored. Remember, a trained professional (you) has already used sound instructional design principles and a subject matter expert to build the course. Trust your experience and skill, trust your subject matter expert, and don’t try to please all the test participants.  Look for those few little feedback treasures that make sense, and which will enhance your course.

Conclusion

The purpose of the Beta Test for an on-line course is to simply check the effectiveness, usability, and functionality of the course from a typical user perspective. Course developers and project managers look for feedback from test participants to ensure that the course is effective and usable for a typical user. We looked at two different ways to conduct a Beta Test; the traditional classroom method and the online approach. Both techniques have their positives and negatives. The classroom method allows you to accomplish your Beta Test quickly with deep feedback but can be expensive. The online method is almost cost free but may take some time. The feedback from the online method hinges on your closing survey questions, with little or no opportunity to explore deeper issues.

For more information, I recommend reading Barb Lesniak’s great article, Putting It to the Test: Quality Control for e-Learning Courses from the July 30, 2002 Learning Solutions Magazine. Ms. Lesniak explains the Alpha and Beta Test processes, discusses why they are important, and provides tips on how to conduct them.

While some may take issue with this approach, I encourage Beta Testing even when you’re only at 90% complete. It doesn’t need to be perfect to check it for perfection. Beta Testing is a great way to double-check your work before releasing it for general use. The Beta Test allows you to look at the effectiveness of your instruction and the usability of your online structure. Proper preparation will yield valuable feedback that will guide you to make small changes with big effect. The changes you choose to make will enhance your final product and build your stakeholder’s confidence in you and your course.

Comments

Login or subscribe to comment

Related Articles

Harbinger Raptivity: Packaged Interactivity

4/14/10

Raptivity (Harbinger Knowledge Products Inc.) allows designers to create and customize over 225 types of interactions. Interactions are Flash SWF files and can play independently or within any authoring environment that accepts SWF files. You don’t use Raptivity to build whole courses, rather, you build rich interactivity with it, one interaction at a time.

Insure E-Learning ROI: Define User Needs With Help Desk Data

11/11/09

Needs analysis helps instructional designers identify performance problems to be solved by e-Learning. You can perform this vital first step with surveys and interviews. But there is another resource, the Customer Support database kept by most Help Desks. Here’s how to tap into the information available, and how one designer saved time and money while creating effective solutions.

Book Review: e-Learning 2.0: Proven Practices and Emerging Technologies to Achieve Real Results by Anita Rosen

6/22/09

Sometimes, if you are just starting out in e-Learning, or if you want to keep projects simple (because your world is already complicated enough), all you need is a little basic guidance. If that describes your situation, this is the book for you!