Online Enterprise Surveys: Construction and Management

Many educators now use enterprise server software systems, such as Blackboard, to provide alternate educational forums for supplementing or replacing traditional classroom lectures. These enterprise course management systems not only provide a site for student learning and testing, they also provide an opportunity for educators to conduct surveys.

Surveys incorporated into e-Learning can gather information about a course from students, and the opportunities go far beyond these in-course surveys. Enterprise course management systems can be used to create course sites, to provide information, and to gather evaluative data. Online survey delivery is superior to traditional methods for tracking respondents and participation, as well as for delivering immediately analyzed results. No HTML knowledge is necessary since items can be typed directly into the assessment item template. Respondents create their own data as they answer survey questions, thus providing inexpensive, summarized results. The purpose is not only to create surveys but to do it in a Web environment that is cost-effective, accurate, user-friendly and convenient for respondents. This article explains the step-by-step use of enterprise course management systems for surveying.

Background

Education is rapidly changing from traditional classroom instruction to more flexible methods of teaching and learning, including videoconferencing and Web-based learning in higher education. The use of multiple methods in education has been termed “blended” learning and a recent study by Josh Bersin (2003) profiled 15 corporate applications with population sizes ranging from 250 to over 25,000 learners. Blended learning often involves Web-based learning via enterprise course management systems that provide a course site for materials that can include the syllabus, course schedule, PowerPoint® presentations, online quizzes, and online discussions. Faculty upload course materials and students can use them at their convenience. Thus, education achieves a 24- hour, 7-days-a-week opportunity for students to access educational materials.

Blackboard, Inc. boasts that more than 2,000 institutions now run Blackboard to bring education online. Despite the broadening use of the technology, Frank Greenagel, writing in the September 2003 issue of Learning Circuits, warns that education is far more than simply providing electronic page turners. His point reinforces the need to provide a means of gathering feedback from the learners if the education is to be effective. The feedback provided via online surveys can be used to evaluate learning styles and to avoid the failures experienced when the information doesn’t match the needs of learners.

Thus, as the methods of instruction change, so too must the methods of evaluation. With little or no classroom time available for surveys, alternate evaluation strategies are necessary. As online education expands, more online evaluation strategies are needed.

While these points might seem logical, educators have been slow to act on them. In 1999 three nursing faculty members, (Marilyn Ryan, Kay Hodson, and Nagia Ali) noted a gap in the literature on systematic evaluation of World Wide Web educational delivery. Nonetheless they used classroom administration of a survey to evaluate course delivery via the Web. In the same year, The Review of Contemporary Research on the Effectiveness of Distance Education in Higher Education, sponsored by the American Federation of Teachers and the National Education Association, reiterated the gap in evaluation of distance education. In 2000, Andrew Mills provided detailed instruction on creating Web-based multimedia and interactive courses for distance learning but included only faculty evaluation of student learning, and no discussion of student course evaluations. A lengthy guide to Learning on the Web, published online in 2002 by TeleEducation in Canada, details numerous steps in creating online learning courses, but simply alludes to the use of online evaluation with a checklist entry stating, “A student course evaluation form and a general student feedback page to take comments while the course is in progress.” No information is presented on how to create or use evaluation online.

This article explains the online evaluation strategies utilized for both course and program surveys within the College of Nursing at Texas Woman’s University, a public university with approximately 10,000 students. An online survey has been used for course evaluations at the undergraduate and graduate levels. Additionally, the College of Nursing Program Evaluation Committee uses online surveys to gather program evaluation information from all undergraduate and graduate students in their last semester.

Enterprise course management systems not only provide a site for distance learning, but also a site for online surveying. The surveys can be easily developed and managed without HTML skills. Course evaluations were the first online surveys used in my higher education nursing courses, and now online surveys are used to gather information about the program.

Survey creation, use, and analysis

The survey will need a home and the enterprise course management system provides this within a course shell. An information technology system (ITS) manager creates a shell designated to the course number and title. Faculty members are given access to the control panel enabling them to upload documents and PowerPoint® presentations, and to create assessment tools, including surveys. Course shells do not need to be limited to educational courses, but can also be used as communication vehicles with names such as “Undergraduate nursing students,” or “Graduating students.”

Next, all enrolled students’ IDs are uploaded into the course shell thus providing student access to all posted materials. The student access level does not permit them to access the control panel, thus they cannot change any posted materials on the course site. They can, however, print or save all documents onto their own computer. Using the university server address, the ITS manager also creates email addresses for all students but students can change their address to their preferred mail server, often their home email address. Course faculty can send mail to all the students in the course, or just to individual students. This communication system can be used to tell students when a new survey is posted and to remind individuals who have not yet completed surveys to do so.

Creating the survey

Creating the survey on the enterprise course management system is easy. Sidebar 1 graphically displays the steps in Blackboard format. First, simply select and click on the Assessment Manager. Second, choose the “Add Survey” button. Third, give the survey a name, such as “Course Evaluation Survey.” In the description box, type a brief note of explanation and a statement meant to encourage participation, then click “submit.”

Fourth, type the instructions and explain to learners that a bullet will appear by each name in the grade book as the survey is completed and that responses will not be read until all students have completed the survey. That assures anonymity.

Fifth, select the question type, such as multiple choice, true-false, or open-ended. Then, click “Submit.” Sixth, type the stem of each item and the choices. If the survey uses Likert responses ranging from strongly agree to strongly disagree, each choice is typed within the multiple choice answers. The assessment manager generally uses four multiple choice answer options, but this can be changed to offer more options. Next, click on “Add Item” and repeat the steps from selecting the question type to typing the item. Unlike tests, no “Correct Response” is indicated on surveys.

Step 7 shows that once all items have been entered, you can click on “Preview” to see the entire survey. Read each item carefully to be sure it is clear and error free. The preview pane allows item modification so you can correct any mistakes. After editing, click on the “Save and Make Available” option, which is the eighth step. Pay close attention to where you save the survey, such as under course documents.

Survey use

Prompts allow you to determine a start and stop date for making the survey available. An announcement can also be generated so students accessing the course will see that a new survey has been posted when they access the course and view the initial course announcement page. The announcement creates a direct link to the survey to make it easier for students to click on and begin the survey. You should also generate an email asking students to complete the survey.

Another attractive feature of online surveys is that students can complete them at their leisure. Faculty can determine the desired time frame when they want students to complete the survey by setting the start and stop dates for the survey availability as mentioned earlier. Students decide, within the time frame of the survey’s availability, when they feel like doing it.

Tracking completion of the survey is also easy. Simply review the gradebook spreadsheet and bullets appear next to those students’ names who have participated. To access the grade spreadsheet, go into the online course control panel and click on the Online Gradebook. When the next screen appears, click on the spreadsheet view. Depending on the number of students and grade book items, it will take some time for the spreadsheet to load as indicated by the loading bar at the bottom of the screen. Once loaded, names appear in rows and check marks appear in columns under the survey title for individuals who have completed the survey. The appearance of a question mark indicates the student previewed but did not complete the survey. A lock appears if the student was disconnected while taking the survey. Both question marks and locks can be cleared so the student can try again and complete the survey. A dash indicates students haven’t taken the survey and an email can be sent as a reminder.

Analyzing results

Once all students have completed the survey, or the response rate is adequate, the results are just a few clicks away. Response rates can be calculated by dividing the number of respondents by the total group size and are generally expressed as a percentage. With this online survey system there is no time consuming data entry necessary. Sidebar 2 shows the step-by-step method of obtaining results in the Blackboard format. First, within the course control panel, click on the “Online Gradebook.” Second, click on the “Report By Item.” Third, click on “List All” twice, and the names of all tests and surveys will appear. At step four, click on “Grades” to the far right of the survey title. Step 5 shows that the results begin with item information and since, on a survey, there are no correct answers these initial results are reported as NA (not available). Click on “Detailed Analysis” and the next window will provide frequency data by item and response. Step 6 shows that the initial item was dichotomous and 100% of the respondents were from Dallas. For a Likert-type item with 5 choices ranging from strongly agree, through agree, neutral, disagree, to strongly disagree, the numbers would show the percentage of students selecting each response. Survey responses can be printed using the standard print command under the “File” toolbar. Detailed analyses, and the narrative responses, can also be printed in the same manner. However, they are not printed along with the survey responses; they can only be printed by clicking on the detailed analysis button for each item and then printing. No identifying information is available with responses, so participants are, and remain, anonymous.

Survey reproduction

Once created in a course, survey forms can be copied and pasted into other courses by the ITS manager. This method of uploading surveys saves time and provides consistency in format, content, and directions. The survey can still be modified as needed, but the entire survey does not need to be typed into each course. Consistent survey content is helpful because it allows comparison across courses and programs — thus, surveys delivered by enterprise course management systems facilitate evaluation within courses and programs.

Shortcomings of online surveys

Surveys conducted online using a course management system do have a few drawbacks. Without additional work, this methodology does not allow for statistical analysis beyond the reported descriptive statistics of item analysis. Reliability, non-parametric, and parametric statistical analyses are not feasible because the data are aggregated rather than being individual scores. Percents are reported as whole numbers without fractions, so using the sample size and percentages to calculate scores is time consuming and has error. Blackboard version 6 used at my university doesn’t allow data to be saved as a file and transported into a statistical program such as Excel or Statistical Programs for the Social Sciences (SPSS). The addition of data-saving and the ability to transfer to a statistical package would enhance data analysis capabilities.

A second shortcoming of online surveys is that they appear to be longer online than they do when printed on paper. The time involved in completion is generally longer too. One survey used in the past to gather demographic data on all enrolled nursing students, along with their satisfaction and concerns regarding the nursing program, involved 100 items. When printed on paper with a font size of 10, the survey of 100 items took three printed legal-size pages and generally took students 12-15 minutes. Online, the survey gives the impression of being longer and took students 20-25 minutes. Loss of online connections results in an incomplete survey attempt that must be cleared before another attempt is possible, and data are lost when the first attempt is cleared. Students who had experienced failed attempts were less likely to complete the survey later.

Summary

Educators are trained to evaluate. Accrediting agencies expect documentation to support both evaluation and modifications based upon evaluation. Surveying using an enterprise course management system is superior to traditional methods in tracking respondents, facilitating participation, and delivering immediately analyzed results. The method is inexpensive and requires no data entry after respondents complete the online surveys. The availability of enterprise management systems for education provides many opportunities for data gathering via online surveys. The reviewed steps for survey creation and management are fast and easy.

References

Bersin, Josh. What works in blended learning. Learning Circuits, July 28, 2003. [online]. Retrieved 9/25/03. http://www.learningcircuits.org/2003/jul2003/bersin.htm

Blackboard Inc. Blackboard Course & Portal Solutions. [online]. retrieved 9/25/03. http://products.blackboard.com/cp/index.cgi

Greenagel, Frank L. Lead balloons, stone canoes, and learning styles. Learning Circuits, September 15, 2003. [online]. Retrieved 9/25/03. http://www.learningcircuits. org/2003/sep2003/greenagel.htm.

Institute for Higher Education Policy. What’s the difference? A review of contemporary research on the effectiveness of distance learning in higher education. [online]. April 1999. http://www.ihep.com/Home.php

Mills, Andrew C. 2000. Creating webbased, multimedia and interactive course for distance learning. Computers in Nursing, 18 (3), 125-131.

Ryan, Marilyn E., Carlton, Kay H. & Ali, Nagia S. 1999. Evaluation of traditional classroom teaching methods versus course delivery via the World Wide Web. Journal of Nursing Education, 18 (6), 272-277.

TeleEducation NB. Learning on the web. 2002 [online]. Retrieved 10/6/03. http://teleeducation.nb.ca/content/pdf/english/ lotw2002.pdf

Comments

Login or subscribe to comment