The review of the various sources on evaluating online courses was exciting. Most sources offered me new approaches to evaluating online courses.
Each review points out concepts that were interesting to me and hopefully to you.
A. 10 Peer-Reviewed Articles/Scholarly Books:
1. Clothier, Paul. The Complete Computer Trainer. New York: The McGraw Hill, 1996.
Chapter 4 on evaluating computer training starts by showing the reader how to develop an Evaluation Sheet. The first step is the "KISS" Method of Keeping
It Simple, Stupid. The realistic author realizes that students are not usually eager to fill out evaluation forms. The author emphasizes clear and concise
language. The author also states that the instructor should avoid giving out course evaluations before the end of the training. Filling out the forms before
the course is over usually results in partially completed forms. Also the instructor should avoid collecting the forms individually, so that the evaluation
has the appearance of anonymous feedback. The instructor should make notes on any feedback mentioned consistently.
Compare pre-and post-tests to show the impact of training: Testing can be implemented using questionnaires, task-completion exercises, or testing software.
The instructor must keep in mind that questionnaires and surveys may only give a subjective view of the effect of the training. A learner may feel very confident
about what was covered in class and still be unable to perform back on the job. Software simulation "integrated" tests, where the learner is using the actual
software on the job, fare much better success. The problem is analyzing how well the instruction is. The system may give a poor score to someone
who might be competent, but didn't understand the instructions.
But if the goal of the training is to have the computer user be more productive at work, then there might be more problem in measurement. The more accurate
in gauging the effect of the training, the more time, money, and effort the organization must invest. These concepts are for small schools to consider.
2. Cunningham, Craig. Curriculum Webs. New York: University of Chicago, 2003.
Chapter 10 deals with Evaluating and Maintaining Curriculum web sites. The author offers a "Table of Questions to Consider While Evaluating a Curriculum Web," p. 162,
which are used by outside reviewers/subject matter experts in observing how learners of different abilities responses to the curriculum.
Pilot testing and later field testing of learners in different settings are critical.
Web Design Evaluation has a wide variety of criteria to evaluate:
Navigation scheme
Platform on Macs and PCs
Different screen sizes and windows
Color depth
Browsers
Images
Links
Forms of Java Script and Java applets/Flash images.
Evaluation plans can range from simple one-line statements to elaborate field-testing and revisions. The important factor is that there is feedback to
the developer.
Maintenance, revising and updating web sites are critical at least once a year. Updating includes seeing how the web site looks, checking current links,
and tracking visitor usage, including search engines exploring pages linked to site.
The most useful comment was publicizing web sites using the tag (page 167) of attribute KEYWORDS for search engines to pickup on, as well as the
importance of registering with search engines. Some search engine's spider searchers ignore these Mega tags unless they are included as HTML comments in
the head of the document with a Title tag .
This is obviously a must read book.
3. Driscoll, Margaret. Web-Based Training. San Francisco: Jossey-Bass/Pfeiffer, 2002.
Chapter 10 deals with implementing and evaluating WBT Programs. As the author noted, the analysis, design, and development phases often take longer than
anticipated; therefore, program managers may take shortcuts in the implementation and evaluation phases. The author offers a somewhat dry, but useful
checklist for the Evaluation Stage with review by:
a. Subject-Matter Experts for content as soon as the scripts and navigation maps are ready. If need be, paper-based screens or power Point Slides
may be used at this stage.
b. Rapid Prototype to identify issues related to interface, navigation and flow of instruction and client satisfaction. This should be conducted
on the first module to see if it works.
c. The Alpha-Class Test should determine if the instructor-led materials can be used. This is the study with the instructor as a member of the
development team.
d. Pilot Program. This study evaluates the complete program for educational effectiveness to correct tactical issues, such as bandwidth and
platform problems. This should be conducted as soon as the program and audience are ready. At least 85% of the learners should be able to complete the program
with less than 7% of the student calling the helpdesk for support. The product should reduce the number of failures in the field.
4. Fisher, Mercedes. Designing and Teaching on the Web, Oxford: Scarecrow Education, 2003.
I particularly enjoyed the opening remarks in Chapter 5, evaluating the Student: To begin with the end in mind means to start with a clear understanding of
the destination. The learners gain understanding when they construct their own knowledge and develop their own cognition maps of the interconnection
among facts and concepts. Per Wiggins and McTighe, the backwards design method (p. 198) helps the learners keep a clear picture in their minds of what the goals
for the course will be to construct their learning. Students apparently are more successful at tasks and report more interest if they receive not only
a final grade but also incremental deadlines and feedback narratives. Furthermore, on-going assessments need to be based on real-world examples that are
authentic, challenging, diverse, and enjoyable. Students are allowed to use complex reasoning and communication skills in various forms of assessment:
a. Pre-assessment survey to show the instructor is aware of their needs and issues.
b. Formative or on-going assessments or unit module quizzes, projects, and small-group activities with feedback in the form of student-discussions
teacher-student observations, portfolios, group projects (community shared practice) and homework.
c. Summative assessment. A formal assessment must first have a rubric or checklist of standards so that the students can perform better knowing
their goals upfront.
The author shows extensive examples of e-mail feedback to students; group and individual responses to help motivate and reassure students. Further in the
chapter, the author gives extensive examples of designing a rubric of standards. The major Pepperdine Model used has three sections:
a. Construct: where students can show effort, creativity, and persistence.
b. Reflect: where students reflect on their resources and demonstrate an understanding of their new knowledge in online discussions and projects.
c. Exhibit: where the students have a chance to demonstrate what they have learned online during the last face-to-face meeting.
The author's Course Evaluation Form was also immediately useful for the reader to implement.
5. Graham, Charles, et al. Seven Principles of Effective Teaching - A Practical Lens for evaluating Online Courses. Technology Source, March 2001.
The author offers the following seven points in evaluating web sites:
Principle 1. Good Practice Encourages Student-Faculty Contact by establishing policies describing the types of communication over different channels.
Examples include technology questions vs. content questions and average response time to a student's e-mail.
Principle 2: Good Practice Encourages Cooperation Among Students. To avoid discussions without focus, CRLT required:
Learners should be required to participate.
Discussion groups should remain small.
Discussions should be focused on a task.
Tasks should result in a product by the group discussion.
Learners should receive feedback on their discussions.
Instructors should post expectations for discussions.
Principle 3. Good Practice Encourages Active Learning.
Students learn a lot by seeing and discussing their peers' work. Sometimes instructors forge this principle.
Principle 4. Good Practice gives Prompt Feedback.
As the semester progresses, often instructors give less feedback and often to the class as a whole instead of to each individual student. Instructor
must be careful in maintaining energy in the second half of the course.
Principle 5. Good Practice Emphasizes Time on Task.
Online courses need deadlines. Regularly distributed deadlines encourage student to spend time on tasks and help students with busy schedules avoid
procrastination.
Principle 6. Good practice Communicates High Expectations.
The keys are challenging tasks, sample cases, and praise for quality work, which communicate high expectations.
I guess this principle is evolving more in online courses.
In addition, examples of student work from previous semesters are good models for current students in meeting course expectations.
Principle 7. Good practice Respects Diverse Talents and Ways of learning. Students can shape their own coursework by choosing project topics according
to a set of guidelines.
The Seven Principles are worth reading.
6. Horton, Sarah. Web Teaching Guide. New Haven: Yale University Press, 2000.
In Chapter 5, Site Assessment, there are two site assessment factors: its usability and its effectiveness as a teaching tool.
With page request statistics, online instructor can find out if the students are reading the required material. If multiple hits correlate with incorrect
answers on a quiz question, the instructor should rethink the teaching approach.
Various techniques include:
(1) Tracking software provides various degrees of detail of how the site is used.
(2) Sever logs to provide how often students are using the site.
(3) Feedback on which parts of the site are useful or which content is lacking. Same questions are listed on page 209.
Formal assessment can also be followed by Do-It-Yourself Assessment of each part of the site.
(1) Testing Environment, where the site can be tried out before going online.
(2) Defining Objectives by using a list of testing objective questions on page 212.
(3) Developing a Test Plan with known participants, test monitor, materials to be tested, procedures such as thinking aloud in noting their
thought processes in using the site.
(4) Video taping initial usages of the site.
(5) Developing Questionnaires.
(6) Debriefing participants.
(7) Administering the site and summarizing the data collected about the test session.
(9) Refining and expanding the site from the test results.
All of the above, various techniques, and Do-It-Yourself techniques, are useful to the instructor/web course designer.
7. Horton, William. Designing Web-Based Training. New York: Wiley, 2000.
Before designing Web-Based Training (WBT) technologies, one should weigh the potential/requirements of WBT and cost estimates.
The author breaks down the parts of WBT, including:
A. Ideal learner
B. Ideal course
C. Advantages of WBT:
1. Savings of 30-80% in training costs (travel, facilities, supplies, administrative costs, and salaries, lost work time).
2. Improved learning (activating learners, exposing real-world data, and providing more in-depth learning, better thinking skills and
responding skills, collaborative skills, and knowledge resources other than the local library).
3. Equal Effectiveness as classroom training: There appears to be no significant differences between Distance Learning (DL) and classroom
training and sometimes DL does better in student satisfaction, best instruction world-wide, possibility of just-in-time instruction, with easy approachable
instructors, adaptable to the learning style (visual/verbal, analytical /experimental, morning/evening, sprinters/ plodders, extroverts/introverts),
with immediate feedback.
4. Savings for instructors (teach from anywhere, with less travel, less administrative time for printing, collecting and returning graded
assignments).
5. Advantages for organizations: delivering high-quality training around the globe training and recruiting workers as students.
D. Disadvantages of WBT:
1. In addition to the typical instructor effort for course conversion, more student effort is required in discussions, problem solving,
using technology, and getting unstuck through e-mails and phone calls.
2. Using interactive lectures and questions is also important.
E. The author also notes on-going concerns:
1. WBT distorts learning, such as possible frequent learning disruptions.
2. Traditional training is viewed as a reward (travel, respite from daily work monotony, qualification of workers for a new job).
This author gives the online instructor/web course designer points to ponder.
8. Ko, Susan. Teaching Online: A Practical Guide. New York: Houghton Mifflin Company, 2001.
In the subsection on assessment, student assessment methods include the typical list of multiple choice, short answer, and true/false exams.
However, the emphasis is on the instructor using multiple forms of assessment to better test material.
Various software, for example:
a. allow a student to use only one IP address to access a test, restricting access ports to a test.
b. search for similarity among students' test answers to identify possible cheating.
c. point out to the student incorrect answers and automatically, which sends instruction for possible remediation, include text page numbers.
d. tell the instructor if the student has entered a section of the site to read it and how long the student has been in the section.
This might correlate with whether the student has spent at least minimal time in reading the material effectively to answer questions.
I recommend this book section for instructors starting to research course evaluation techniques.
9. Newby, Timothy et al. Instructional Technology for Teaching and Learning, 2nd Ed. Upper Saddle River, NJ: Merrill, 2000.
The Evaluation section deals with the purposes of evaluating student learning and the effectiveness of instruction, techniques for evaluating
students and instruction, use of developmental guidelines to construct, and assessing evaluation instruments.
Evaluation identifies areas that are unclear, not helpful, or have the highest priority for revision.
The author discusses the advantages/disadvantages and examples of obvious techniques of evaluating students, such as True/False, Matching,
Completion/Short Answer/Multiple Choice, Essay, Performance, Portfolios, Interviews/Oral Evaluations, Logs/Journals, Open-Ended Experiences,
Long-Term Projects.
The author also notes that instruction can also be evaluated with Student Tests, Student Tryouts, and Direct Observations, Talking with Students,
Peer Review, and Teacher Pre/Post Review.
Most importantly, the author provides guidelines for each technique to those instructors new to online teaching.
10. Owston, Ronald. The World Wide Web: A technology to Enhance Teaching and learning? Educational Researcher, vol. 26, Nr. 2, March 1997, pp. 27-33.
Also at Owston, retrieved on February 14, 2004.
The author presents exciting ideas about enhancing Online Teaching and Learning. Concerning barriers to Online Access, the most critical problem is the
loss of student discussions because of technological problems. If the university's infrastructure, including computer systems (telephone lines,
capacity servers, and bandwidth), is not adequate, then students may feel less inclined to contribute further. The result is a downward
spiral of discussions intensity. Although no medium in itself improves learning in a significant way, online discussion threads are preserved for future examination,
elaboration, and extension. The result can be richer,
more thought provoking discussions.
The Web appears to be students' current learning model. Through the University of Texas's registry of college and university course home pages,
many course home pages are shown to contain interactive learning materials, lecture notes, assignments, exams, and links to course-related resources.
The UT's World Lecture Hall registry is one window to illustrate exciting learning online.
Websites provide flexible learning. Although face-to-face contact may lessen, student readings and virtual conversations are available on the net for
review by instructor and students. Instructors are no longer "deliverers of instruction," but "creators of learning experiences." Course discussions
(student-instructor, student-student) are more thoughtful and, therefore, perhaps superior to classroom instruction. This is an excellent example for web designers
to consider.
Web enables new kinds of learning. The author provides an eye opener: with an authentic audience, students develop better written communication skills.
They become more conscious of their vocabulary, syntax, and grammar. Students develop their skills beyond hastily, informally written work. Students
are called on for critical thinking, problem solving, written communication, and collaborative work. In other words, students grow in their web communication abilities.
Can the Web Help Contain Costs of Education? The author offers the importance of hidden costs of web design. Faculty develop their own course material
in a continuous "work-in-progress." School web resources may be available at no charge, such as museum art work and library references, which would
otherwise be well out of reach of all but the most affluent school libraries.
Finding the University of Texas database site is worth the reading
B. Links to Other Resources (non-journals and books).
1. Distance Education at a Glance, Guide #3. Instructional Development for Distance Education,
Guide 3, retrieved on February 14, 2004.
In the Evaluation Stage, the author recommends that students be given pre-addressed and stamped postcards to complete and mail after each session. These
"mini-evaluations" focus on course strengths and weaknesses, technical or delivery concerns, and content areas in need of further coverage. The evaluations
could be quantitative relying on statistically quantities of data or qualitative in the forms of interviews and observations to query small number of
respondents. The best approach seems to combine both quantitative and qualitative measurements.
From the Evaluation Stage, the Revision Stage must be field-tested prior to future course use. The pilot changes could be tested on small groups of
distance learners, content specialists, and colleagues.
This author offers another viewpoint for the instructor's review.
2. Evaluation for Distance Educators. Distance Education at a Glance, Guide #4. Distance Education: An Overview,
Guide 4, retrieved on February 14, 2004.
Informally, the online instructor should collect data to determine:
Student comfort with method used to deliver the distance instruction.
Appropriateness of assignments.
Clarity of course content.
If class time is well spent.
Teaching effectiveness.
How a course can be improved.
In the Course Summative Data collected, the instructor may also want to ask students to:
List five weaknesses of the course.
List three - five strengths of the course.
If teaching the course, what would the student do differently?
What would the student recommend to a friend planning to take this course?
What should be covered in the course, but was not?
Would the student recommend this course to a friend? Why or why not?
Qualitative Evaluation Questions are less affected by typical small online class size and allow the student output of topics. Questions should be
sequenced by asking for improvements before asking for what is good. Open-ended questions should be placed after quick answer questions, so that
students can have built-in thinking time.
Quantitative Evaluation Questions, on the other hand, usually result in less than a 50% return rate with very positively or negatively reactions to
the course. Forced survey offers respondents a limited number of possible responses. Fresh insights and unique perspectives often go unreported.
Statistical analysis often results in an illusion of precision that may not be real.
The solution for me is to use both types of questions, Qualitative and Quantitative, in online surveys.
3. Fitzelle, Jr., George et al. (1996) Survey Evaluation of Web Site Instructional Technology; Does it increase Student Learning?
Fitzelle, retrieved on February 14, 2004.
In the literature research on Computer Assisted Training (CAT), it was found that CAT improves student achievement on an average of 20%. In addition,
simulation, interactive video instruction, hypertext programs, bulletin boards, and networks were also found to be effective in enhancing learning.
In this 20-item survey of 68 students enrolled in a research methods class, the students used various survey techniques, including Likert scaling,
ranking, and qualitative question. Evaluation Findings show that students thought that the web site in question significantly enhanced their
learning of course content. However, the students did not find the bulletin board useful.
This website might be another good model for online survey development.
4. Henderson, Tom. Classroom Assessment Techniques in Asynchronous Learning Networks. Assessment, September/October 2001,
Henderson, retrieved February 14, 2004.
Classroom assessment is important feedback to the instructors to implement continuously quality improvement techniques in the classroom.
The author noted that discussion questions only generated few comments. Through the Classroom Assessment Techniques (CATs), the author asked
two questions:
1. What is the one thing that helped you learn the most in this week's activities?
2. What is the one thing in this course that is least helpful to your learning?
The results of the CATs show that the biggest student concerns were that other class members could see their work on the threaded discussion list
before they were ready for it to be seen. From the results of the survey, the author required that only three projects would require group work,
the remainder of the coursework cold be completed independently. Students were still required to post answers to weekly questions and comment on
three peer postings weekly on the threaded discussion list, but not as a group. The CAT helps the author recognize his misperceptions and move the
focus of the course from teaching-centered to learning-centered. The CAT seems to be an accurate means of providing feedback to the instructor about
the author's theories on how the class is running.
CATs are another course assessment tool worth reviewing.
5. Key Elements of an Online Program, Evolving Technology Assisted Education Bring You Tomorrow, Today, Illinois Online Network.
Key Elements, retrieved on February 14, 2004.
In a collaborative learning environment in the Virtual Classroom, highly dependent on written dialog and high synergy, students shape their learning
objectives by contributing information related to the course content, but directly applicable to their own academic and professional goals. Mature,
open-minded, self-motivated, collaborative, trusting students have good written communication skills and a minimum level of technological experience
with time commitment.
The author offers exciting concepts in Online instruction. Online the visual barriers are largely eliminated. Participants are more willing to risk
more in written participation rather than spoken. Therefore, lectures should be reduced and compensated with open-ended remarks eliciting comments.
Online curriculum focuses on the application of knowledge to the real world and fosters critical thinking skills. The online process integrates life,
work, and educational experiences, draws a connection between learned concepts and work experience, utilizing a minimal amount of memorization. The
Virtual Classroom must be open, honest, sincere, and conductive to learning.
The author also offers his viewpoints about the role of the Online Instructor/Facilitator. The training facilitator is also the designer of the
curriculum and monitor of the online course. However, the facilitator is no longer the distributor of information as in the teacher-centered classroom.
The facilitator is now the medium of learning in the student-centered environment. No longer the "sage on the stage," the online instructor is the
"guide on the side" by helping others to discover and synthesize the learning material. Students in a 24/7 classroom demand relevance and may
require extra support due to their busy lives. The leadership model is shifting away from the authoritarian style towards a more democratic style.
Therefore, presentation sequencing and pacing of content is important to avoid content overload.
This is also a must read article for online web curse design hints.
6. Learning Styles and the Online Environment. Evolving Technology Assisted Education Bringing You Tomorrow, Today.
Learning Styles, retrieved on February 14, 20044.
Because learners have different learning styles or a combination of learning styles, the author provides examples of activities matching various modes
of learning. Examples include:
a. Visual/Verbal. These Students prefer to read written forms, visual aids (Power Point and Blackboard), textbooks, and class notes. They
prefer quiet rooms. They visualize information in their "minds eye."
b. Visual/Nonverbal. These students use graphics or diagrams. They benefit from instruction in film, video, maps, and diagrams. They, therefore,
visualize information in images. They also prefer to work alone.
c. Audio/Verbal. These students prefer to listen. They mentally hear the way the information is explained. Group participation in a
listening/speaking format is important.
d. Tactile/Kinesthetic. These students prefer physical hands-on experiences. They do better in hands-on activity by physically being active
in the learning situation. They benefit from in-class demonstrations. Online simulations with 3-dimensional graphics can replicate the physical work.
Lab session can be discussed online. Outside field work can be incorporated well in discussions.
These practice activities examples are useful to the online instructor/web course designer.
7. O'Quinn, Lisa. Factors that Deter Faculty from Participating in Distance Education.
Factors, retrieved on February 14, 2004.
Community college faculty and division chairs across five campuses were asked to rate their web programs. Faculty with combined delivery courses had
heavier teaching loads than those colleagues who only taught classroom courses or only distance courses. Higher percentages of faculty who teach distance
courses had received distance training than faculty who taught classroom courses.
Combination - Delivery faculty, like Distance - Only counterparts, did not agree that any factors inhibit them from participating in distance education.
The factors for both types of instructors included:
Lack of money support for distance learning design.
Concern for faculty workload
Lack of salary increase
Lack of administrative, technological, or secretarial support
Concern about the quality of students who enroll in distance courses.
Lack of distance education training provided by the college
Lack of release time for website development and maintenance.
The deterring factors are worth reviewing.
8. Wilson, Carol (1998). Concerns of Instructors Delivering Distance Learning via the WWW. Journal of Distance Learning Administration Contents, Vol 1,
Nr. 3, Fall, 1998, at Wilson, retrieved on February 14, 2004.
At the time of the survey, instructors were more concerned about their fears of non support than about the potential of using the World Wide web for
conducting courses. The fears included:
Insufficient - Instructor time to develop and maintain
course materials
- Technical support
- Administrative support
- Time to interact with students online
- Technical training
- Student familiarity with computers
- Equipment reliability
- Guarantee of intellectual property rights.
These factors should be taken into consideration in designing and evaluating websites. For example, only 58% of the schools gave compensation time for
web designing, and only 38% of the instructors were given time to maintain websites.
Instructors for the most part had problems accessing technical support or getting web design training. The author also noted that administrative support
was not consistent throughout the chain of command for many instructors. As noted, administrative support has many unanswered questions.
Finally, unreliable computer systems adversely affect student satisfaction with online courses.
9. Wright, Clayton. Criteria for Evaluating the Quality of Online Courses. Instructional Media & Design.
Criteria, retrieved February 14, 2004.
The author discusses several criteria for evaluating the Quality of Online Courses:
Layout criteria include guidelines of using bold face type sparingly and underlining only for hyperlinks, not for emphasis.
Goals and Objectives are related to knowledge, skills, competencies, behaviors, and attitudes with a discernable sequence of expectations.
Course Content criteria include constructive, relevant, and frequent feedback to promote clarification.
Learning Resources Multimedia clips (audio and video) should be included only if the learner has access to appropriate hardware/software.
Evaluative activities must be feasible, relevant, accurate, and congruent with the objectives. The relationship between course learning outcomes, evaluation
strategies, and course assignments must be evident to the learner. Learners are able to track and evaluate their own progress.
The author criteria review is worth reading for designing websites.
10. Elements of Instruction. Evolving Technology Assisted Education Bringing You Tomorrow, Today. Illinois Online Network at
Elements of Instruction, retrieved on February 23, 2004.
The author offers a cookbook approach of important points for the online instructor to produce purposeful interaction to increase a learner's knowledge or
skills in a specific, pre-determined fashion. Adapted from Gagne's nine universal steps of instruction for design of learning materials, feedback is
critical for students. As all web course designers know, instructors give feedback to students individually and in groups and students view each other's
work online. Also online quizzes give students immediate feedback. But the question remains: how can feedback strategies be integrated into the instructional
design?
Since the online student needs more feedback than the student in the traditional course because of the threat of alienation in the Virtual Classroom,
strategies are important to incorporate feedback:
1. Clarify communicate how the participants will be grades. This information should be provided on the firs t day of class.
2. Arrange telephone office hours including evening hours.
3. Use a variety of delivery systems for feedback, including one-on-one and conference e calls, ax, electronic mail, video and computer conferencing,
as well as traditional office hors.
4. Note which students do not participate during the first sessions and contact them individually after class to see what the problems are.
5. Return graded assignments privately within 48 hours.
6. Feedback to the student adds insight to the student and affords the instructor an opportunity to reaffirm key concepts and module closure.
7. Students should never be more than 7days away form current grade status. This serves to keep the student on track and noticed, even with an opportunity
to resubmit any misdirected assignments.
8. Respond privately to personal messages within 24-36 hours.
9. Thank students public ally for insightful comments submitted to the classroom. This gives positive reinforcement to the student and models types of
responses from other participants.
10. Make interpretive as well as descriptive comments to model the time of critical thinking the instructor expects.
11. Integrate theory with observations and applications.
12. Participate in conferences by providing extensive critique, feedback, and encouragement to help students to become more involved.
13. Let conversations develop and give students a chance to participate before jumping in with in depth comments/feedback or analysis.
14. Use student feedback to improve the course in content, relevancy, pace, delivery problems. Guided change, not just change for change sake, is
important for course continued developed and evolution.
Each recommended strategies adds to the online instructor's web design repertoire.
_______________________________________________________________________________________________________________________________________________________________________
Return to Jim Rorke's Profile Web Page.
Please address questions and comments to: [email protected].
Last updated on February 24, 2004.