An evaluation of The Magic School Bus
Submitted by Huda Hashim Obaid
For the module EEL880
Date February (2003)
Introduction
Evaluation is the systematic assessment of the worth or merit of some object.
�The Scholastic�s the Magic school Bus� produced by Microsoft corporation (1999) is designed to learn the students in primary school. The type of program is Sencie, the students� ages� average of between 7 to 12.
  That types of program is instructional, the program is requires:
C-D capacity (521 MB).
Computer device.
Speaker.
Speaker Card.
Experienced teacher.
  C.D room.
  Multimedia Program.
That program was tested on (15) students, they are included girls and boys; in primary School, some of the students study in public schools and the others study in private schools. Chose that learns for evaluation software specific program.
These schools have not computers devices, for that reason the learners come to university and the teachers run the program for them, then the learner require of working in program.
The researchers setting number of instrument for evaluate that program, such as observation, questionnaire, and interview with learners and teachers. Also, collection the data and makes analysis that data.
Addition uses level (1) and level (2) of Kirkpatrick?s levels for evaluation that program. Through the research?s question, this type of program is aiding the teachers in performance learning process. Also it is help the students to understand the lessons more than traditional teaching, and it is promote the feedback positive.
It was found that program was a good at subject, animation, sound is clear and include various information.
That program is teaching the students to build the whole from the parts, and to dismantle the whole to it is parts. Additions the students are know more information of the animals, specialty, it is rare.
That program is bad at, it is includes much hyperlink, also some information is difficult for students who are under 10 years.
Background
The computer technology revolution had started at the middle of the twentieth century. Then progressed for wards rapidly with more facilities and less complications and troubles. This technology had become available over all the educational levels in the developed countries where there are strong economy that ensures the financial burden of such anew technology.
But unfortunately in developing countries such as Sudan, it is too early to be available over the various levels due to many troubles like illiteracy, poverty, and so on. Regarding the educational filed the pre- university level, computer is never used in the general schools both primary and secondary levels. But the majority of the private educational schools teach computer as apart of their curriculum, but unfortunately this policy is confined to how to use computer as new technology and not involve using the computer in teaching the various subjects in a way that facilities their explanation and consequently in creasing the understanding abilities of the students.
In which is concerned with the university level computer is widely used by the Doctors in the lectures, Researches presentations and also the students do.
The commonest fields in Sudan for computer use are the governmental and economical fields, and this was imposed due to the pressures of specialization and internationalization. For examples, in the governmental aspect, computer is need for connection of the various branches of banks among each other and for recording data.
Specialization encourages many international companies to invest in Sudan and such companies, which have wide economical activities and spread everywhere allover the world including developed countries transfer computer technology and it is appendices to this country.
This program is chose because it contains a good subject because children like animals. It is contain much and various information. Those students where chosen because they have some characteristics that qualify them to help the evaluating this program, so at to get a good result.
These Students includes girls and boys, they are learning in different primary schools. Some students are learning in general?s schools, and other studying in private schools. These students? ages average of ?between 7 to 12?.
Those students are learning in private schools, theirs own abilities or faculties more than those students are learning in general schools. Because, the private schools are owns more resource than generals? schools.
This program is claim a computer. Video. And students are own some what skills for run computer. Also, the researcher make notes about the programs for available the learners to understanding program.
Literature survey
What makes good educational software?
Hannafin & peck (1988,p 409) proposed a series of evaluation criteria based on their learning foundation for computer-aided instruction. They presented four categories of evaluation: instructional, cosmetic, program, and curriculum adequacy.
Cosmetic adequacy
The focus of cosmetic evaluation revolves around two areas: appearance clarity.
Appearance lessons that are unattractive or sloppy are likely to be rejected despite the potential of the basic design. Clarity in a cosmetic sense, Clarity refers to the capacity to display readily interpreted images- test, graphics, and sound?Etc. (Hannafin&Peck, 1988, p 320).
The programs are good cosmetic should be used, Color should enhance the impact and legibility of the screen. Also text layout, the amount of in formation should be limited to one Pointe plus supporting ideas, The font should be chosen for legibility, Where sound is used, It should be not be distraction from the main point of the screen (Text layout Olson &Wilson (1985). Addition use of hypertext to facilitate navigation and help the learner to develop understanding by providing useful lines. Screen should be include a well laid- out combination of text and graphics/ animation which work together to make a particular point clear (Screen Layout Olson & Wilson (1985).
Support that (Szabo & Poohkey, 1996), A learner is generally attending to something. Attention should be focused on the important information in a program rather than incidental information. Picture, Especially animated ones, capture attention more than test. Thus, graphic presentations should be designed for the more important information.
Recent finding suggest that picture can exert strong positive influences on learning given certain condition. Further evidence suggests that childerns? dependence on picture decreases with age. As childern grow  orlder they become better able to produce their own internal images (Pressley, 1977)
On the other hand, Information presented by video may be more memorable due to it is visual detail and it is emotional impact (Swan, 1996).
Technical adequacy
The evaluation must be concerned with basic program design, unlization procedures, and lesson efficiency. Evaluation criteria for program design are below the surface, that is, not readily observable.
However, Lesson design can be easily constructed or reconstructed simply by documenting the sequence of activities followed under repeated trials.
Verify the effects of providing unanticipated responses, of not following directions, and of otherwise acting in ways not prescribed in lesson directions. The failure of software designers too truly foolproof the lesson is alarming. Make certain that the basic lesson sequences remain intact under less-than-optimal student responses.
If the software is part of a set of related lessons, examine elements of protocol and procedural consistency a cross lessons. Much of the software presently available performs and tracks only simple reporting and is not sensitive to unplanned variation.
The efficiency of externally produced program is largely undetectable to most evaluators. some of the telltale signs that might be used to gauge efficiency are excessive overall execution time; delays in screen displays or computer responses to in put, unusual, frequent, or distracting disc access during lesson execution, and  the requirement to reenter information previously provided. (Hannafin & Peck, 1988, p320-322).
Reeves (1994) suggestion, the program a good technical should be, the mastery are entry levels available to set degree of difficulty of program. Also the users are able to exercise control over program, and should be the user was able kept the record along term. And to able the user is include the percentage correct, and help the user to make the records a accurate, then, keep the time?s of user was spent on items.
The best software uses techniques carefully and prudently. Again, the rule of consistency is important- use the same types of techniques to convey the same sorts of information throughout the software. The software should also use the same text format to show which word always have the user perform the same action (e.g., click a button or Keystroke) to trigger access to the glossary. (Rieber,  2000, P 209).
Curriculum adequacy
Moves than other levels, Evaluation of curriculum adequacy features provide aguage of the likelihood of effective installation of lessons in their intended settings.
The alignment of computer assisted instruction lessons with intact, or planned, Curriculum has become an increasing consideration for large- scale software designers, of particular concern are the thematic consistency with the overriding curriculum, the shelf- life, or useful longevity of the information presented, the availability of, and alignment with, related and the capacity to accommodate student and teacher styles.
User-supplied lesson parameters such as selection of example. Number of practice items, difficulty levels, And so on have become major factors influencing curriculum adequacy. (Hannifin & Peck, 1988,321-322).
Lee and Colleagues (1995) provide guidance in this area, as they have found three curricular components common to high schools that have successfully restructured their instructional program.
1.Acommon academic curriculum. Student achievement gains were found in schools with a common academic curriculum, where course offerings are narrow and academic content is strong.
2. High levels of academic press. This curriculum expectation centers on the notion that all students will meet high academic standards and devote considerable effort to academic endeavors.
3. Authentic instruction. Students are engaged in sustained, disciplined, and critical though a variety of instructional approaches, such as independent study, project- based learning, and real- world problem solving.
Pedagogical adequacy
Instructional adequacy can be considered under four major areas design, lesson procedures, efficiency, and clarity.
The lesson activities must be to follow. Student should not be forced to invest mental effort attempting to understand lesson direction and options. Well- designed lessons have the capability to adapt to varied learning rates by increasing or decreasing features such as the number of examples, practice items and so on.
The information presented- text, graphics, animation, and sound must be readily interpreted. (Hannafin &Peck, 1988,319-320 ).
The consider for good Instructional adequacy, It is contain the work not too easy ,and it is not too difficult. Also, it is not challenging the goals set at the start of the lesson.
A good Instructional, the student is can be make control a bout the program and  allowed the student was chose menus and movement. Reeves (1994).
Addition, It is give the student feedback positive and it is provide hints to guide user in problem solving. Also give the leaner correct answer after a number of incorrect responses. Reeves (1994).
Malone (1981) suggested three relevant factors: challenge, curiosity, and fantasy. In later work (Malone & Lepper, 1987) he added learner control.
Challenge, the most important principle is that the level of challenge should be individualized for and adjusted to the learner.
Curiosity, encourage the learner to seek new information that remedies the conflict.
Control, three rules are relevant to learner control: contingency, choice, and power. According to the contingency rule, what the lesson dose should be clearly a result of the learner?s action and responses. The choice rule encourages procedures, such as menus and global branching options that permit the learner to determine sequence or lesson parameters, such as difficulty. The notion of power is that lessons in which learners? actions have ?power effects? will be very motivating. Such lessons include environments in which the learner creates computer programs or uses computer tools, Such as graphics programs.
Fantasy situations encourage learners to imagine themselves in imaginary context or events using vivid realistic images.
The more a program includes these four elements, the more successful learning is because people enjoy it more.
What are the aspects to consider for good teaching of The Magic School Bus?
The program should be addressed the learners to      discover new information build on the previous information.
The teacher use questions way and discussion to      approach with student s in certain result.
The teachers should be use more than one example to      explain to students the lessons.
The teachers should be encouraging the students to      discover principles by themselves.
Evaluation methodology
Evaluation can be used to determine whether the training achieves it is objectives. Evaluation can also assess the value of the training identify improvement areas, and identify unnecessary training that can be eliminated. (Kirkpatrick, Kramer and salinger, 1994).
(Trochim, 1996), suggests an evaluation that occurs throughout the entire ?life cycle? of the program. Dynamic evaluation includes the interaction between the target group and the program.
This program is dynamic evaluation because, happing interaction between the groups students and the program. The objective of this evaluation is to examine, beyond the current situation, what can hap in the future?
Summative evaluation is a method of judging the worth of the activities; the focus is on the outcome (Bhola1990).
(Trochim, 1999) suggests summative evaluation examine the effects or out come of programs, determining the overall impact. 
The program is summative evaluation because, after finished of program the researcher collection the data and make analysis of outcome of program. Outcome evaluations investigate the demonstrable effect of program on specifically defined target outcomes.
Kirkpatrick (1959) and Kirkpatrick (1996), suggests that there are four levels at which training should be evaluated. Level ?1? (Reaction), Level ?2? (Learning), Level 3 (Behavior), and Level 4 (Result).
This program use level (1) and (2), for evaluation.
Level ?1?(Reaction), The method?sseuvaluation level 1 such as observation, interview, and questionnaire.
The researcher uses this level, because t he researcher is want to knowing how the student reaction to the program, also he is want to know of any type of program that the students like. Reaction is important to emphasize that it does not include is so easy to measure; and it is important to determine how people feel about the programs they attend. Also to evaluate a good job of measuring the reaction and feeling of people who participate.
Level ?2?(Learning), This level evaluation techniques are most reliable when pre-and post- evaluations are utilized. The methods use of level ?2? such as questionnaire, interview, and observation.
Use this level, because the learning can be measured by pre- and post test, either through performance tests. Also possible are applying classroom activities such as individual performance, of the skill being situation can be used as evaluation techniques.
Perspective
The research focuses on perspective teachers and students. Followed in this research qualitative approach, first, Qualitative research is multi method in focus; involving an interpretive, naturalistic approach to it us subject matter. This means the qualitative researchers study thing in their natural settings, attempting to make sense of or interpret phenomena in terms of the meanings people bring to them. (Denzin & Lincoln, 1994,p.2).
Researcher used this approach because, used small group in research, Researcher is part of learn process. The nature of the research question requires an exploration of what is happening in a particular instance, and need to present a detailed view of the topic (Creswell, 1998,pp.17-18). Add the qualitative approach in order to study individuals in their natural setting.
Target population
Al-Sahafa city, include many schools of various educational levels are spread .For doing my study I have sleeted group of students from the base school.
In order to do this selection I went first to the locality to take permission, then directed to wands the target-school, One for boy and the other of girls. The system of the pre university education in Sudan is not co-education. So, I have to go to two different schools one for males and other for females. First I have gone to the boys? school, met the head officer and taken permission and then gone to the class to select group of boys having an ages pan from 7 to12 years (Elsheed El taher boys). Then I went to girls? school (African conference for private study).
To avoid bias I have selects boys and girls of various levels regarding demographics, Socio economic status, and cultural back ground and scholastic achievement.
These in additions to the above two factors, which are, age and sex.
I have selected students of different demographics. They were of various Socio economic classes, Low, Moderate High classes. The cultural background was different from one to another because every one has his ethnic social and geographical origin that farms his cultural back ground.
Those who have well academically achievement and who are good in arts, sports and other scholastic activities. And also there are students who have moderate and low scholastic achievement and activities. This school have not computer, for that reason the learners come to university, for to work on the program, these students to study in second, third and fourth class.
Research matrix
Instrument
Questionnaire
Observation Interview
Question
Dose it improve students? attitude to science?
This program is a good cosmetic?
Dose program have clear language?
Does this type of program help the teacher?
Observation
Observation is basic to assessing human skills and behaviors. It is make control about students, before, and after they are working of program.
Use this is instrument because observed the work behavior of students before, during and after working. Carefully choosing observers, giving observers standard form to fill out, and trying to minimize the presence of the observer can minimize the impact of the observer. (Marrlli, 1993).
Performance should be measurable with observable results, based on the objective for the working program. (Kirkpatrck, Kramer&Salinger, 1994
Interview
Interview provides a means to collect in-depth information from participants who are reluctant to fill out questionnaires. (Marrlli, 1993).
Chose to use this instrument, because interview can be done in small groups. This type of group interview is often called a (focus group) and can used to collect in-depth qualitative information.
Before the focus group meet, methods for recording, reviewing and synthesizing information should be established. (Marrlli, 1993).
Questionnaire
Questionnaire can use only with respondents who are literate.
Questionnaire is a printed or computerized from using questions such as multiple-choice, ranking scale, rating or open ?ended.
Chose this is instrument because, make evaluation of student is a chives and performance.
Research method
Before run the program, the researcher introduces himself for student, his name, college and university, also define the name and type of program, and target for it is presentation.
. Fore then, the researcher explain for learners before will do when the program is running. Also the researcher give the learners are some what question about computer; such as dose use computer easily?, are they use any program before? And so on.
While, the students were working on the program, the researcher remark the students, observe the learner interacting with program, they are enjoyed for watching, and understand information of program, also some times the researcher examine with learner about of information in the program.
In the interim, the students are working, once in a while, they are working in group and they are make discuss.
Some of people help me for run my program, my teachers help me to run software and solving somewhat problem to face me, also my colleagues study with me the program, once in a while make control on the learners or direction the learners.
The learners are does difference method of evaluate the program, some learners are having difficulty understanding the language or the material, addition to, another learners was enjoyed when we are answer or make step is correct, and another was worked randomly or choose the part is like them.
During the learners working on the program, sometimes they are direction the question in the group and sometimes ask alone.
After the program finished, the learners fill in the questionnaire, after that, they are discuss with teacher a bout anew knowledge is use in the learn, and the goals are produced from use like this program.
At last the teacher discuss with the learners, how did program is improve or unimproved the performance students.
Validity and reliability
Lacity and Jansen (1994) define Validity as making common sense, and being persuasive and seeming right to the reader.
Reliability is the consistency of your measurement, or the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects.
Each question is measured with more than one instrument to ensure the validity, these are cluing question. The instruments questionnaire, test, interview and observation to ensure reliability and used of triangulation.  
Findings
In this section the findings will be discussed in the sequence in which they answer the research questions. The most important results will be presented first, followed by other results that support or contradict the findings
Dose it improve students? attitude to science?
Quetionniar Responses
Figure 1 shows the responses rate for the nine questions, on the questionnaire, 8o% of respond said Magic School Bus had helped them, and 70% found it too long. 69% would like to do it again, while 65%wanted to take it home, and 49 % wanted to work together, while 30% felt they needed their teacher there, and 13%found it too easy.
These result suggest that in general there is a positive attitude to the programmer and using it to teach? the Magic School Bus?,
From observation found that the students? performance is improved step after step, especially the students who the lowest abilities. Also (93%) from teachers are seeing these type of program is improve students? performance.
Teaching by this program to make the bond between the stimulus and the response, when the student find his answer is correct, the bond between the stimulus and the response is strengthened and reinforcement is positive, it the answer is incorrect, the bond is weakened, and reinforcement is negative.
the interview support the findings, they are yielded a good information. About (92.5%) from students and teachers are seeing  use software inteaching to aid establish learning goals.
It would seem then that the soft wear is construct the students? performance and promote peace the learning process.
This program is a good cosmetic?
The instruments that yielded the best information to answer this question were the observation, Questionnaire and interview.
Through the observation, it is founded that the students are react to the program, they are listen to the sound carefully and the program?s elements to aid the students to understand the information in the program.
Through the questionnaire, it is founded that (97%) t0 the students are given answers ensure the program is cosmetically good, through the interview with teachers; this instrument is support above instruments.
Both the observation and interview yielded the much information.
It would seem then that the program is contains many elements to promote the learning process.
Does the program have clear language?
Figure 2 shows the response rate for the three questions     on the questionnaire, 15% of respond said the language?s program too     difficult, and 35% found it suitable, while 50% found it too easy.
the observation  support that idea, because the students given different answers. the students who the lowest abilities, they are not understand the program?s language, the students who the moderate abilities, they are found difficulty to understand the program?s language, the students who the higher abilities, they are understand the program?s language easily.
The findings are backed up by the observation; questionnaire and test yielded the following information.
It would seem then that the program?s language is clear, although some students are found difficulty to understand it.
Does this type of program help the teachers in teaching process?
The instruments that yielded the best information to answer this question was the observation and the interview.
Through the observation, it is found that the teachers have enough time to directive the students. Also the teacher is able to show information when need.
Through the interview with teachers, (88%) from teachers are found that the program to aid the teachers in teaching process.
The findings are backed up by the observation and interview that yielded the following information.
Both the questionnaire and test support above the instruments, the percent?s the student to give support answer equal (95%)
It would seem then that both the teachers and the students are support use software in learning process.
Conclusions
When the above findings are considered the following conclusions can be made.
The type of program is improve students? Attitude to scince:
Putting together the following findings and the following literature, it can be conclude that the type of program is considering base to development the education process. The program can be expected to change student attitudes towards sciences. Addition this type of program increase the student?s learning skill.
The program include well curriculum and technical  available,it is aid the student is improving performance.
This type of program should be include good elements cosmetic:
Putting together the following findings and the following literature, it can be concluded that the program is good cosmetic. It is include well elements to aid student to study the subject, for example the sound and language are clear, the color is attractive, The animated cartoon aids the learner to understand the contain of the program, the animation is interesting throughout the program. The program include good elements cosmetic, the more successful learning is because people enjoy it more.
The program have English Lauguge sutiblity:
Putting together the following findings and the following literature, it can be concluded that, about (15%) of students are found difficult to understand the language?s program, sometimes it is contain some of ward is ambiguous, and (35%) of students are found it suitable and (50%) of students are found it too easy.
This type of program to more help the teachers in teaching process
. Putting together the following findings and the following literature, it can be concluded the program is contain amount of information available and unambiguous, also it is contain amount menus to aid the teacher make control on formation, add the teacher find enough time to take care the students.
Recommendations
Based the table below shows the recommendations that arise from the conclusions above.
From the table we can see that the most important conclusions relate to ?
Conclusion
Recommendation
If In use   few of hyper link.
The   type of program is improve students? Attitude to scince.
putting   the options available for aid the leaner to direction the program.
  In use   more example to aid the student to understand the lesson.
  In use the   sound and speech more clear.
The program is good cosmetic.
In use   amount of information reasonably .
In use   animation should be readily interpreted.
  The   language use in the program should be simplest, consider different levels of   students.
These programs have English language   suitable.
The   lesson should be including information unambiguous.
This type of program helps the   teachers in learning process.
   If In   use some hints to aid the teachers in problem solving.
From the above tale it can be seen that the most  important recommendations relate to the language?s program, improve thetechnical program and most look again the amount of information in the program.
References
Alessi &Trollip. (2001). Multimedia for Learning: methods and Development.3rd edition. United States of America.
Creswell, J.W (1998). Qualitative inquiry & research design: Choosing among five traditions. Thousand Oaks, CA: Sage Publications.
Denzin, N.K. & Lincoln, Y.S. (1994) Handbook of Qualitative research.
Thousand oaks, CA: sage publication.
Hannafin, M.J., & Peck, K. (1988). The design, development and evaluation of educational software. New York, NY: Macmillan.
http://hagar.up.ac.za/catts/learner/patriciam/evauat.htm
 http://www.und.ac.za/users/clarke/Kzb/ped.htm (1/30/2003)
http://www.und.ac.za/users/clarke/Kzb/individ.htm (1/30/2003)
http://www.und.ac.za/users/clarke/Kzb/record.htm (1/30/2003)
http://www.und.ac.za/users/clarke/Kzb/mal.htm (1/30/2003)
http://www.und.ac.za/users/clarke/Kzb/ques.htm (1/30/2003)
Kirkpatrick, D.L. (1996). Evaluating Training program. San Francisco: Author.
Kirkpatrick, D.L, Kramer, G., & Salinger, R. (1994). Essentials for Evaluation. Inc. Hodell, Instructional Systems Development. Alexandria Virginia: American Society for Training and Development.
Kirkpatrick, D.L. (1959, December). Techniques for Evaluating Training Programs- Part 2: Learning, Journal of the American Society of Training Directors.
Lacity, M.; & Jansen, M. A. (1994). Understanding qualitative data: A framework of text analysis methods. Journal of Management Information System.
Lioyd P. Rieber. (2000). Computer, Graphics, & Learning, the University of Georgia- Athens.
Marrelli, A. F. (1993). Ten Evaluation Instruments for Technical Training. Technical and Skills Training, July 1993.
Reeves, T.C. (1994). Systematic Evaluation Procedures for Interactive multimedia for Education and Training. In Reisman, S (Ed) Multimedia Computing: for the 21-st century. Harrisburg, PA: Idea Group.
Streubert, H. / & carpenter, D. (1995). Qualitative research in nursing.
Philadelphia: Lippincott, P.12.
Back
Hosted by www.Geocities.ws

1