Running head: EVALUATION OF MATH SKILLS SOFTWARE PACKAGE

Evaluation of a Math Skills Software Package based on the

Principles of Precision Instruction

Edrian M. Sani

Jacksonville State University

Abstract

The present study evaluated the performance and characteristics of a computer-based software package for Precision Instruction. If adopted, it would replace the software package in use at the Academic Center for Excellence at Jacksonville State University. The current software does not meet the organization’s needs. The intent of this research was to measure effectiveness of the software in increasing student fluency. Fluency is the rate of performance that makes the skill not only useful in everyday life, but also causes it to be remembered even after a significant period of no practice.

One hundred eight college students from Academic Success Skills classes served as subjects in the initial pool. The subjects were divided into experimental and control groups. Pretest scores indicated equivalent performance on addition and subtraction problems. The experimental group used software specifically designed to develop math skills based on the principles of Precision Instruction, while the control group used math skills training software which was not based on the principles of Precision Instruction. The development of fluency in experimental subjects was plotted on Standard Celeration Charts which could not be graphed for Control subjects because frequency of correct responding was not measured under the control condition. A Post-Test compared each group’s performance in terms of total correct and frequency correct on addition and subtraction. Data indicated an increase in frequency of correct responding of the subjects who used the Precision Instruction software and a decrease in frequency of correct responding of the students who used the non-Precision Instruction software. However, the study contained several methodological flaws, which required resolution, before firm conclusions could be drawn concerning the value of the Precision Instruction software.

Evaluation of a Math Skills Software Package Based on the

Principles of Precision Teaching.

The goal of education is to enable students to behave quickly and correctly in a variety of situations. The term for this capacity is “Fluency”. Fluency is a function of the skills developed. The raison d`être of Precision Teaching is to build fluency and implement several procedures, the most important of which is timed practice. Precision Teaching was developed by Ogden Lindsley (1956) when he applied Skinner’s principles of operant conditioning to human beings in an educational setting and found the effectiveness of speed drills for acquiring fluency. The reason Precision Teaching succeeds is the power of repetitive work which meets a speed requirement to increase proficient learning. Precision Teachers have therefore further defined fluency as the rate of performance that makes the skill not only useful in everyday life, but also causes it to be remembered even after a significant period of no practice (Binder, 1987, 1988; Haughton, 1980; Johnson, 1992). Individuals achieve fluency as the result of frequency of timed practice with immediate feedback of correctness.

Precision Teaching methodology develops fluency with the use of frequent and short daily practice sessions, which are timed. It charts the rate of responses on Standard Celeration Charts. The Standard Celeration Chart graphically depicts the responses per unit time. The Standard Celeration Chart has a vertical logarithmic scale for rate and a horizontal arithmetic scale for time. Since learning is usually exponential, the Standard Celeration Chart presents a linear representation of learning. The use of Standard Celeration Chart allows for an indication of change of rate by the slope of straight lines. It also allows the individual or the teacher a graphic view of a direct measurement of performance. Users are declared fluent only when they have reached a specific score based on a combination of frequency and rate (Johnston & Pennypacker, 1982). An added advantage of Celeration Charts is that a prediction of when mastery will occur can be made. If a student's performance appears erratic on the Chart, both the student and teacher can assess what is happening in the environment or academic life that causes these variations and can make necessary changes. The details of Lindsley's concept of Precision Teaching can be partitioned into several major activities. The participants do the following:

Use Standard Celeration Charts. The Standard Celeration Chart is a device that graphically depicts the unit time per response. It also allows students to analyze, graph, and therefore monitor their own progress. This also allows instructors to make decisions about behavioral intervention based upon curriculum achievements (Pennypacker, Koenig, and Lindsley, 1972). An important aspect of the Standard Celeration Chart is that the user does not see changes in learning as simple increases or decreases in scores but instead sees a change in rate of learning.

Aim at fluency. At the core of Precision Teaching is the development of fluency rather than getting an answer correct. The best learning is occurring when the corrects are rising and the learning opportunities are dropping. On most charts, data indicated the combination of corrects rising and errors falling. Haughton and Haughton, noted that when elementary school students performed basic tool skills (e.g., addition) at frequencies from 100 to 200 per minute, they had more retention, more endurance, and more generalization than when they were taught to full accuracy but at frequencies, around 20 per minute. The advantage of teaching to higher frequencies have also been found at other grade levels, including graduate school in addition to use based measures, fluency has been shown to increase confidence and motivation (Binder, 1990; McDade, 1993).

Focus on rate of learning (celeration). Since an individual performance is a dot on the Standard Celeration Chart, and learning is the function connecting a series of performance across days, both performance and learning can be seen on the Standard Celeration Chart immediately. An important goal of education is increase the rate of learning in addition to simply teaching facts. The Standard Celeration Chart depicts rate of learning and can be used to gauge the success of interventions, which are used to increase rate of learning in addition to simply determining specific factual gains.

Precision Teaching is a practice monitoring system that allows the student to optimize efforts applied in many learning tasks. The methodology indicates strengths and weaknesses that need more work in any curriculum area. Researchers to date have implemented the principles of Precision Teaching at Quinte Learning Center (Maloney & Humphrey, 1982), the Haughton Learning Center (Freeman & Haughton, 1993), the Cache Valley Learning Center (Desjardins & Slocum, 1993), Morningside Academy (Johnson & Layng, 1992) and Jacksonville State University (McDade, 1987, 1980).

The most extensive research on the effectiveness of Precision Teaching was conducted in the Great Falls, Montana, public schools. This research included two validation studies included in the "Best Practices" section of the United States Office of Education's Joint Dissemination and Review Panel report. These were the first federal, validation studies (1975-1976), on children with skill deficits in the early grades. Three schools using Precision Teaching were compared with children in the same population put attending schools that did not use Precision Teaching. The children exposed to Precision Teaching were significantly superior in 15 of 19 (79%) post-instruction comparisons. There were no differences in three (16%) of the comparisons; and the control group was superior in one (5%) of the comparisons (Beck & Clement, 1991; Berquam, 1985; White, 1986). In the second validation study, regular (non-special education); fourth graders at a Precision Teaching school were compared to fourth graders at non-Precision Teaching schools over a four-year period. Precision Teaching students had significantly higher scores on the reading and mathematics portions of the Iowa Test of Basic Skills. Fourth graders instructed by Precision Teaching were performing in the 95^th percentile in reading and the 86^th percentile in mathematics during the fourth year of the study (Beck & Clement, 1991; White, 1986). Additional studies performed in Great Falls, Montana replicated the findings showing the superiority of elementary school students taught with Precision Teaching methods. The study showed that the skills acquired by students who were remediated by Precision Teaching did not regress after Precision Teaching was withdrawn. These experiments showed a significant superiority of Precision Teaching over standard techniques for teaching basic skills development in elementary school students (Beck & Clement, 1991; White, 1986).

Ferrucci (1984) compared the performance of fifth-grade students in two different school districts in achievements of rational number skills. Students in one district were taught through Precision Teaching and the other students in the other districts through traditional methods. The rate, computational skill development, and concept development with rational numbers, in the experimental group was significantly better than the control group in all cases.

Snyder (1992) reported on the use of computer-based rate building technology in several business-training applications. The author described a computer chip manufacturing facility's use of computer-based, verbal rate-building methods on procedures for donning anti-static suits and gear. This training was found to measurably decrease the amount of practice needed for the skill. Another company taught medical terminology to insurance claims examiners and brought participants to a high level in their rate of responding; this resulted in improvements in the examiners' accuracy and speed in checking medical claims. The author concluded that computer-based rate building training "has proven effective in terms of shorter training times, retained knowledge, and knowledge application for problem solving" (Snyder, 1992).

Computer-based software packages using Precision Teaching interventions have significant impact on the learning process through practice and time management. Studies by McDade & Olander (1987) and Olander & Merbitz (1980) found that computer-based programs could increase student performance. Well-designed computer-based Instruction software can greatly simplify the measurement of student performance (McDade & Olander, 1987; Olander & Merbitz, 1980). A progressive and successful computer-based instructional course requires the basic methodology of Precision Teaching and a flexible, user-friendly authoring system (McDade, 1992).

Since 1978, the Academic Center for Excellence (formerly, the Center for Individualized Instruction) at Jacksonville State University has been based on computer implementations of Precision Teaching (McDade, Brown, & Olander, 1988), this software integrates the fundamental concepts of Precision Teaching (Lindsley, 1972), the Personalized System of Instruction (Keller, 1968), and Direct Instruction (Engelmann & Carnine, 1982) into a unique learning environment. The integrated approach is based on user self-pacing, repetitive testing over given material to obtain mastery. Peers, tutors, and immediate feedback in addition predefined scripts of small units are verbally presented by the instructors and repeated orally by the students.

In the early part of the 1990's, Course Builderä and AuthorWare Professionalä software were used by Center for Individualized Instruction (Jacksonville State University) to introduce a Computer-Based Precision Learning Systemä. The program is a flexible, course-authoring template that allowed individual instructors to present material in any discipline and to choose mastery criteria, practice formats, test formats, and feedback conditions for the students. Faculty develops a large pool of test items to allow students to repeat testing with little duplication until they obtain mastery. Both course-authoring systems were used because nonprogrammers can easily develop interactive presentations, self-paced learning sequences, intelligent tutoring systems, customized performance reports, and simulations. Course Builder and AuthorWare offer the course designer flexibility in creating courseware, allowing the designer to organize the presentation order and routes while connected to various subroutines--the basic building blocks of these course-authoring systems. The end template presents current information to the learner in the form of text, color, graphics, animation, sound, hypermedia, and interactive videotape/disc technology. The format allows students to specify responses to questions and to receive feedback.

The major weakness of Course Builder and AuthorWare lie in their inability to copy or print some data, as well as their lack of powerful tools to affect all programming functions; there are additional inadequacies in reporting functions. Once users type the questions into the presentation template, they may not alter them to verify correctness or balance until the whole program is completed. Although they can alter the course map, once most subroutine programs are running, they cannot change its contents. Cut and paste functions are not available under all conditions. In summary, the inadequacy of available course authoring programs does not provide for the additional requirements, required by Precision Teaching, to be incorporated in their pedagogical sequence. The Center needed a user-friendly course authoring system that could accurately time student performance and output the rate data. Critical for effective Precision Teaching it was determined that the Center would have to develop its own software.

The present research study is an expansion of research first carried out in1992 to assess the effectiveness of the Computer-Based Precision Learning System (McDade, Brown, & Vance, 1993). The pilot program is written in Visual Basic 6 because that language is the most flexible for developing a mathematics tutorial. The package as envisioned would permit instructors to design courses using Visual Basic 6, AuthorWare, or Course Builder, whichever suits their needs. The new developmental mathematics program, based on the principles of Precision Teaching incorporates a unique approach to developmental math skills software packages (Beckett, 2000). The new software incorporates the principles of Precision Teaching for enhancing students’ performance. It possesses the following characteristics:

1. Randomly generate large numbers of practice problems, organized by concept and difficulty.

2. Provide students with brief explanations of how to solve problems, when requested.

3. Provide more problems of a specific concept/difficulty level until a predetermined success rate has been registered.

4. Assign Precision Teaching modules based on specific types of practice problem errors.

5. Match concepts to a text book.

6. Provide the student with:

< A timer

< Sets of tool skill "flashcards" which may be randomly shuffled to allow repeated practice and timings.

< Hardcopy flashcards of specific tool skills so for the student to use to practice at home.

< Feedback on progress by graphing results on a Standard Celeration Chart.

< A software module that would allow students to install a timer on their own home computers, so that they can practice the tool skills at home and bring in their results on a diskette.

The format and features available in the new software conform to the specific requirements of Precision Teaching. The rationale for present research is to find program weaknesses in the math skill software package based on the principles of Precision Teaching and rectify them. Additionally, the software evaluation will also help to identify and correct procedural problems in the entire pedagogical system.

Method

Participants:

The study was begun with 108 undergraduate students drawn from Academic Success Skills classes at Jacksonville State University; however, only 83 students were actually used in the study. Twenty-five students were dropped from the study because they failed to take the Post-Test or did not sufficiently use the assigned software. At the conclusion of the study, the control group consisted of 48 students, and the experimental group consisted of 35.

Each participating class was randomly divided into two groups before the Pre-Test. The Control Group was designated as Group C and the Experimental Group was designated as Group E.

Design:

The study used a Pretest-Posttest Design with randomly assigned treatments. The independent variable was the Precision Instruction math skills training program for Addition and Subtraction. The Control Group was given math skills practice on addition and subtraction using a non-Precision Teaching math skills training program. All participants were required to take a Pre-Test and Post-Test which evaluated their basic math skill and to complete assigned math skills practice programs. All participants were required to sign in and out with both the date and time. The progress of 8 subjects randomly chosen from the Experimental Group was plotted on the Standard Celeration Chart.

The Control Group used MathCueä Course Management (Version 2.2) a commercially available program for addition and subtraction. Only participants in Group C had open access to the section on addition and subtraction. Group C was required to answer at least 32 out of 40 correct answers (80% mastery level) before they could end the session. Participants were required to print a hardcopy of their results, so that their achievements could be certified.

The Experimental Group was required to use the math skill software (based on the principles of Precision Teaching) for addition and subtraction from the Center’s new program. Only participants in Group E had open access to this program. Group E had to answer at least 20 out of 25 correct answers (80% mastery level) on a unit. In addition, Group E was required to use a practice sheet program commonly used for Precision Teaching along with a random number generator incorporated in the software. Within a set of predefined questions, the random number generator allowed the integers in the problems to be changed constantly without merely being reshuffled. The program indicated to the user the number of correct and incorrect answers for every attempt in the unit. On each new trial, the user was required to set the timer either at 15 seconds, 30 seconds, 45 seconds, or 60 seconds before attempting to do the practice sheet. The user was required to reach a fluency level of 50 –70 number correct per minute before he or she was allowed to end the unit on any given day.

Observation and Recording Procedures:

A Pre-Test and Post-Test were given to all participants by the researcher in the Academic Success Skills classes. The researcher was present during the interaction with the software to ensure that each user had reached the mastery level as required by the study.

Equipment:

The pilot study used six IBM compatible computers designated solely for the experiment. In addition, the Academic Center of Excellence’s computer laboratory was available for the study when other students in the Center were not using it. To verify their work, all participants were required to sign in and out for each session.

General Procedure:

As stated above, the Pre-Test and Post-Test were required for every participant.

The control group was classified as Group C and used a commercial program, MathCueä. The experimental group was classified as Group E and used the pilot program for addition and subtraction. Both groups were allowed unlimited access only to their own programs. All participants interacted with the program assigned to them between the Pre-Test and Post-Test. All participants were allowed to use the software at their own convenience, but within the time period of the experiment.

Both participants in Group C and E used their assigned programs a minimum of five times for a week, and following that, all participants took the Post-Test. Once a participant completed the test, a letter of participation was presented to him or her, granting bonus points for course credit in Academic Success Skills.

Measures:

An Independent t test ( t_.01, two-tail test) was performed to measure the equivalency in the Pre-Test results for Group C and Group E for 108 students. However, there was differential attrition within groups after Post-Test. This differential attrition makes interpretation of significance problematical. As a result, an independent t test ( t_.01, two-tail test) was performed to measure the equivalency in the Pre-Test results for Group C and Group E for 84 students. In this way only those students completing the test would contribute to the “before” baseline. Since in the Post-Test the Experimental Group had an n of 35, while the Control Group had an n of 48, an evaluation of equivalency of Pre-Test results for dropouts in Group C and Group E was done to evaluate if there was a difference in the performance of the dropouts that could had caused the data to be skewed.

Dependent t tests (t_.01, two-tail test) were calculated based only on the students finishing the program. Using a dependent t test (t_.01, two-tail test), a comparison was performed for Pre-Test to Post-Test changes between Group C and Group E performances for total correct and frequency of correct responding. An Independent t test (t_.01, two-tail test) was performed to measure the equivalency in the Post-Test results for Group C And Group E. Using a dependent t test (t_.01, two-tail test), a comparison was performed for Pre-Test to Post-Test changes between Group C and Group E performances for total incorrect. Standard Celeration Charts displayed the celeration of students in Group E as they mastered the math skills software package based on the principles of Precision Instruction.

Results

Evaluation of Equivalency of Pre-Test Performance for 108 Students

Based on a t test (t = -0.85, 107df), there was no significant difference in the total correct between the Pre-Test for Group C (mean of 16.61 with n = 54) and Group E (mean of 17.28 with n = 54). The hypothesis that there was no significant difference between the Pre-Test total correct in Group C and Group E was therefore accepted.

(See table 1.)

Based on a t test (t = -0.85, 107df), there was no significant difference in the frequency of correct responding between the Pre-Test for Group C (mean frequency of 5.54 with n = 54) and Group E (mean frequency of 5.76 n = 54). The hypothesis that there was no significant difference between the Pre-Test frequency of correct responding in Group C and Group E was accepted.

(See table 2.)

Evaluation of Equivalency of Pre-Test Performance of 83 students

Since 25 students did not complete the training or did not take the Post-Test and because there was differential attrition within groups, dependent t tests were calculated based only on the students finishing the program. In this way only those students completing the test would contribute to the “before” baseline.

Based on a t test (t = -0.95, 83df), there was no significant difference in the total correct between the Pre-Test for Group C (mean of 16.79 with n = 48) and Group E (mean of 17.66 with n = 35). The hypothesis that there was no difference between the Pre-Test total correct in Group C and Group E was therefore accepted. (See table 3.)

Based on a t test (t = -0.95, 83df), there was no significant difference in the frequency of correct responding between the Pre-Test for Group C (mean frequency of 5.60 with n = 48) and Group E (mean frequency of 5.89 with n = 35). The hypothesis that there was no difference between the Pre-Test frequency of correct responding in Group C and Group E was accepted. (See table 4.)

Evaluation of Equivalency of Pre-Test for Dropouts in Group C and Group E

It should be noted that the two groups had differential attrition. Since in the Post-Test the Experimental Group had an n of 35, while the Control Group had an n of 48, this differential attrition makes the interpretation of significance problematic.

Based on a t test (t = -0.74, 23df), there was no significant difference in the total correct for dropouts between the Pre-Test for Group C (mean of 15.17 with n = 6) and Group E (mean = 16.58 with n = 19). The hypothesis that there was no difference between the Pre-Test total correct in Group C and Group E was therefore accepted. (See table 5.)

Based on a t test (t = -0.74, 23df), there was no significant difference in the frequency of correct responding for dropouts between the Pre-Test for Group C (mean frequency of 5.06 with n = 6) and Group E (mean frequency of 5.53 with n = 19). The hypothesis that there was no difference between the Pre-Test frequency of correct responding in Group C and Group E was therefore accepted. (See table 6.)

Evaluation of Pre-Test to Post-Test Change

Based on a t test (t = -5.49, 47df), Group C had a significant drop in the total correct from Pre-Test to Post-Test (mean of 16.79 and mean of 14 respectively with n = 48). The hypothesis that there was a difference between Pre and Post-Tests total correct in Group C was therefore accepted. (See table 7.)

Based on a t test (t = 5.33, 34df), Group E had a significant increase in the total correct from Pre-Test to Post-Test (mean of 17.66 and mean of 21.09 respectively with n = 35). The hypothesis that there was a difference between Pre and Post Tests total correct in Group E was therefore accepted. (See table 7.)

Based on a t test (t = -5.49, 47df), there was a significant difference in the frequency of correct responding for Group C’s Pre and Post-Test (mean frequency of 5.60 and mean frequency of 4.67 respectively with n = 48). Differences indicated a drop in the frequency of correct responding. The hypothesis that there was a difference between Pre and Post-Tests frequency of correct responding in Group C was therefore accepted. (See table 8.)

Based on a t test (t = 5.33, 34df), there was a significant difference in the frequency of correct responding for Group E’s Pre and Post-Test (mean frequency of 5.89 and mean frequency of 7.03 respectively with n = 35). Differences indicated an increase in the frequency of correct responding. The hypothesis that there was a difference between Pre and Post-Tests frequency of correct responding in Group E was therefore accepted. (See table 8.)

Evaluation of Equivalency of Post-Test for the Experimental and Control Groups

Based on a t test (t = -6.63, 81df), there was a significant difference in the total correct between the Post-test for Group C (mean of 14 with n = 48) and Group E (mean of 21.09 with n = 35). The hypothesis that there was a difference between the Post-Test total correct in Group C and Group E was therefore accepted. (See table 9.)

Based on a t test (t = -6.63, 81df), there was significant difference in the frequency of correct responding between the Post-Test for Group C (mean frequency of 4.67 with n = 48) and Group E (mean frequency of 7.03 with n = 35). The hypothesis that there was a difference between the Post-Test frequency of correct responding in Group C and Group E was therefore accepted. (See table 10.)

Evaluation of Pre-Test to Post-Test Change in Total Incorrect

Based on a t test (t = 1.17, 47df), Group C had a significant increase in the total incorrect from Pre-Test to Post-Test (mean of 1.69 and mean of 2.13 respectively with n = 48). The hypothesis that there was a difference between Pre and Post-Tests total incorrect in Group C was therefore accepted. (See table 11).

Based on a t test (t = -0.16, 34df), Group E had a significant drop in the total incorrect from Pre-Test to Post-Test (mean of 1.71 and mean of 1.66 respectively with n = 35). The hypothesis that there was a difference between Pre and Post Tests total incorrect in Group E was therefore accepted. (See table 11.)

Evaluation of Standard Celeration Charts on a Sample of Students in Group E

Standard Celeration Charts on a sample of students from the Experimental Group show the obtained changes in the students' learning skills. On each Chart, the students had a gradual increase in the slope (celeration line). It can be concluded that whole sample (n = 8) is accelerating. (See accompanying Charts.) On the Chart, the number of corrects is indicated by dots, joined together from day to day. The best learning is occurring when the dots are rising. When this happens, a line is drawn through the middle of the dots from the earliest to the most recent. The slope of the line is the celeration of the student’s performance. Celeration is a linear representation of performance growth (or decrement). Acceleration is indicated by an “x” sign with a numerical indicator of change, so “x2” means doubling of performance, while “x4” means quadrupling.

Daily charts indicate only the highest number of frequency correct per response for the day derived from the timing chart. These indicated that the students were improving on at least "x1.2 to x2.3" rate of learning. In fact, a few individual student’s timing chart, used to plot every single attempt made, indicate a jump in acceleration from day 1 to day 5 (i.e., “x1.5 to x4.5”, “x1.2 to x2.6”, “x2 to x2.3”, “x1.8 to x2”, and “x1.8 to x2.3”). On the last day of the study, all the students had achieved a higher frequency correct with a slope ranging from “x1.2 to x4.5.”

Discussion

This research found a difference between the basic adding and subtracting skills of students completing a computerized training program based on the principles of Precision Teaching, as compared to those students using a training program that was not based on the principles of Precision Teaching. The significance of this finding must be tempered, however, by flaws in the experimental design and results which challenge the study's internal validity. Factors which may have influenced internal validity include 1) differences between the treatments received by the Experimental and Control Groups other than those which differentiate Precision Teaching from traditional methodology, 2) the differential dropouts (attrition) in the Experimental Group and the Control Group, 3) the significant drop in the performance (Pre-Test to Post-Test) of the Control Group, and 4) non-methodological factors.

Differences between treatments received by Experimental and Control Groups other than those which differentiate Precision Teaching from traditional methodology need to be addressed. The study compared commercially available software Math Cue™ (Control) with software designed for Precision Instruction (Experimental). Many differences existed between the two software packages, including number of digits in practice problems (as discussed earlier), number of questions per module, variable time constraints chosen by students, and mastery vs. fluency requirements. One question is whether a fair comparison was made on frequency testing. The Control Group had to do 40 questions, while the Experimental Group had to do 25 questions, in addition to the tool skills (timed drill of math facts) module which has a series of single or two digit problems (i.e., 1 + 1 =, 0 + 1 =, or 8 – 0 =).

Control Group students were required to do five or more sets of questions with at least 80 % accuracy for the whole week. For each interactive session, students were only allowed to stop for the day if they have achieved at least 32 correct out of 40 questions per session. Example: Day 1 – On the first attempt, the student got a result of 38 correct; the student had fulfilled the requirement, so he could end the interactive session for the day. However, if the student’s results were below 32 correct, the student was required to do it again until mastery was achieved. Most of the students did a minimum of 10 sets of questions for the whole week; time to complete each set ranged from 20 –30 minutes.

Experimental Group students spent approximately 5 to 15 minutes per session. Most of the time spent by the Experimental Group was on the tool skills. Most of the students chose to use the 15 seconds per attempt because their frequency of correct responding was higher with less time and effort. Students could choose intervals from 15, 30, 45, or 60 seconds according to their individual needs. Students were able to avoid the more effortful 60 seconds interval by selecting the 15 seconds interval and consequently avoiding more work. The subject must obtain a rate of approximately 13 correct answers in 15 seconds before he or she can expect to reach a mastery level of 52 correct per minute. Most of the students spent more time on the tool skill module than they did on the 25 questions part of the program (mastery of 80% correct).

Examination of the Pre and Post-Test questions revealed that, despite having the same question format, the Experimental Group, working with single digits on tool skills rather than double digits, could have had an advantage in solving the problems. The use of single digit problems could increase individual performance skill level. Precision Teaching has shown that once students have an increase in their fluency level with basic mathematical facts (addition and subtraction), the skill is generated to more complex problems. Practicing the tool skills (timed drill of math facts) in a sequential and progressive manner required the experimental students to practice all their arithmetic facts from 0 to 9 digits. The practice sheet is designed to have no more than one or two digits for rapid performance at frequencies above 50 per minute and includes more questions than could be answered in the allotted time (Lindsley, 1997).

A secondary flaw that needs to be discussed is the differential dropout rates (attrition) of the Experimental Group and the Control Group. In the Experimental Group, 19 students dropped out while only 6 of the Control students dropped out. The high dropout in the Experimental Group (using the math skills software based on the principles of Precision Instruction) raises questions as to whether the subjects lost interest in doing the work, did too much work for too little reinforcement, and/or found the program too demanding, aversive, taxing, or time consuming. Concerned for the high number of dropouts, the researcher compared dropout group performance and non-dropout group performance during the Pre-Test. Data indicated no significant differences in their results, suggesting no selection-mortality threat responsible for the dropout rate. A detailed analysis indicated thirteen students who dropped out did not take the Post-Test (three from the Control Group and ten from the Experimental Group). Two students had medical excuses. Ten students (three from the Control Group and seven from the Experimental Group) did not satisfy the requirements of interacting with the assigned software. In future studies, more control over the subject pool is necessary, since data from dropouts could have changed the outcome.

Lack of motivation to complete the work was one of the possible causes for dropouts in both the Experimental and Control Groups. The work that the students had to do in the study was not stimulating, since the work was not related to anything that could assist in their current classes. Individual charting of performance progress is one of the main components in Precision Teaching. However, in the study the students were not made to chart, the researcher selected only 8 students (Experimental Group), and plotted their results in the Standard Celeration Chart. The fact that they did not chart could have adversely influenced their performance. The bonus points that students could gain from finishing the study may not be reinforcing enough when compared with other contingencies in their lives. In addition, consideration of confounding variables such as holidays and exam schedules must be addressed in future studies. This could be reason for distraction and fatigue, which could explain their lower level of performance.

A third issue that needs to be addressed is that the Post-Test performance of the Control Group was lower than the Pre-Test performance. An increase in both the Experimental and Control Groups’ performance should be expected after instructional software designed to improve performance. Post-Test performance in the Control Group should have been the same or slightly higher. However, results indicated a drop in the overall mean performance of the Control Group from Pre-Test to Post-Test. This contrasts with data indicating a modest increase in the mean performance of the Experimental Group. This could be due in part to the fact that students were allowed to work within the minimum practice requirement (the set number of problems that the students were required to do). Control Group students had to complete 36 out of 40 questions; Experimental Group students had to complete 23 out of 25 with Tool Skills above 54 correct per minute, as compared with those subjects who practiced above the minimum requirements. This design presents another possible selection – instrumentation threat in the study. Not all the students in the study finished in the required amount of time or at the desired mastery level before they ended the session. The minimum number of practice sessions could account for the low performance results on both the Control and Experimental Groups.

It was expected that the Experimental Group would decrease errors from Pre-Test to Post-test, and that the Control Group would show no change or a slight decrease in errors. However, there was no significant change in the numbers of errors from Pre-Test to Post-Test for the Experimental Group. In addition, there was a significant increase in the number of incorrect from Pre-Test to Post-Test in the Control Group, as well as a decrease in the number of attempts.

Since these findings were inconsistent with expectations, it is difficult to understand what factors may account for these outcomes.

Non-methodological factors, which were criticized by participants of the math skill software package (based on the principles of Precision Instruction), include instructions on how to use the equipment during implementation. They stated that the protocol presentation template was too small, and that when enlarged, the template did not improve. The timer that informed the students of the number of seconds remaining in each session distracted them. In addition, the timer that started the software tended to freeze when the user accidentally pressed a key other than the enter key. The sounds within the program received a mixed response; some students found them advantageous and helpful, while others found them to be annoying. This further complicated the experiment with another confounding variable. Most of the students agreed that a help icon would assist in clarifying any confusing or unclear components. These suggestions are being used to modify the math skill software package based on the principles of Precision Instruction.

Another major concern and confounding variable was the mechanics used during testing. Many/all students felt they lacked the typing skills needed to key in the answer quickly. They felt they were unable to beat the timer on the module. That, to a certain extent, hampered their ability to improve. The problem was noted during the early stages of exposure to the software and should be addressed in future studies. Even though the students who participated in the Precision Instruction software performed better, the inherent flaws in the methodology forbids the conclusion that the software alone was effective in raising students’ performance. In the study, students using the Precision Instruction based software were constantly urged to improve their previous results, competing against themselves and not with others. Each time students reset the program, a random selection was presented; therefore, their performance skill levels were variable from drill to drill. The continuous change in the selection of the questions made it difficult for them to get the same level of results on each attempt. All of these factors combined to make the students more aware of their performance skill level, urging them to do their best in every attempt because each session demanded skill. Simple mathematics concepts allowed each individual student to work at his own pace and gradually to speed up to his mastery level requirements. Students using the software were not penalized for low scores but were encouraged to beat their poor time by increasing the number of correct answers per minute. The goal of the software was to help each student to develop at his or her own pace.

Standard Celeration Charts from randomly chosen students in the Experimental Group indicated fluency development from the computer program's interactions. Charts indicate that the students were improving on a Celeration of at least “x1.2 to x2.3” rate of learning. In the Chart, the number of corrects is indicated by dots, joined together from day to day. The best learning is occurring when the dots are rising at the highest slope. When this happens, a line is drawn through the middle of the dots from the earliest to the most recent. The slope of the line is the celeration of the student’s performance. Celeration is a linear representation of performance growth (or decrement). An acceleration is indicated by a “x” sign with a numerical indicator of change, so “x2” means doubling of performance, while “x4” means quadrupling. A deceleration is indicated by a “)” sign, so a “)2” means a decrement of doubling down performance. In typical classroom, weekly improvement in skills is “x1.1” or very little change. Precision Teaching interventions are considered successful with “x2” or more improvement. The increase in slope (i.e., celeration) could be due to an improvement in performance of typing skills, as a result of practice, rather than increase in math skills.

The hypothesis that the Precision Instruction based software can increase the fluency of the student cannot now be accepted because of methodological flaws and ambiguous results, which threaten internal validity. Future studies must address the below-mentioned areas of concern:

1. The significant number of dropouts

A. Better system of reinforcement

B. Encourage careful prompting by instructors

C. Plan testing with consideration to distracting factors

I. Holidays

II. Mid-term and Final Exam schedules

2. Requirement standards should be raised

3. To improve sampling methodology, future test groups should be divided into two distinct groups independent of each other, guaranteed by more stringent randomizing of subjects to groups. Independent selection of the groups will decrease possible contagion between groups.

4. Keyboarding skills should be measured and controlled prior to experiment

In retrospect, consideration should be given to using more difficult problems requiring a higher skill level (i.e. division and multiplication that will take more effort and learning). This would better demonstrate the skill acquisition of the student than does basic addition and subtraction problems used in the current study. Future use of software package should require longer drills; 15 seconds is much too short. It is hoped that this study will stimulate further investigation into this field of integrating educational software with the principles of Precision Instruction. It is apparent that additional research will be required to determine the efficiency and stability of the Precision Instruction program after modifications have been made to ensure validity.

Table 1

Total Correct Performance for 108 Students