Learner Perceptions of TOEIC Test Results and Language Skill Improvements |
||||||||||||||||||
| “I don’t want to study English, I want to study TOEIC” | ||||||||||||||||||
|
By Howard Brown, James English School |
||||||||||||||||||
|
Background
|
||||||||||||||||||
|
The TOEIC test is extremely popular among business interests in Japan. It is widely held to be the test for assessing business English skills and a business professional’s TOEIC score is commonly used as a benchmark for promotion or even job retention. Also, despite being not particularly effective at assessing individual learner achievement over the short term (Childs, 1995), the TOEIC is regularly used as a pre and posttest for corporate training sessions, sometimes as short as 12 contact hours. For commercial language schools and textbook publishers TOEIC preparation is big business. Traditional TOEIC preparation programs focus on analysis and practice of discrete items. Students are generally happy with this approach because they can appreciate its “TOEIC-ness”. It is not unusual for an eikaiwa student to ask for TOEIC preparation lessons saying “I don’t want to study English, I want to study TOEIC.” Programs that focus on English skill building as a route to TOEIC success tend to be a tough sell as learners may not perceive them as being effective. This paper explores one such alternative course, the learner’s perceptions of the course and the lessons learned in its development. BackgroundThe development process for this course grew out of the idea that traditional discrete point analysis TOEIC classes were not the most effective way. Class observation and interviews with the learners identified several problems with the approach. l For low-level learners, the vocabulary and syntactical complexity of the sample problems used in class had to be controlled. This led to a lack of fit with the actual TOEIC test. The students were unable to recognize question patterns studied in class when they encountered them in authentic TOEIC questions. Miller (2003), in an extensive survey of commercial TOEIC preparation materials, has cited this lack of fit with actual test items as part of a general trend of low quality materials. l The isolated point teaching led to inappropriate test taking strategies. The learners were focusing on questions as linguistic puzzles rather than looking at the meaning. This led to a great deal of test time being used searching for the answer in the words – trying to find matching key words, rather than trying to understand what the questions were asking. While this strategy may have some value on sections 5 and 6 of the test, it is not generally very helpful on other sections. l The learners displayed a generally low tolerance for ambiguity. The feeling that it was necessary to understand each and every word in a problem was common among the learners. Having a general understanding was insufficient. Strategies such as process of elimination or narrowing the filed to two choices before guessing were taught but were not being used. This led to inappropriate guessing strategies such as immediately answering B for every question that was not fully and confidently understood or, in one extreme case, simply skipping an entire section of the test because it was “too hard”. l A recurring problem for lower level students on the listening section was not being able to read the questions fast enough to understand and answer them before the tape had moved on. These same students reported not being able to finish the test in the time allowed because they could not read the questions fast enough. The course was doing nothing to address this issue. The course designer decided to operate on the assumption that skills-focused general English or business English classes could be as effective at improving TOEIC scores as specific test preparation classes (or perhaps even more effective). Research into this assumption in the Japanese context appears to be somewhat contradictory. Robb and Ercanbrack (1999) for example concluded that TOEIC focused study is in fact more effective at improving TOEIC scores, but only for lower level students. Also, Cunningham (2002) found that improvements in TOEIC score do not correlate with improvements in communicative skills suggesting that skills based work may not be effective for improving TOEIC scores. On the other hand, Narron, Hirase, Minami, Takekata and Adachi found, in their work with TOEFL preparation, an “absence of any marked practical benefit in coursework specifically tailored to test preparation” (2003, p.12). In all three cases, these studies were conducted in a university environment. The CourseThe development process started with the idea that reading was a critical skill for TOEIC success. As well as the actual reading comprehension questions students are required to read and understand 75% of the questions on the test. Two goals were set for the program. The first was to increase the students’ reading speed and ability to focus on meaning in reading. The second was to improve their confidence in reading English. To achieve these goals the use of graded readers was settled on. Graded readers are known to have “four linguistic benefits: building reading speed, lexical speed access, reading fluency, and the ability when reading to move from working with words to working with ideas” (Waring, 1997, para.4). These four benefits seem to be directly relevant to the four problems identified with the earlier test-focused approach. Thus graded reading was assumed to have the potential to benefit TOEIC students. It was assumed that skills developed in pleasure reading would transfer to intense tested reading. Because the course was conducted in a commercial environment, it was impossible to move completely away from the traditional TOEIC-focused approach. Program stakeholders on both the course provider side and the client side insisted that the program contain some TOEIC specific elements. The graded reading program was instituted as a supplement to the existing TOEIC course. Also, at the request of the client, the original self-access extensive reading program was modified to a more controlled homework and in-class review model. The ParticipantsThe program had a total of 25 participants divided into 3 achievement levels. The group was overwhelmingly male (23 male 2 female) and composed of university-educated professionals - engineers for the most part. Students ranged in age from 22 to 45. None of the participants had majored in English though several had had experience studying English abroad for varying lengths of time. The StudyThis study focused on students’ reactions to the program. It was assumed that reading the graded readers could help the learners’ improve their TOEIC score. The question being studied was whether the participants could be “sold” on the idea of reading novels as a TOEIC study tool. The rationale for this decision was two-fold. Firstly, investigating the actual effectiveness of graded reading was beyond the scope of what could be accomplished in a commercial context. Secondly, and more importantly, learner affect is known to be a key factor in the success or failure of a program. Data was collected in surveys of program participants, reading reports submitted when the students had completed each novel, and follow-up interviews based on the reading reports. ResultsInitial results were consistent with expectations. At the outset of the course, an information session was conducted and the course was greeted with a great deal of skepticism. As expected the students could not see the “TOEICness” of the course. Reading reports collected when the students finished their first book confirmed the problem. The average score given by students, when asked to rate the statement “Reading this book helped my TOEIC score” on a 1-5 Likert scale, was 2.4. Fully half of the students responded, “Disagree” and only one participant agreed with the statement. Interestingly, 3 statements focusing on improvements in language skills (overall improvement, reading speed, reading comprehension) were rated higher (see table 1). An unpaired 2-tailed t-test was calculated and the differences were found to be significant (t<0.0001, t=0.0006 and t=0.0002 respectively). This indicates a lack of connection, in the students’ minds, between improvements in English skills and improvements in TOEIC scores. This indication was borne out by comments collected in follow up interviews with students. Table 1: Student’s reactions to graded reading
This lack of connection was seen to disappear as the course continued. Subsequent administrations of the same survey questions showed a much clearer understanding of the value of the reading for TOEIC. Surveys conducted at the 6-month mark in the course showed slight improvements in the students’ opinions of the value of reading for their overall English, reading speed and reading comprehension. However, the connection between the reading program and TOEIC score was significantly higher. The differences between the four items ceased to be significant. (t>0.1 in all cases). This change of attitude appears to have come about as the result of several factors. •Bilingual
course outlines detailing the rationale for using graded reading •Mock
TOEIC tests timed to coincide with the end of reading cycles •Regular
counseling and feedback sessions conducted in the learners’ first
language •Reading
speed self-checks using TOEIC style materials •Self
evaluation reading reports at the end of every book It seems that the students were able to come to see improvements in their test taking abilities and their confidence in the program improved as well. This pattern was seen to continue throughout the two years of the program. Incoming students consistently underrated the contribution of graded reading to their TOEIC score. Then, following their first mock TOEIC the rating improved. Outgoing students’ comments were consistently positive in their evaluation of the reading program. In fact, the reading program was seen by the stakeholders to be such a success that it was modified. The controlled reading and homework model has been replaced by a self-access extensive reading model in the current version of the TOEIC course. The reading element is optional but has an 80% participation rate. The question of the actual effectiveness of grading reading for TOEIC improvements was beyond the scope of this study. There are simply too many uncontrollable factors to make such a study feasible in a commercial environment. A great deal of further research would be required to ascribe the improvements in TOEIC score directly to the use of graded readers. It would be interesting to track learners reading speed in both novels and TOEIC material to establish a correlation between improvements in reading for pleasure and focused tested reading. This would establish that a transfer of skill was in fact taking place. ConclusionsIn a commercial environment, where the learners are both students and clients, teachers often face a conflict between giving the students what they need and giving the clients what they want. In the case of TOEIC preparation, the discrete item focused approach may be what the majority of clients want, but is it what the students need? Graded reading can be an effective tool for TOEIC study but it will not be readily accepted by many learners, and other stakeholders. This study was able to confirm that learners looking for TOEIC score improvements often do not see a connection between test scores and actual language skills. Detailed first language course descriptions and a mock TOEIC administered soon after the beginning of a graded or extensive reading program may help learners see the connection. Acknowledgements
The author would like to extend appreciation to the teaching and management staff of James Eikaiwa, Joetsu Branch who made this report possible. Also, though the course participants in this study have requested anonymity for themselves and their company, they deserve thanks for allowing their learning experiences to be shared. References
Boldt, R.F., &
Ross, S. (1998). Scores on the TOEIC
Cunningham, C. (2002) Do test score gains correlate with increased competence? Unpublished master’s thesis, University of Birmingham, UK. Retrieved January 6, 2003 from http://www.cels.bham.ac.uk/resources/essays/Cunndiss.pdf Miller, K. (2003). The Pitfalls of Implementing TOEIC Preparation Courses. Retrieved January 12, 2003 from the Shikoku University department of English Language and Culture website: http://www2.shikoku-u.ac.jp/english-dept/pitfalls.html Narron, M.,
Hirase, K., Monami, T., Takehata, S., Adachi, T. (2003). Teaching the test
or teaching the language. A look at test prparation. The Language
Teacher, 27(9), 9-12. Robb, T. N., Ercanbrack, J. (1999). A study of the effect of direct test preparation on the TOEIC scores of Japanese university students TESL-EJ 3(4). Retrieved January 11, 2003 from http://www-writing.berkeley.edu/TESL-EJ/ej12/a2.html
|
||||||||||||||||||
(Test of English for
International Communication) test as a function of training time and type.
Princeton, NJ: Educational Testing Service.
