A PROJECT USING ITEM ANALYSIS AND ON

Home

Resumé

Ph.D. Studies

A PROJECT USING ITEM ANALYSIS AND ON-LINE TESTING FORMAT FOR MULTIPLE-CHOICE QUESTIONS

Diane L. Jackson

A Project Presented in Partial Fulfillment Of the Requirements for

ED723 Online Assessment

Capella University, June, 2001

INTRODUCTION

The purpose of this project is twofold: (1) to use basic item analysis features to determine the usefulness of particular multiple-choice questions and (2) to practice on-line assessment techniques. This paper, however, is intended to be a compilation of notes acquired while completing the project and not as a summative evaluation of either item analysis or on-line assessment techniques.

ITEM ANALYSIS

With computer technology improving, teachers can now become educators. While test analysis was always a part of teacher training programs, it has not been an integral component of teachers' activities, perhaps due to time constraints. However, if teachers are to become professionals in the 21^st century, we will need to look at how we can use computer technology to improve the services we offer students of all ages and needs. Interestingly, with more than 13 years teaching experience, I has never conducted an item analysis on any questions. Unfortunately, quiz and test situations have become an entity in and of themselves with little to bear on the learning process other than the mark given to the students.

Item analysis (IA) can be used to provide for better class discussion of test results as poorly answers questions can be analyzed to determine whether the fault is due to poor question technique, a lack of understanding by the students, or a failure by the teacher to adequately address the issue. IA provides a better focus for reviews and remediation and improves teachers' test development skills. Blessum (1969) concluded that the use of item analysis "resulted in an improvement, not only in the quality and fairness of each individual examination, but also in the technical and educational quality of successive tests."

ITEM ANALYSIS PROCEDURE

Earlier research has shown that selecting the top and bottom 21% of any testing group provides accurate analysis information. Additional scores may be analyzed but the end results show negligible changes; therefore, the extra work is not justified. The author worked with two groups of students: a class group and non-class group. The purpose, of course, was to determine the effectiveness of the test measurement in measuring knowledge. The class group received the test in a normal testing situation and required about 20 minutes to complete all 48 questions. The non-class group consisted of volunteers who received credit for "community service" in return for answering the questions. This particular group consisted of a 50% female/male mixture. 67% of the group classified themselves as "poor" or "average" regarding computer knowledge and were not enrolled in a computer class at the time of the quiz.

Description of Total Procedure.

After scoring the papers, rank order from highest to lowest scores.

Put the 20% of the highest and lowest together.

For each test item, count the number of students in the high scoring group who selected each alternative. Do the same for the students in the low-scoring group, and record this information.

Compute the difficulty level (the percentage of students you are using in the analysis who got the item correct) of each item. A 50% DL is ideal but 20%-80% is good. However, any questions yielding either all right or all wrong answers should be regarded as suspicious.

Compute the discrimination index to show how well each option distinguishes between high and low scores of each option. A positive index of discrimination suggests that getting that particular item right correlates positively with higher total test scores. A negative index of discrimination suggests that students who get that item right generally score lower on the test than students who get the item wrong. If each item is measuring a part of the content covered by the total test, it is not a good sign if an individual item is not positively correlated with success on the total test. Listed below is a chart that may be used to determine whether or not a test item is good.

Index of Discrimination Item Evaluation

.40 and up very good item

.30 to .39 good item

.20 to .29 reasonably good item

.10 to .19 marginal item, usually subject to improvement

below .10 poor item, to be rejected or revised

Item Analysis of 48 Multiple Choice Questions. (Note: British spelling is used for the test.)

Students

Non-students

Questions

Totals

Notes: H = High; L = Low; DL = Difficulty Level; DI = Discrimination Index

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			0	0	1. What controls and coordinates the operation of the various types of equipment in a computer system?	50%	0%
				I	II			a. operating system software
II	II							b. systems software
								c. utility software
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	.50	2. What does applications software allow you to do?	75%	.25%
					II			a. allocate and assign system resources
II	II			II				b. apply the computer to solve a specific problem or perform a specific task.
								c. monitor system activities
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	.50	3. According to Is, what is the single most important piece of software in a computer system?	75%	0%
I	I			II	II			a. operating system software
I								b. systems software
	I							c. utility software

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			25	.25	4. What is the part of the operating system that manages the computer system's resources called?	62.5	.125
				I	II			a. client
								b. server
II	II			I				c. supervisor
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	.25			0	0	5. The form of multitasking that enables the computer to perform more than one task at a time within an application, such as editing a spreadsheet while the program performs some calculations in the background is referred to as	12.5	.125
I	II			II	II			a. multiprocessing
								b. multiprogramming
I								c. multithreading
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	0	6. When the operating system rapidly rotates through all users on the system, each user is allocated a very short slice of CPU processing time (a time slice), and the CPU performs whatever operations it can for that user within that time slice and then moves on to the next user in the rotation. This process is known as	62.5	.125
	I				I			a. CPU sharing
				I				b. program sharing
II	I			I	I			c. time sharing
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	.50	7. A program that contains a GUI that can be inserted to act as an interface between the user and an older operating system, such as DOS, is referred to as	75	.25
					II			a. a multitasking program
II	II			II				b. a shell program
								c. a systems program
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			75	-.25	8. What type of operating system works with only certain kinds of computers?	87.5	-.125
								a. portable
II	II			I	II			b. proprietary
				I				c. stable
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			100	0	9. DOS/Windows 3.1, Windows 95, Windows NT, OS/2, Macintosh OS, and UNIX are all examples of	100	0
								a. multiprogramming
II	II			II	II			b. operating systems
								c. portable systems

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	0	10. Machine language consists of	75	0
II	II			I	I			a. 0s and 1s
				I	I			b. alphabetic abbreviations
								c. symbols
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	.5	11. Assemblers, compilers, and interpreters are the three basic types of	75	.125
					I			a. assembly programs
					I			b. high-level languages
II	II			II				c. language translators
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	.50	12. The following -- COBOL, Turbo Pascal, Visual Basic and C+ -- are all examples of	62.5	.125
I	II			II				a. high-level languages
					II			b. low-level languages
I								c. medium-level languages
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	-.25			50	-.50	13. Which of the following tasks do utility programs perform:	37.5	-.375
								I. formatting disks
								II. copying files from disk to tape
								III. sorting files
								IV. editing files

								a. I and II
	I				II			b. I, II, III, and IV
				II				c. II, III, and IV
II	I							d. III, IV, and I
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	0			50	.50	14. Most application software packages share some basic characteristics including which of the following:	50	.25
								I. written documentation in the form of a user manual
								II. user interfaces that are just as graphical as those used for operating systems
								III. technical documentation

I	I			II				a. I and II
I	I				II			b. I, II, and III
								c. II and III
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			75	.25	15. Which type of software helps you to store, organize, and retrieve data efficiently?	75	.25
II	I			II	I			a. database
								b. integrated
	I				I			c. spreadsheet

H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	0	16. The third generation of programming languages, in which programs are written using English-like words combined into sentence-like statements are known as	62.5	.125
								a. higher-level languages
II	I			I	I			b. high-level languages
	I			I	I			c. low-level languages
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			75	.25	17. The word processing feature that automatically moves a word that will not fit within the margin at the end of a line down to the next line is known as	87.5	.125
								a. hyphenation
					I			b. truncated superscript
II	II			II	I			c. word wrap
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			75	-.25	18. The most common typeface sizes are	100	-.125
								I. 8 point
								II. 10 point
								III. 12 point

								a. I and II
				I				b. I and III
II	II			I	II			c. II and III
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	.25			0	0	19. An example of a sans serif typeface is	12.5	.125
	I			II				a. Courier
I								b. Helvetica
I	I				II			c. Times New Roman
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			75	-.25	20. In a spreadsheet package, a set of contiguous cells is known as a	100	-.125
								a. absolute address
				I				b. cell address
II	II			I	II			c. range
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	-.25			50	0	21. Automatic recalculation is a powerful feature of spreadsheet software that makes it easy to	62.5	-.125
I				I	I			a. copy formulas
								b. identify graphical predictions
I	II			I	I			c. perform what-if analyses
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	-.25			0	0	22. The most useful chart/graph for showing data over time and for comparing individual values at a particular point in time is	12.5	-.125
	I							a. a bar graph
II	I			II	I			b. a line graph
					I			c. a pie chart

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	0	23. A sequence of predefined keystrokes or mouse clicks to which you can assign a name is referred to as a/n	75	0
				I	I			a. automatic recalculation
								b. function
II	II			I	I			c. macro
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	-.25			50	0	24. Complex queries with multiple conditions require you to use	62.5	-.125
I	II			I	I			a. logical operators
								b. mathematical operators
I				I	I			c. relational operators
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	-.25			100	0	25. A type of software that combines word processing, spreadsheet, database management, and data communications into one easy-to-use package is referred to as	87.5	-.125
I	II			II	II			a. integrated
I								b. a suite
								c. distributed
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	.50	26. What type of software allows you to make charts and graphs, produced by analytical graphics programs, more visually appealing and to create professional looking charts, graphs, diagrams, and other presentation materials from scratch?	75	.25
					II			a. computer aided design software
								b. drawing software
II	II			II				c. presentation graphics software
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			0	0	27. Which input device is not used with drawing and paint programs?	25	.25
								I. digitizing tablet
								II. light pen
								III. electronic stylus

	II			II	II			a. I
II								b. II
								c. III
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	0	28. Quark Xpress is one of the most popular types of	62.5	.125
III	I			I	I			a. desktop publishing software
	I			I	I			b. spreadsheet software
								c. word processing software

H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			75	.25	29. Software that combines graphics, database management, and some spreadsheet capabilities to display data geographically is referred to as	62.5	.375
					I			a. database management system software
II				II	I			b. geographic information system (GIS) software
	II							c. presentation graphics software
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	0	30. What computer program is used to help manufacture products?	62.5	.125
					I			a. CAL
II	I			I	I			b. CAM
	I			I				c. CPM
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	.50	31. What type of software helps keep track of the influx and outflow of money in an organisation?	62.5	.375
II	I			II				a. accounting software
					I			b. mapping software
					I			c. project management software
	I							d. scheduling software
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			50	.50	32. The following components are basic parts of what type of system?	62.5	.375
								§ computers (or input or display devices) that send and receive the data involved
								§ a communications channel over which the data is sent
								§ communications equipment that helps send and receive data
								§ communications software that helps control the functions of the system

	I							a. communication modem system
					I			b. integrated digital service system
II	I			II				c. telecommunications system
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	.25			75	.25	33. "Lines" and "wireless" technology are two types of what type of media?	87.5	.125
								a. control
					I			b. protocol
II	II			II	I			c. transmission
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			0	0	34. What type of transmission media consists of copper wire surrounded by several layers of insulation?	25	.25
II								a. coaxial cable
					II			b. fiber optic cable
	II			II				c. twisted-pair wire

H	L	DL	DI	H	L	DL	DI		DL	DI
		75	.25			75	.25	35. What type of transmission media can transmit data at up to 2.5 gigabits per second?	75	.25
					I			a. coaxial cable
II	I			II	I			b. fiber optic cable
	I							c. twisted-pair wire
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	-.25			75	-.25	36. A communications satellite acts as a relay station for an earth station by receiving data, amplifying it, and then re-transmitting it. Where does it re-transmit it to?	50	-.25
I	I							a. another communications satellite
	I			I	II			b. another earth station
I				I				c. another relay station
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	0			50	0	37. Microwave systems need to be placed relatively close to one another because	50	0
								a. microwave frequencies are crowded
I	I			I	I			b. microwaves are lost in extreme weather
I	I			I	I			c. microwaves travel in straight lines
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			75	.25	38. Put the sentences in the correct order to describe a cellular telephone system	37.5	.375
								I. As the car moves into a new cell, a computer at the mobile switching center transfers the call to a channel in a new cell.
								II. A call from a regular telephone to a cellular phone is routed to a mobile switching center.
								III. It is transmitted by radio waves to an antenna in the geographic area (or cell) where the cellular phone is located and the phone then picks up the transmission from the antenna.

								a. I, II, and III
II	II			II	I			b. II, III, and I
					I			c. III, I, and II
H	L	DL	DI	H	L	DL	DI		DL	DI
		25	.25			50	.50	39. When was the Telecommunications Act passed that overhauled regulation of the telecommunications industry in the United States?	37.5	.375
I	II				I			a. 1994
I				II				b. 1996
					I			c. 1998

H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			75	.25	40. What term does not refer to the number of times a wave repeats itself, or how many times it completes a cycle, per second?	62.5	.375
II				II	I			a. amplitude
								b. frequency
	II				I			c. hertz
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			0	0	41. What "rate" is referred to in the following description: At low speeds (up to 2400 bps), 1 bit is transmitted with each signal change, and so the rate and bps are identical. At higher speeds (over 2,400 bps), more than 1 bit may be transmitted with each signal change, and so the bps will be greater than the rate.	25	.25
II								a. baud
	II			I	I			b. frequency
				I	I			c. transmission
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	0	42. The difference between the highest and lowest frequencies is known as	75	0
II	II			I	I			a. bandwidth
				I				b. baud
					I			c. hertz

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			50	0	43. Which word does not apply to the word modem?	75	0
				I	I			a. demodulation
								b. modulation
II	II			I	I			c. remodulation
H	L	DL	DI	H	L	DL	DI		DL	DI
		75	0			75	0	44. In what type of transmission can data travel through a communications channel in only one direction?	75	0
	I				I			a. duplex
								b. half-duplex
II	I			II	I			c. simplex
H	L	DL	DI	H	L	DL	DI		DL	DI
		0	0			25	-.25	45. What type of software enables you to control a personal computer from another personal computer located thousands of miles away?	12.5	-.125
					I			a. remote control software
I	II							b. telecommunications software
I				II	I			c. telecommuting software

H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			25	.25	46. What is the name for a common set of procedures or rules that allows the different components of a telecommunications systems to communicate with one another?	62.5	.125
					II			a. algorithm
II	II			I				b. protocol
				I				c. transmission controls
H	L	DL	DI	H	L	DL	DI		DL	DI
		100	0			25	.25	47. TCP/IP is used to establish a connection with	62.5	.125
				I	II			a. a modem
II	II			I				b. the Internet
								c. the telecommunications satellite
H	L	DL	DI	H	L	DL	DI		DL	DI
		50	.50			75	.25	48. What type of transmission is being utilized if data is being transmitted a number of characters at a time?	62.5	.375
	II				I			a. asynchronous
								b. simplex
II				II	I			c. synchronous

CONCLUSIONS

Although this test had been given to the students at the culmination of three units of study, the results obviously show that it would be worth dividing the test into two parts. The questions had never been used before and were designed to follow good multiple-choice test strategy design; however, the material was simplistic and questions of a higher order thinking class were addressed with guided essay questions. One format that was completely new to the author was providing three distractors instead of four or more. Some research has shown that three distractors offer the most accurate exam interpretation but the author would have thought it was easier to get the correct answer. It would appear that, in terms of the absolute score, there was no difference between those questions offered with three distractors and later questions offered with four. However, the students perceived the test to be easier than other past tests that used more distractors! Additionally, because the final examination is set by external examiners, it was important to include the multi-tiered MC question that involved combinations of answers as this format is used on a regular basis.

Next year, the author shall offer the easier part of the test (Difficulty Level = 100% and Discrimination Index = 0) as a pretest for the students in the class as the majority of them would have studied computers in the past. As a pretest, the questions will give some focus to the unit. The final MC assessment will be a composition of the other questions (Difficulty Level = 25%-75% and Discrimination Index = -.25 - .50). Another option would be to design two tests. The more difficult test would be given to the higher level students and the second test would be given to the lower level students. Different grade boundaries would ensure that each test would be treated as individual performances. As these types of tests are not granted high percentages of total marks, there shouldn't be any difficulty with the marks.

The questions to be used for the first type would be: 1, 2, 3, 4, 7, 8, 9, 10, 11, 17, 18, 20, 23, 26, 33, 38, 42, 46, and 47. All other questions would be used in the second quiz.

The author would like, however, to investigate the possibility of putting all questions into a larger database of MC questions that would be used on-line and individual quizzes and/or tests would be randomly generated. As there is a MC component to their final paper, this would help provide them with the necessary practice to distinguish between objective and subjective examination questions.

ON-LINE ASSESSMENT RESOURCES

Although distance learning via the Internet and posting course information via the Internet has become very popular over the last several years, the issue of on-line assessment is not as clear cut for many reasons. First and foremost, many teachers are not aware of the on-line possibilities and, although there are many commercial e-learning providers, the costing information is difficult to obtain without making a financial obligation. The author was able to locate two comparative analysis reports:

(1) www.brandon-hall.com: Brandon Hall, a noted e-learning expert, has recently published a book, Authoring Tool Strategies, that offers practical information designed to help companies/individuals make the right choices in determining an authoring strategy and in selecting the right technology to meet their goals. Unfortunately, the book costs approximately $600.00!

(2) www.c2t2.ca/landonline/evalapps.asp: Offers a comparison of approximately 45 different e-learning solution programs and attempts to offer both user, hardware, and costing information for all. However, the site would be more useful if there was an additional component that allowed the user to compare side-by-side or offered a search facility to organize the information by cost, etc., depending on the needs of the company and/or individual.

Unfortunately, for many educators, the choice of materials will be limited by what one is physically able to accomplish by oneself. The author conducted several searches for material that might be helpful to teachers who would want to work more with on-line assessment techniques and was able to find several useful sites.

General Information.

http://www.awbl.com/id/theory.html#evaluation: Theory and assessment http://www.thejournal.com/magazine/vault/A2089.cfm: Working with student feedback to improve teaching in web based courses. http://school.discovery.com/schrockguide/assess.html: Using assessment rubrics http://www.distance-educator.com/portals/webcourse_eval.html: Web course evaluation http://fpg.uwaterloo.ca/WEBTEST/: Create an interactive test with various features. http://www.utm.edu/bd/other/tutor/index.html: An older site offering the facility to generate on-line testing environments. http://landau1.phys.virginia.edu/teaching/quiz/home.html: May be used to generate quizzes from text documents. http://www.oyston.com/ExamMail/home.html: A freeware program that offers multiple-choice format techniques. http://www.fln.vcu.edu/cgi/3.html: Offers JavaScript tutorials to help develop interactive sites. http://www.net-shopper.co.uk/creative/education/languages/Martin/hotpot/index.htm: The former Hot Potatoes suite of applications used to help create interactive exercises. http://www.attotron.com/pub/Quizmake.htm: Entering a question on this site will convert it to JavaScript which can then be pasted into your own files. http://sunil.umd.edu/files/mctest.zip: A zipped file that can be used to help generate multiple choice questions. http://www.le.ac.uk/castle/: This service, operating out of the United Kingdom, is free to educators and can be used for test generation. http://www.vlearning.com/cyberexam/index.html: CyberExam (commercial package) http://www.learningware.com/quizfatory/: Quiz Factory (commercial package) http://www.questionmark.com/: Question Mark (commercial package) http://www.barint.on.ca/!cschafer/exammail/wiz.html: Quiz Please (commercial package) http://biomedia.bio.purdue.edu/TestPilot/: Test Pilot (commercial package) http://impactx.com/webtest/over.html: WebQuiz Writer (commercial package) http://www.perseus.com: Perseus Solutions (commercial package) http://www.testserve.com/: Testserve (commercial package) http://www.microsoft.com/education/planning/online/score.asp: FrontPage 2000 Quizzing Engine that is a plug-in designed to create quizzes. Downloadable with instructions but no support is offered. http://search.officeupdate.microsoft.com/TemplateGallery/ct149.asp: MS Office Templates that might be used to facilitate question development.

SITES INVESTIGATED FOR STUDENT SURVEY

Although the above information proved useful, the author thought that an added feature would be to take a look at some current sites that employ some type of on-line assessment and then ask the students what they prefer. One of the problems with on-line assessment is that there are no standards although some organizations, e.g. Click2Learn, are trying to establish standards that will at the same time both enhance and improve our use of on-line assessment.

Even a cursory glance at current educational sites reveals a myriad of different options when it comes to assessment. Most of the sites, however, use some type of objective testing while more open-ended types of questions are left to e-mail and/or suggested mark schemes. Within the multiple-choice environment, there are many possibilities involving immediate or delayed feedback, hints or no hints, graded or non-graded responses, etc. Interestingly, in some cases, the answers for the questions were identified through View à Source which, of course, would be discovered by the students very quickly!

In working with the students, the author decided to provide them with a couple of sites providing on-line assessment and then to ask them a series of questions. The following sites were used:

1. http://www.quia.com/jg/19638.html: The Quia site is used by teachers to generate on-line exercises including: matching, concentration, flashcards, and word searches. Personally, the author doesn’t care for this site because the quizzes and exercises are out of context and seem, frankly, useless. For example, why would anyone use a concentration format to identify new words and definitions when the user don’t know what the words are in the first place? As an educator, the author believes we must get away from these isolated exercises that are not linked to other parts of the unit in a cohesive fashion.

2. http://www.learnsign.com.au/: This site offers some lessons on learning sign language and then offers a little quiz; however, it is not clear what the content of the quiz will include and, as a result, a user may go to the quiz without having completed all the units above including the alphabet, common words, and numbers. In this quiz, there is no real total score, however, the user is allowed to choose another answer.

3.http://www.biols.susz.ac.uk/home/George_Mather/Linked%20Pages/Physiol/Quiz.html: Interestingly, this quiz offers no instructions other than "the answers can be found in the text"! However, further investigation will show that there are actual links to the answers in the text and so this type of testing would be used for review purposes only. Of course, finding all the answers in the text means that only certain learning skills are addressed as there are no analytic skills required.

4. http://web.cs.bgsu.edu/maner/xxicee/html/welcome.html: An interesting site about ethics. This site provides a frame-based setup with the presentation of different vignettes posing ethical dilemmas in the main frame. Each vignette includes hyperlinks to other sites and/or definition lists. A smaller frame poses a multiple choice format and the third frame is initiated by the user. That frame contains the breakdown of how various other people have answered the question e.g. males, females, etc. The only problem is that the numbers are supposed to reflect the responses of the last 50 people but the numbers don’t appear to be accurate. There is no total score provided although there are 20 questions asked. This site used client-side cookies to generate information as the author discovered through a look at its source code.

5. http://www.learnspanish.com: This site is an excellent learning site for beginning students, well designed, from an assessment point of view -- although not interesting to students from a graphical point of view as there is no interactivity. Basically, the site consists of book information used in the Internet medium. However, the quiz format, although multiple-choice, offers a practice session with an end result that indicates the wrong answers and provides clues as to the correct responses. The practice quiz is then re-marked and an appropriate message given. A test can also be generated and the end result is marked but not made known to the student. Instead, the result is e-mailed to the teacher who has already pre-registered. Although this service has a fee component that offers more information, the basics are free and can be used by both students and teachers who register.

Some of the source code for this particular file is included at the end of this document in Appendix A because it represents coding that can be easily understandable by those teachers who might want to adapt it to their own needs.

QUESTIONS ASKED ON STUDENT SURVEY

The following questions were chosen because the author felt that the emphasis for the project's on-line questions should be on the type of feedback preferred by the students as well as whether or not it would be important to offer second and third chances to answer the questions. The results clearly indicated that there is no "one" correct answer, for them, in terms of on-line assessment. In fact, several students mentioned, on more than one occasion, that it was important to know what was being tested in order to determine what type of assessment was most appropriate.

Is feedback important? Why or why not? 100% of the students believed that feedback was important. Interestingly, two of the students thought that feedback referred to whether or not the program would work! The author, therefore, didn't include their responses; however, did learn that, for the students, feedback means different things which we, obviously, hadn't covered in enough detail.

Should it be humorous? 67% of the students felt that feedback should not be humorous because they don't see quizzes and/or tests as something comical. The remaining 33% believed that a humorous reply would only be appropriate if the situation warranted. Those students also felt that adults would probably appreciate humorous replies less than children.

When should feedback be offered -- immediate or delayed? 66% of the students felt that the feedback should be immediate because that would be more appropriate for learning. On the other hand, 17% thought that offering immediate responses would be disruptive to the student's own learning because it would be distracting. Instead of concentrating on the appropriate answers for each question, the student would be focusing on "wrong" or "right" answers for past questions which would cause more stress for future questions. 17% also thought that there would be particular occasions whereby immediate feedback would be better and other situations where delayed feedback would be more beneficial.

Is it good to have a chance to correct your answers? Most of the students (60%) thought that they should have a chance to correct their answers but 40% did not want to because, again, they thought that would disrupt the "flow" of the quiz. Most of those who said "no second chance" believed that students would begin guessing more often if they could get other chances and so they did not believe their learning would improve.

Do you want to have hints? Split evenly, the students debated whether or not hints would be useful. Those who were most skeptical said that it depended on whether or not the hints were helpful. One student wrote, "Hints like 'No. Try again.' aren't helpful and shouldn't be used because they promote guessing but a real hint that offered more information about the question would be useful." Another student wrote that hints should not automatically be offered but that the user should be able to choose whether or not to receive a hint.

What's more important? Clear directions/Feedback? 50% of the students said that feedback was most important and 17% said the clear directions would be more important. As one student wrote, "Without clear directions, users don't know what to do … like on that quiz site." The student was referring to the Quia site which didn't list enough directions and the students usually ended up figuring out what to do through trial and error. Interestingly, 33% said that both were equally important and that clear directions without adequate feedback was just as bad as clear feedback without clear directions. The Quia site continued to be mentioned in their comments as did the "odd" numbers mentioned in the Ethics site.

WEB SITE INFORMATION

My general web site that includes a resumé, short interests section, and all the work done-to-date for Capella is located at: http://www.geocities.com/dlmanjack/index.htm or http://www.geocities.com/dlmanjack should work as well. Included in the Studies Section is the link to the course material for this paper as well as the web project that includes the questions. Although I tried to design a web-quiz myself, I just didn't understand enough about where to store the answers, etc. although I will be working on that during the summer. As a result, I posted my quiz onto another site (and hope it works!). The site is designed for educators and allows them to use the service for free as long as there are fewer than 100 questions held in that person's database. The general site is http://12teach.com and the reference site for the generated quiz is: http://12teach.com/Event0.asp?E=1375 which is referenced at the Geocities site. If you log onto the site to try the quiz, you will need to use a password. The author has set up 'capella' as the password, lowercase, and no quotation/speech marks needed.

SUMMARY

In this project, the author attempted to apply good MC question technique to the formation approximately 50 questions. Once the questions were completed, the author gave them to a class of students and another group of non-students to do an item analysis. After the item analysis was completed, it was that the test broke down into two separate components whereby one part of the test could be used for criterion referencing and the second part for norm-referencing. The suggestion was made that either the tests could be given in pre- and post-test environments or as two separate tests designed to be given to both higher and lower level students. After writing the report, the questions themselves were then posted to the web site and will be incorporated into the Information Technology in a Global Society web site at a later date.

APPENDIX A: SOURCE CODE FROM http://www.learnspanish.com

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">

<HTML>

<HEAD>

<TITLE>Spanish Grammar: gender of nouns part one</TITLE>

<style>

<!--

a{text-decoration:none;}

a:hover{color:#FF0000;}

a:hover{text-decoration:underline;}

//-->

</style>

</HEAD>

<TR>

 <BR>

<TR>

 <BR>

</TD>

<FONT FACE="HELVETICA,VERDANA,ARIAL" SIZE="1"><A HREF="../lessons/genoun1.htm"><FONT COLOR="WHITE"><B>Back to Lesson</B></FONT></A><BR>

learn spanish  <BR>

<A HREF="../pronunciation/index.htm" TARGET="_top" onMouseOver="window.status='Pronunciation Exercises'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Pronunciation</B></FONT></A><BR>

<A HREF="../tutorial.htm" TARGET="_top" onMouseOver="window.status='Grammar Tutorial'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Grammar</B></FONT></A><BR>

<A HREF="../vocab/index.htm" TARGET="_top" onMouseOver="window.status='Vocabulary Lessons'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Vocabulary</B></FONT></A><BR>

<A HREF="../verbs/index.htm" TARGET="_top" onMouseOver="window.status='Verb Drills'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Verb Drills</B></FONT></A><BR>

<A HREF="../travel/index.htm" TARGET="_top" onMouseOver="window.status='Spanish Travel Helper'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Travel Helper</B></FONT></A><BR>

<A HREF="../comps/index.htm" TARGET="_top" onMouseOver="window.status='Cultural Notes'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Cultural Notes</B></FONT></A><BR>

<A HREF="../audiocourse.htm" TARGET="_top" onMouseOver="window.status='Learn Spanish Audio CDs'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Audio CDs</B></FONT></A><BR>

<A HREF="../unltd.htm" TARGET="_top" onMouseOver="window.status='Premium Services'; return true;" onMouseOut="window.status=''; return true;"><FONT COLOR="WHITE"><B>Premium Services</B></FONT></A><BR>

study spanish  <BR>

<A HREF="../lessons/genoun1.htm"><FONT COLOR="WHITE"><B>Back to Lesson</B></FONT></A><BR>

</FONT>

</TD>

</TR>

</TABLE>

</TD>

<P><FONT FACE="HELVETICA,VERDANA,ARIAL" SIZE="4"><B>Quiz #1<BR>Gender of Nouns: Part I</B></FONT></P>

<P>

<B>A. Find the correct answer.</B>

</FONT>

</P>

<LI>Which word is masculine?

<P>

<INPUT TYPE="radio" NAME="1" VALUE="3">el libro<BR>

<INPUT TYPE="radio" NAME="1" VALUE="4">la ventana<BR>

</P>

<LI>Which word is feminine?

<P>

<INPUT TYPE="radio" NAME="2" VALUE="1">el chico<BR>

<INPUT TYPE="radio" NAME="2" VALUE="2">el hermano<BR>

<INPUT TYPE="radio" NAME="2" VALUE="3">el abuelo<BR>

<INPUT TYPE="radio" NAME="2" VALUE="4">la hermana<BR>

</P>

<LI>Which word is masculine?

<P>

<INPUT TYPE="radio" NAME="3" VALUE="1">el cuaderno<BR>

<INPUT TYPE="radio" NAME="3" VALUE="2">la pluma<BR>

<INPUT TYPE="radio" NAME="3" VALUE="3">la maestra<BR>

<INPUT TYPE="radio" NAME="3" VALUE="4">la profesora<BR>

</P>

<LI>Which word is feminine?

<P>

<INPUT TYPE="radio" NAME="4" VALUE="1">el número<BR>

<INPUT TYPE="radio" NAME="4" VALUE="2">el teléfono<BR>

<INPUT TYPE="radio" NAME="4" VALUE="3">el muchacho<BR>

</P>

<LI>¿Cuál es la palabra masculina?

<P>

<INPUT TYPE="radio" NAME="5" VALUE="1">la mañana<BR>

<INPUT TYPE="radio" NAME="5" VALUE="2">la palabra<BR>

<INPUT TYPE="radio" NAME="5" VALUE="3">el diccionario<BR>

<INPUT TYPE="radio" NAME="5" VALUE="4">la tarea<BR>

</P>

</OL>

<P>

<B>B. Choose between masculine or feminine.</B></FONT>

</P>

<LI>el libro

<P>

<INPUT TYPE="radio" NAME="6" VALUE="1">is a masculine word<BR>

<INPUT TYPE="radio" NAME="6" VALUE="2">is a feminine word<BR>

</P>

<LI>la pluma

<P>

<INPUT TYPE="radio" NAME="7" VALUE="1">is a masculine word<BR>

<INPUT TYPE="radio" NAME="7" VALUE="2">is a feminine word<BR>

</P>

<LI>el cuaderno

<P>

<INPUT TYPE="radio" NAME="8" VALUE="1">is a masculine word<BR>

<INPUT TYPE="radio" NAME="8" VALUE="2">is a feminine word<BR>

</P>

<LI>el escritorio

<P>

<INPUT TYPE="radio" NAME="9" VALUE="1">is a masculine word<BR>

<INPUT TYPE="radio" NAME="9" VALUE="2">es una palabra femenina<BR>

</P>

<LI>la silla

<P>

<INPUT TYPE="radio" NAME="10" VALUE="1">es una palabra masculina<BR>

<INPUT TYPE="radio" NAME="10" VALUE="2">es una palabra femenina<BR>

</P>

</OL>

<P>

<FONT FACE="HELVETICA,VERDANA,ARIAL" SIZE="3"><B>C. Select the correct letter.</B></FONT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<LI>el cuadern<SELECT Name="17" SIZE=1>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<LI>el vestid<SELECT Name="19" SIZE=1>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

<P>

<LI>la corbat<SELECT Name="20" SIZE=1>

<OPTION VALUE="1">o

<OPTION VALUE="2">a

</SELECT>

</P>

</OL>

<P>

</P>

</FORM>

<P>

<FONT FACE="HELVETICA,VERDANA,ARIAL" SIZE="3"><B>D. Suggested writing exercise: Write ten Spanish words.</B></FONT>

</P>

</BLOCKQUOTE>

</FONT>

</TD>

</TR>

<TR>

 <BR>

</TD>

</TR>

<TR>

<TR>

 <BR>

</TD>

</TR>

</TABLE><BR>

</TD>

Please report errors to webmaster!<BR>

<BR>

<A HREF="mailto:[email protected]">[email protected]</A><BR>

</FONT>

</BLOCKQUOTE>

</TD>

</TR>

</TABLE>

</BODY>

</HTML>

Hosted by www.Geocities.ws