016007

Program Design and Evaluation

Subject Lecturer: Geoff Riordan

Paper 1- Program Evaluation

Plan a program evaluation for the given case study “Asia Pacific Bank”.

**********************************************************************

Background and purpose of the training program “quality customer service”

This evaluation program is planned upon the request by the Customer Service Manager and the Human Resource Manager in order to determine if the training program “Quality Customer Service” has changed the behaviour of customer service staff.

The evaluation have to make sure whether the trainees have achieved what they were expected to achieve, in terms of behaviour and result. It has to meet the demand of different sector, e.g. customer, customer service manager and human resource manager. Also it needs to determine whether the changes can last for long time. The findings can help improve the training and probably it forms the base of further training programs.

the perspective

With the consideration of the background of the training program, the training is raised because of the rising number and the widening range of complaints regarding the customer service that the bank is recently provided, the evaluation is devised from a practical perspective for the Customer Service Manager and the Human Resources Manager to determine if the training has changed the behaviour. The objectives of the training program as well as the evaluation are expected to be organization-focused, which is emphasized on the performance objectives rather than learning objectives. Thus the result of the evaluation can also be used as a reference to determine the return-on-investment (ROI).

Hence, several critical points of the evaluation are devised.

The program should really benefit the individual trainee and the organization, particularly in terms of the behaviour and result, i.e. whether trainees can apply their new learning on the job, and whether the training offer tangible result like raising the ROI for the bank according to Kirkpatrick. (Kirkpatrick 1975, cited in Athanasou 1995 & Chase 1997)
The program should result in the acquisition of new knowledge and skills, and should be transferred to the practical daily use in the workplace.
Such new knowledge and skills should be required by the organization immediately, and in the next few years.
The merit and worth of the program, other than the practical value of the immediate need of the organization, can benefit the organization in the long run. (Stake 1967, cited in James 1995)

The objects of evaluation

Objects	Relevant Questions
The program content.	Whether the program content is consistent with the learning objectives and overall intended outcome. Whether the level of the content matches with the level of the trainees. Whether the content is too much or too less.
The trainers.	Whether the trainers have sufficient professional knowledge in what they were offering in the training program. Whether the trainers can successfully communicate with the trainees, and give positive and relevant responds to the trainees. Whether the trainers can provide relevant and useful examples and other teaching material to help presenting.
The assessment model.	Whether the amount of assessment is sufficient or not enough. Whether the depth of assessment is suitable for the trainees or it is too difficult. Whether the assessment is consistent to the learning objectives and can help trainees understand the content better. Whether the assessment is corrected or given response by the trainers.
The trainees.	Whether trainees can participate into the program. Whether trainees are motivated. Whether trainees are really need to be trained. Whether trainees can learn what they were expected to learn.
The resources and equipment.	Whether the resources and equipments are enough. Whether use of these can help presentation. Whether use of these can help maintain the motivation of the trainees.
The learning activities.	Whether the activities can involve all trainees. Whether the activities can trainees discover the content deeper. Whether the activities were clearly instructed, brief and debrief. Whether the activities can help boosting knowledge and skills. Whether the activities can provide enough practicing chances for the trainees.
The modes of program delivery.	Whether it is proper to use the existing modes of program delivery. Whether it can provide maximum outcome. Whether it can help performance level.
The way the program was developed.	Whether the program was well organized in terms of the easiness and time management.
The support services trainees were provided with.	Whether the supporting services are available and reliable.
Learner’s transfer of new skills to their jobs.	Whether the training can provide opportunities to apply the new skills. Whether the trainee can really transfer the skills. Whether the trainee can maintain the changes for long time.

Data collection plan

In order to evaluate the objects, relevant information should be collected before, after and 2 to 3 months after the training program, that is to ensure whether the trainees can transfer the new skills in the workplace and to make sure the transfer can maintain consistently.

Information sources and techniques of data collection adapted from the illuminative evaluations by Parlett & Dearden (1977) and Kemmis (1980) (James 1995) includes observation, assessments, questions and records are edited as follow:

Critical Questions	Information Sources	Techniques
· Whether the trained staff has any positive behavioural changes? · Whether the customer services provided by the bank is satisfying?	· Customers (randomly selected) · Manager (observe the staff)	· Daily conversation · Observation
· Whether the trained staff has obtained the required knowledge and skills?	· Staff	· Knowledge tests & Skills tests
· Whether the trained staff has sufficient professional skills to have good performance in their job position? · Whether the trained staff can provide good service to customers? · Whether the customers are satisfied with the services?	· Manager (in-depth interview) · Customers (randomly selected interview, comprehensive questionnaires) · Staff (self-assessed questionnaires) · Trainers (interview)	· Interviews · Questionnaires
· Whether the complaints reduced? · Whether the range of complaints reduced?	· Complaint records	· Other materials

The collected data can be interpreted qualitatively and quantitatively. Qualitative interpretation should be backed up by the particular context. Quantitative interpretation should be done after the data converted into graphs and figures.

However, there may be some problems when collecting the data. First, it is obvious that information collected from conservation or observation could be highly subjective. And though the information sources, i.e. customers, are abundant, some of them might give a casual response before they really observe the staff thus may not be reliable.

Secondly, there would be an objective and bia-free standard for the knowledge tests and skill tests, however, it could hardly determine whether the staff can really transfer them in their job positions. Also, there are things like working ethnic could not be measured by simple tests but it does influence the individual behaviour.

Thirdly, in-depth interview could be time consuming so can just limit to the managers and trainers. However when there are too many trainees, it is difficult for both the manager and trainer to address the individual problems. It could help to obtain a broad image of the whole group of trainees, or help to identify some extreme good or bad case, but could not help to determine the performance level of every individual.

Fourthly, questionnaires could be easy and can help collecting massive information, however, the questionnaire design might limit the aspect of the thinking if there are too much rating and ranking. However, when open questions are given, people might like to skip the questions or provide personal comments which are not related to the customer service problem.

time schedule

Time Frame Items	Time Allocation
Specification	1 week in total
Plan	1 week in total
Staff Approval	1 week
Initial Interview	2-3 weeks
Prepare Questionnaire
Administrator Questioner
Analyse Results	1 weeks
Conduct Final Interviews	1 weeks
Write Report	2 weeks (including the analyze of the final interview)

conclusion

Though the merit and worth of the training program would be measured, the evaluation design is basically devised from the practical perspective. However, the effectiveness of the training program may also affected by some other factors, for examples, the dismiss rate, number of retired and newly employed, errors in data processing, evaluator bias. Such factors will have to be analyzed in the evaluation report and the whole evaluation program could be finished in approximately two months depending on the approving time and whether there will be modification during the program.

BIBLIOGRAPHY

Athanasou J. 1995, “Issues in the Evaluation of Adult Education”, Understanding Adult Education and Training, ed G Foley, pp. 75-88

Abernthy D. 1999, “Thinking Outside the Evaluation Box”, ASTD Trainers Toolkit: More Evaluation Instruments, American Society for Training and development, pp. 177-181

Chase N. 1997, “Raise your Training ROI”, Quality, vol. 36, no. 9, pp. 28-41

Bayley S. 2000, “Responding to Evaluation Findings: Handy Hints for Program Managers”, Evaluation News and Comment, Australian Evaluation Society, ACT

Yates, L, 1997, “Research methodology, education, and theoretical fashions: constructing a methodology course in an era of deconstruction”, Qualitative Studies in Education, vol 10, pp 487-498.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

HOME