MEASURING IMPACT OF CBR PROGRAMS
APPROPRIATE PROGRAMME MONITORING EVALUATION SYSTEMS
- C. Mahesh, Bangalore
(Presented during Handicap International's South Asian Regional Workshop, held in New Delhi, on 2nd Sep 2006)
01. ABSTRACT
In my presentation, I will be sharing my experience in developing and implementing a Programme Monitoring Evaluation System for measuring the impact of CBR programme.
Initially, the focus of my presentation will be on the current developments in CBR following the review of two decades of CBR, the CBR Matrix. Then I will be covering some of the theoretical aspects of Programme Monitoring, Evaluation and Indicators. Finally, I will cover one of the practical approaches of PME the ‘Individual Rehabilitation Plan’ that we have developed as a team to measure the impact of a CBR programme.
02. BACKGROUND
The World Health Organization (WHO), in collaboration with other United Nations
Organizations, has promoted community-based rehabilitation (CBR) for over two decades.
CBR is defined as ‘CBR is a strategy within general community development for rehabilitation, equalization of opportunities and social inclusion of all children and adults with disabilities.’
Community-based rehabilitation (CBR) was developed as a response, to the need, to reach people with disabilities in the community.
It is a well established fact that all communities are different in terrain, culture, their political systems, socio-economic conditions and many other factors. Therefore, there cannot be one model of CBR for the whole world. And as we are well aware, it cannot be the same even within the same country.
There are many models of CBR programmes, each one unique to the situation. This is the uniqueness of CBR and, at the same time, the challenge of CBR programmes.
Although, the definition, major objective, and principles of CBR have not changed since 1994, there has been an evolution of concepts in CBR.
CBR is implemented through the combined efforts of people with disabilities themselves, their families, organizations and communities, and the relevant governmental and non-governmental health, education, vocational, social and other services.
There are many models for addressing disability “the medical model, social model and rights based model”. Like stated earlier every CBR programme is unique: aimed at making the maximum impact on the lives of people with disabilities, especially the poor, what we need is an approach that is a combination of all the three models taking into account the social, economic, civil and political conditions.
It must be noted that CBR in today's world is being implemented against the backdrop of the globalization and privatization. It is also the time when the Millennium Development Goals for eradicating poverty are bearing an influence on development programmes and it is also the time when drafting the ˜International Convention on Protection and Promotion of the Rights and Dignity of Persons with Disabilities' by the UN has been completed.
03. THE MAJOR OBJECTIVES OF CBR ARE:
Source: CBR, Joint Position Paper, 2004, International Labour Office, World Health Organization and United Nations Educational, Scientific and Cultural Organization.
04. REVIEW OF TWO DECADES OF CBR
In May 2003, an International Consultation was organized to review the impact of 2 decades of ‘Community Based Rehabilitation’ and this event was held in Helsinki.
Some of the highlights of the review are -
CBR is promoted as a strategy for achieving the rights of people with disabilities within the context of their communities and societies. It is a strategy relevant to community development and to national strategies for poverty reduction. CBR should be an integral component of these national programmes and strategies.
The consensus that was arrived was that CBR has evolved from a model that was often perceived only as medical to a more comprehensive model, and that, CBR now has a major focus on human rights. As a result, the full participation of persons with disabilities at all levels of planning, implementation, monitoring and evaluation is all the more critical.
There is a general consensus on the need for a multi-sectoral approach where the Ministries of Social Welfare, Health, Education and Labour play an equally important role.
Some of the areas during the consultation that were identified as areas that need to be strengthened to make CBR programmes more effective are -
Following the review, the WHO in collaboration with various agencies is in the process of conducting consultations in developing the guidelines for Community Based Rehabilitation. The guidelines will be launched in 2007 and jointly published by ILO, UNESCO and WHO.
05. CBR MATRIX
One of the outcomes has been the development of the CBR Matix which illustrates the topic areas which can make up a CBR strategy.
GOAL: HUMAN RIGHTS ~ SOCIO-ECONOMIC DEVELOPMENT ~ POVERTY ALLEVIATION
The CBR Matrix consists of 5 components (Health/ Education/ Livelihoods/ Empowerment/ Social), each divided into 5 elements.
The components and elements are underpinned by a number of principles. These principles are in no way just theoretical or abstract but intended to be translated into tangible ways of working and should be observable in programme activities.
The principles mentioned below are overlapping, complimentary and inter-dependent – they cannot be separated one from the other.
Accordingly – The principles of a CBR programme are -
|
Participation |
Participation means the involvement of disabled people as active contributors to the programme from planning to policy making to implementation and evaluation, for the simple reason that they know best what they need |
|
Inclusion |
Inclusion means the removal of all kinds of barriers which block people with disabilities from access to the mainstream. Inclusion also means convergence. It means including all forms of impairment |
|
Sustainability |
First, the benefits of the programme must be lasting. This means an approach to poverty alleviation where the socio-economic gains last beyond the short-term and benefit not just the present but future generations too. |
|
Self-advocacy |
Self-advocacy means the central and consistent involvement of people with disabilities defining for themselves the goals and processes for poverty alleviation. |
|
GOAL: HUMAN RIGHTS ~ SOCIO-ECONOMIC DEVELOPMENT ~ POVERTY ALLEVIATION
|
||||
|
COMMUNITY BASED REHABILITATION (CBR)
|
||||
|
PRINCIPLES: PARTICIPATION ~ INCLUSION ~ SUSTAINIBILITY ~ SELF ADVOCACY
|
||||
HEALTH |
EDUCATION |
LIVELIHOODS |
EMPOWERMENT |
SOCIAL |
|
· PROMOTIVE · PREVENTIVE · CURATIVE · REHABILITATIVE · ASSISTIVE DEVICES
|
· EARLY CHILDHOOD DEVELOPMENT · NON-FORMAL · FORMAL SCHOOL · HIGHER · SPECIAL/ TRANSITORY
|
· SKILLS DEVELOPMENT · INCOME GENERATING ACTIVITIES · ACCESS TO FINANCIAL SERVICES · OPEN EMPLOYMENT · ECONOMIC CONTRIBUTION & SOCIAL PROTECTION
|
· SELF-HELP GROUPS · DISABLED PEOPLE'S ORGANIZATIONS · SOCIAL MOBILISATION · POLITICAL EMPOWERMENT · LANGUAGE & COMMUNICATION
|
· LEGAL PROTECTION · CULTURE & RELIGION · SPORTS & LEISURE · RELATIONSHIP MARRIAGE & FAMILY · PERSONAL ASSISTANCE
|
Source: http://www.who.int/disabilities/cbr/031105_CBR_Guidelines_2nd_meeting_report.pdf
06. PROGRAMME MONITORING EVALUATION SYSTEMS
Appropriate programme monitoring evaluation systems enable us to steer the CBR programme on course in achieving its goal.
And appropriate indicators are used to measure the success/ impact of the CBR programme.
We, as practitioners of CBR, need to show the benefits of the programme.
The emphasis has been on evidence based practice and development of appropriate indicators to measure success.
6.1 EVALUATION
Often people are doing informal evaluation and may not realize it. For example – as a member of the CBR team you realize that children with disabilities not in going to school regularly.
Evaluation is not something that you do at the end of the programme or when the funding is finished, it ought to be done throughout the intervention.
There is a direct link between planning and evaluation. If you do not have a clear idea of what you want to achieve through your activities, then you will not know if you succeeded.
The aim of Evaluation is to determine:
ð Relevance – Are the intervention objectives responding to the needs and priorities?
ð Effectiveness – Have the intervention objectives been achieved?
ð Efficiency – Have the available means been optimally exploited?
ð Impact – What wider effects have been caused by the intervention?
ð Sustainability – Will the positive effect of the interventions continue after the project period?
a. Finding out the value of the programme:
One of the common question asked is “How are we making a difference in the lives of persons with disabilities?”. Some of the other questions that tell you the value of the programme are “How have we changed the awareness of disability in the general community?”
b. Making decisions to improve the programme:
One of the best ways to improve a programme is to review it. Even informal evaluations can identify weaknesses in the programme. When problems appear decisions about new approaches can be made. For example – observing that loans availed for income generation programmes by the members of the self-help groups of persons with disabilities and their families are not making much difference in the lives of children / persons with disabilities. Following, which building the capacity of the CBR facilitators in addressing the gaps is also part of evaluation.
c. Funding
Not only is evaluation important to staff, participants and community members, it is also important to get programme funding. Donors are very hesitant to give money or time to a programme that has not been reviewed or evaluated. Funding agencies look for some type of evaluation of the current programme or a feasibility study to prove the need before considering a proposal for funding.
d. Finding new knowledge of CBR
Evaluation studies reveal how CBR programmes work and help participants understand which strategies work where. New information of CBR that is gained could influence the spread of CBR in other communities and influence Governments.
6.1.3 Barriers to Evaluation
There are many reasons why evaluation is difficult:
It is important to recognize these barriers to evaluation and it is necessary to promote the idea of evaluation in the programme before the start.
(Source - Guidelines for Conducting, Monitoring and Self-Assessment of CBR programmes – Using Evaluation to improve programmes – WHO International Disability Consortium http://whqlibdoc.who.int/hq/1996/WHO_RHB_96.3.pdf)
6.2 MONITORING
6.2.1 Its purpose:
Monitoring is collecting simple and relevant information to keep people informed about what is happening in the programme. Monitoring is keeping track of CBR activities mainly through numbers or statistics and in addressing the quality of the intervention. For example number of persons with disabilities in the programmes, the number of people requiring assessments, gathering feedback from persons with disabilities and families about the relevance of the intervention.
Monitoring information helps CBR Manger and Facilitator to keep track of his or her own work and make simple day to day decisions about CBR activities. It is part of the daily/ weekly/ monthly routine and not something special that you do at the end of the year.
Gathering monitoring information and making relevant decisions based on it usually should involve person with disabilities, the care-giver and CBR Facilitators.
6.2.2 Indicators for monitoring
(a) What is an indicator?
It is a tool to check whether an activity has been carried out, an output produced or objective achieved.
Short-term outcome indicators: Utilization of services, quality of care, access to services, improvements in mobility, activities of daily living skills (ADLS), and so on.
Long-term outcome indicators: Change in community attitudes, change in self-esteem of disabled persons, change in economic status of disabled persons, participation in social relationship activities, participation in education and work, participation in civic and community life and so on.
(b) Why do we need indicators?
An indicator can only be formulated if the description of both the present situation (problem) and the future desired situation (Objective) is specific and clear. In addition the desired situation should be realistic.
Indicators are required for all levels of result based management, monitoring and evaluation systems. It offers an opportunity to make adjustments. Ultimately, it increases the likelihood of achieving the desired outcomes.
(c) Re-check
Verify whether each objectively verifiable indicator describes the Overall Objective, Project Purpose or Result accurately. If not, then another indicator or additional indicators should be defined.
(d) Types of Indicators
Quantitative Indicators
Quantitative indicators should be reported in terms of a specific number. Assessing the significance of an outcome typically requires data on both number and percentage.
Qualitative Indicators
Qualitative indicators provide insights into meaningful changes in institutional processes, attitudes, beliefs, motives and behaviour of individuals.
A qualitative indicator might measure perception such as the level of empowerment that the target community received to do their jobs.
Qualitative indicators might also include a description of behaviour, such as the level of mastery of newly learned skills.
Although there is a role for qualitative data, it is time consuming to collect, measure, and distill, especially in the early stages. Furthermore, qualitative indicators are harder to verify because they often involve subjective judgments about circumstances at a given time.
(e) Objectively Verifiable Indicators (OVI)
V Q T T P
Variable
Quantifiable
Target group
Time
Place
• Quantify objectives (50 families produce 10kg carrots)
• Set quality (50 families produce 10kg carrots of grade 1 carrots)
• Specify time (50…10kg…grade 1…by Dec 2006)
• Specify location (50…10kg…grade 1…by Dec 2006 in ‘X’ village)
(f) Tips to write good indicators
If we cannot find an indicator to measure the result of activities, it is of no use to undertake those activities.
(g) Means of Verification
(h) Building baseline information
There are eight key questions that should be asked in building baseline information for every indicator. These questions continue to apply in subsequent efforts to measure the indicator.
(i) Worksheet for monitoring system - Means of verification
|
Name of record |
Purpose |
What Information |
Who collects |
Frequency |
For whom |
Analysis made |
Feedback and follow-up |
|
|
|
|
|
|
|
|
|
|
|
|||||||
(7) PRACTICAL EXAMPLES OF IMPLEMENTING PME SYSTEMS
IN A CBR PROGRAMME
The need for appropriate PME system is one of the most important components of the programme that needs to be well planned, executed and reviewed periodically. PME systems assist to ensure that the project is on track and that the initiatives leads to the attainment of the overall objectives/ goals within defined time, and in ensuring that the quality and processes involved are not compromised.
Let us review the steps usually involved at the beginning of a CBR programme. Based on certain criteria, the NGO identifies a project area, interacts with the members of the community and key decision makers and start the process of building rapport with the community. Preliminary investigations are carried out by few staff and local volunteers to analyse the local situation and in particular the status of persons with disabilities. The situational analysis report forms the basis for developing a project proposal. On approval by the donor agency, staff are recruited and oriented about the project and on the principles of CBR.
Following which, a general case study format is developed. The field staff and members from the community (who could include students from local colleges as volunteers) would be mobilised to identify persons with disabilities in the community.
(Display 01 – General Information Sheet)
The General Case-study form usually has the information related to the name, age, gender, guardian, address, disability type. Here, I would like to mention that to maintain standards – it is ideal to classify and record disability based on the different categories as defined by the local Government, this should be qualified by a brief (5 to 6 words) description of the present impairment / difficulty. The name of the person who records the data must be clearly mentioned.
Based on the data from the General Information sheet, the field staff responsible for the geographical area is trained to consolidated and develop tables on a number of persons with disabilities, break-up by age, gender, disability type. This will be basis for the CBR Manager/ coordinator to develop project level statistics.
(Display 02 – Consolidated Tables - CBR MasterList)
Here, I would like to mention that, even before this preliminary identification, one of the most important tasks for the CBR Manager and other Senior Members of the organization is to plan out the entire data gathering operation. From the way the identification process will be carried out to the way the data will be recorded, stored, retrieved. This is the phase of the programme where a number of errors could creep in as the team members are new and the enthusiasm level will be very high to complete the exercise within a specified time period. Even before the Manager/ Coordinator realizes, the team members could evolve short-cuts for recording/ storing the sheets that would affect the programme in the long run especially during the evaluation process.
It is important to develop a detailed map of the project area that is in agreement with the field team. Geographical boundaries need to marked and agreed upon to clearly differentiate areas of the village/ slum.
In my experience, identification of persons with disabilities in the slums is easier as everyone – field staff, volunteers from local colleges and members of the community would like to play an active role and also there is a sense of satisfaction of doing something tangible. The environment during this time is very positive and the members are busy in gathering, consolidating and reporting. Here the Manager/ Coordinator play a crucial role in maintaining standards for quality and consistency of the collective output.
The quality of information that is being recorded in the General Information sheet is one of the most important pieces of evidence for the project that will always be referred to till the end of the programme.
Therefore, it is best to review the data gathered and conduct a quality check on the information collected at the end of each week. Ideally the Coordinator/ Manager have to accompany the field worker in cross-verifying a sample. If this action is delayed – in has been observed that no-one is ready to take the ownership for any errors.
Some of the common difficulties during and after this process could be – field investigators are not trained well, lack of clarity during identification of persons with disabilities, lack of support from the Managers/ Coordinators in the field, difficulty in re-locating the person with disability in the community - the landmark to identify the house could be confusing. In a slum with all the houses packed in a small area, the houses could have more than one house number (given by the cable operator, water supply department or old number – and the persons collecting the data may not realize/ report this during data gathering), the volunteers who helped in identification are no longer there and or field staff who conducted the survey has left the organization.
Therefore in my experience the involvement of the few persons with disabilities and their families/ prominent/ active members/ from the local community during the process of identification and developing efficient monitoring systems to maintain quality during data collection is an important step.
It is important to file the individual general information sheet according to the geographical area with a unique reference number for each person with disability in box files.
It is important that the Coordinator/ Manager with the field team cross-verify the existence of the person with disability in the project area once every year - as some of the families could have migrated or the person with disability is no longer in the community due to various reasons – such has gone for higher studies etc..
A chart with the consolidated data must be prominently displayed in the CBR Field office and has to be changed every year.
(8) PROGRAMME
MONITORING EVALUATION -
Using Individual Rehabilitation Plan
In my subsequent presentation, amongst the different aspects of PME in a CBR programme, I will be focusing on my practical experience of using the Individual Rehabilitation Plan as a tool for PME.
Following the baseline identification, it is important that the CBR facilitator establishes a good rapport with the person with disability and the community before the next important step in developing an appropriate Individual Rehabilitation Plan.
One of the observations in a CBR programme is that - organizations report that they are working with x number of persons with disabilities but for practical reasons the field team would be actively working with (x-y) persons. In some instances the CBR programme might be working with only 40% of the total number of persons with disabilities identified or reported in the first place.
As time passes (end of year 2 or 3), there could be lack of focus in the CBR programme as a whole and in particular the intervention plans for the persons with disabilities would gradually become hazy. Different members of the rehabilitation team could develop their own set of priorities, the purpose of visiting the person with disability in the community would become a chore, and the members of the CBR programme would end up interacting separately with the persons with disabilities and their families all of which lead to confusing situations / discussions and interpretations. In this whole process the role of persons with disabilities could get diluted. There is a possibility that persons with disabilities and their families who need to be included in the first place would gradually be left out and reduced to the status of a passive beneficiary.
After many years of practice, trial and error, (Oct 2005) we have developed a format for the Individual Rehabilitation Plan that is based on the different components of the CBR Matrix – Health, Education, Livelihood, Empowerment and Social that was presented earlier.
(Display 03a - the IRP Format
Display 03b - IRP Annexure.xls
Display 03c – IRP Growth monitoring chart)
Based on the Individual Rehabilitation Plan – follow-up visits are recorded in the Follow-up Sheet.
(Display 03d – IRP Follow-up Sheet)
It must be mentioned here that a separate file for persons under the follow-up of the CBR programme is opened – the file usually has the following documents –
(Display 03a - the IRP Format)
There are 5 sections in the IRP which include –
The IRP is developed as a team involving the person with disability, his/ her guardian (in case of children), CBR Manger, CBR Facilitator and the rehabilitation specialists such as therapists.
The IRP becomes the basis for developing an appropriate PME system.
To define the goals it is important to understand and define the situation / need in clear terms. There may be differences in the way we write and what we think and what the Manager understands by reading the IRP. Hence, there is a need to develop a common understanding amongst the team. Using of common simple statements will help in consolidation, analyzing and also in coaching new members of the team.
As part of field testing the IRP, each CBR facilitator was asked to complete a few sample IRP’s which were analysed and the feedback was shared with the team. The following are some of the different responses that were identified for further clarifications / expansion -
On analyzing the IRP, the community facilitator is in a position to classify persons with disabilities requiring regular follow-up (meaning at least one interaction in a month), persons requiring a follow-up once every quarter and persons who are independent – meaning who would not require a follow-up on a regular basis.
Based on the experiences of the field team on analyzing the filled IRP we as a team were able to develop simple indicators for identifying the areas of intervention.
As a result the IRP as a PME tool helps the Manager and the team to develop a wholistic rehabilitation plan and ensure its implementation as per the plans. The efficient management of the IRP provides the direction for the CBR programme and helps in demystifying the interventions for all.
(Display 04 - IRP_May_Jun 2006 Plan.pdf)
We as a team decided to conduct the process of reviewing the IRP once every three months.
(Display 05 - Individual Rehabilitation Plan Review Format.doc)
(Display 06 - IRP Jan_Mar 06 Review Final.xls)
The following is the learning from the IRP process in the CBR programme:
Based on the review, we have developed the plans for the months of May-Jun-Jul 2006. The next review of the IRP was conducted in Aug 2006 and we hope to present the next consolidated IRP review by the middle of Sep 2006.
It has been realized that there needs-to-be some basic norms for a valid CBR programme. Evidence is needed that the particular CBR strategy being used is the most effective approach to enhance the quality of life for people with disabilities and their family members.
References:
C. Mahesh