 |
|||
Participatory Evaluation of Biotechnology Research Units
of the National Center for Genetic Engineering and Biotechnology
of Thailand Montri Chulavatnatol* Kenan Institute Asia, Queen Sirikit National Convention Center, Rachadapisek Road, Bangkok 10110, Thailand. * Team experts: Paiboon Kavinlertvatana, Boonserm Witayachumnankul, Naksitt Koowatanachai and Arnond Bunyaratavej |
Abstract
The Kenan Institute Asia (KIAsia) was appointed as an independent evaluation agency to conduct a mid-term evaluation of 5 research units of the National Center Genetic Engineering and Biotechnology (BIOTEC) of Thailand. Four of the research units were set up in universities as network research units by the BIOTEC to conduct specialized biotechnology research in 4 areas, namely plant genetic engineering, medical biotechnology, marine biotechnology and biochemical engineering and pilot plant R&D. The fifth unit was the Central Research Unit of BIOTEC. The objectives of the evaluation were three folds: (1) to assess the quality and the impact of the work done during the past three years; (2) to propose the future direction of each unit; and (3) to recommend the roles of BIOTEC in setting up and management of future research units. The evaluation team decided to use the Balanced Scorecard (BSC) to assess the relationship between the vision and the mission of each research unit and the 4 scorecards, namely “customers”, “finance”, “internal process”, and “learning and growth”. The participatory approach was applied involving the director of each unit but also researchers of the unit. This approach was used in order to encourage the sense of ownership among the personnel of each unit of the recommendations arisen from the evaluation. The participatory evaluation process recovered several key findings. Many units had unclear vision and mission statements. According to the BSC method, unit directors and researchers mostly had difficulty in identifying their “customers”. On “finance”, most unit directors dispensed the budget according to the research projects with little opportunity to maximize the benefit from the budget. On “internal process”, all units were best in doing research but weak in cooperation with the private sector and other users of the R&D outputs. On “learning and growth”, most units devoted too little effort and budget to develop young researchers. The weaknesses in the BSC confirmed that the management of the R&D in each unit was not systematic. The evaluation team thus recommended that the management of each unit reviewed the vision and mission statements and considered applying the BSC method in order to improve their R&D efficiency. It was further recommended that BIOTEC played a stronger role in coordinating the research areas among the units to enhance the synergy and productivity among them. Due to the participatory evaluation process, the researchers and the unit directors were willing to work together in the review of the vision and mission statements. The evaluation team further suggested that the directors of the research units should develop good communication skill and receive training in modern methods of management including the BSC method to improve their R&D management. |
I. Introduction As one of the 4 components of the PDCA cycle, evaluation checks on the planning and the implementation in order to recommend adjustments. This paper presents an experience in conducting an evaluation of 5 biotechnology research organizations in Thailand using participatory approach. The lessons learned should be of interest to research and development managers and policy makers. The paper will begin with a brief description of the organizations being evaluated. This will be followed by the methodology used and the key results. The paper will end with a discussion and conclusion on the lessons learned. Thailand has been actively engaged in the development of biotechnology and its applications in the fields of agriculture, health care and industry. To accelerate the advances in this important technology, Thailand established the National Center for Genetic Engineering and Biotechnology (BIOTEC) in 1983. The Center is now a part of the National Science and Technology Development Agency that is attached to the Ministry of Science and Technology. BIOTEC operates one Central Research Unit and 8 network research units that are located in several university campuses. Both the Central Research Units and the network units are to carry out 5 main tasks, namely development of specialized technologies, conducting targeted research, supporting technology transfer, development of human resources with specific expertise and providing technical services. Each unit has drawn up a 5-year plan (2000-2005). To undertake a mid-plan evaluation, BIOTEC contracted an independent and reliable organization, the Kenan Institute Asia (KIAsia) that has extensive experience in development promotion. The evaluation objectives of the 4-month assignment (September-December, 2003) were 3 folds: (1) to assess the quality and the impact of the work done during the past three years; (2) to propose the future direction of each unit; and (3) to recommend the roles of BIOTEC in setting up and management of future research units. The scope of the work covered both the research performance and the management performance of 5 research units. These units and their research areas are as follows: |
1. BIOTEC Central Research Unit: food
biotechnology, plant cell technology, protein/enzyme engineering,
bio-resources, and mycology. |
2. Marine Biotechnology Research Unit:
molecular biology, genetics and brood stock improvement
of tiger prawns, eco-toxicology, water system of aquaculture
system, and aquaculture nutrition. |
3. Medical Biotechnology Unit: molecular
biology of dengue virus, thalassemia and renal stone disease. |
4. Plant Genetic Engineering Unit:
transgenic plants, transgenic papaya resistant to papaya
ringspot virus, and genetically engineered ornamental plants. |
.5. Biochemical Engineering and Pilot
Plant Research and Development Unit: algal biotechnology,
sensor technology for industrial application, microbial
bioprocess development, waste utilization and biological
waste treatment, and bioinformatics. |
II. Approach, Framework and Methodology To complete the assignment, KIAsia first formed a team of technical experts who had extensive experience in R&D in their respective fields. The team was supported by competent secretarial assistants. The team set out the following approach, framework and methodology. The participatory approach: In addition to the conventional approach of an R&D evaluation by reviewing documentary records and interviewing research directors and researchers, the team decided to introduce participatory approach in this assignment. The rationale was to allow the development of ownership of the recommendations by the members of each research unit. We believed that if the members of the unit have taken part in shaping the recommendations based on the analysis of the evaluation results, the recommendations should be more realistic and practical. The members of the unit would be more willing to implement them because they had the sense of ownership of the recommendations that they helped construct. Balanced Scorecard (BSC): To assess the performance of an organization, the Balanced Scorecard (BSC) is now widely adopted. The evaluation team decided to use the Balanced Scorecard (BSC) to assess the relationship between the vision and the mission of each research unit and the 4 scorecards, namely “customers”, “finance”, “internal process”, and “learning and growth”. Since none of the research units being assessed used BSC in its performance measurement, there was no quantitative data available. So the team decided to use BSC as a “framework” in a simplified form for use in this evaluation. That means the qualitative information collected from the review of the documents, group discussion and questionnaires will be analyzed using key questions: |
| 1. Customers: how do customers see us? 2. Finance: how do we maximize the benefit using the available budget? 3. Internal process: what are we good at? 4. Learning and growth: how do we continue to add value? |
Methodology: The team
employed 4 methods in the evaluation of each unit, namely
document review, site-visit, focus group discussion and
questionnaires. BIOTEC provided the annual reports and relevant
background documents of each unit. The team paid a one-day
visit to each unit. The visit consisted of the morning briefing
by the director of the unit and a guided tour of the facilities.
In the afternoon, the team conducted a 2-hour focus group
discussion with the researchers. The discussion was structured
to discuss the vision, missions and the 4 cards of the BSC.
In addition, the group also did a SWOT analysis. |
III. Results To complete the objectives, the results of the evaluation were arranged into two main groups, namely the research performance and the research management. The followings are the main findings presented as aggregates of the 5 units. Results of the evaluation of research performance: The research units were able to carry out the tasks expected by BIOTEC. These are described briefly below: |
1. Technology development. Most successfully
developed and patented certain technologies. However, several
patents were about technologies used by researchers. Only
few technologies on wastewater treatment found industrial
users. |
2. Research projects. This task was
most effectively carried out. Several good scientific papers
were published in high-impact journals. The team found that
many papers were of moderate quality. |
3. Technology transfer. Most units
had low to medium levels of success. Most transferred their
technologies through training workshops. Some private firms
took selected relevant technologies and further commercialized
them but it was too early to show commercial success. GMO
plants met with a specific difficulty in the technology
transfer process. Because of the legal restriction on commercial
cultivation of GMO plants, the plant unit was unable to
commercialize their technologies. |
4. Human resource development. All
units provided training of graduate students but they were
not much different from many university departments. |
5. Technical services. All but the
medical biotechnology units carried out technical services
to customers from the governmental agencies, private firms
and communities |
The team found that the research did
well on all tasks except the last two tasks, namely graduate
training and technical services. |
Results of the evaluation of research
management: The most striking finding on the management
was the lack of systematic approach to the management of
the units. Each unit had vision and mission statements but
they received no attention of the director or researchers.
Most but the biochemical engineering unit had unclear vision
and/or mission statements. Using the BSC framework, the
team found that the units were unable to see the relationship
between the vision and mission and the 4 scorecards. In
addition, the followings were found about each card: |
1. Customers. When being asked about
their customers, most researchers were unsure of their answers.
Many considered granters as their customers but some included
technology users, private firms and public as customers.
However, most said that whoever their customers were, they
expected the units to be the sources of know-how and cheap
technologies. |
2. Finance. Since the budget was set
in the grants, there was no attempt to maximize the benefit
in using the budget. The medical unit was the exception
in that they took a bold decision to invest in locating
certain researchers in their partner laboratories abroad
in order to transfer frontier research. |
3. Internal process. All units were
clearly good at doing research. From their research, other
by-products such as technology development and graduate
training were obtained. All except the biochemical engineering
unit were weak in cooperating with private sector or users
of technology. So low level of technology transfer was achieved. |
4. Learning and growth. Most researchers
were young but were eager and energetic. The units put in
place a mentoring system in which senior researchers helped
guide young researchers in their research development. With
only few established researchers, the mentoring system was
not so effective. Few opportunities were available to the
young researchers to acquire new techniques or to keep up
with the rapid advances in many fields. |
| Recommendations: From the results, several recommendations were made. The main ones are as follows: |
1. Use of BSC was necessary to improve
the effectiveness and efficiency of the management of the
research units. |
2. The research units should work out
with BIOTEC to designate technology development by conducting
R&D as the core task. Technology transfer should be
a part of the technology development. Training of graduates
and technical services should be considered the non-core
tasks of the research units. This decision will sharpen
the focus of the units. |
3. To implement the recommendation
#2 above, the units must develop working relationship with
private sector or users of technology. They should change
their mind-set from “doing R&D to spin off technology”
to “developing desirable technology by doing only
necessary R&D”. |
4. To implement the recommendation
#1 above, the unit directors must improve their communication
skills and developed better human relationship skills. |
IV. Discussion While the evaluation team was able to make some key recommendations based on the results described above, the team also found that the researchers were eager to help implementing most of them. They already showed the ownership of these recommendations before the reports were written. This indicates that the participatory approach has been effective in building up the willingness among the researchers to put the recommendations into practice. This is an important value of the participatory approach. However, it is also important to note that the participatory approach can be easily made invalid if the director of a unit is strongly dominating and has a full control of the members of the unit in such a way that inhibiting them from speaking their views. This can be a serious limitation of the participatory approach. The use of the BSC framework in the evaluation is another aspect of this report that is unconventional. Normally, BSC is employed to help the management in identifying areas to be improved. The proper use of BSC requires a good investment of time and effort to collect quantitative data. Using certain computer software program should facilitate the data processing. In this case, there has been no quantitative data collected. The benefit of the use of BSC in this evaluation has been making the researchers aware of a useful tool in management. With the simplified BSC used in this case, the team has been able to make the researchers understand the concept of BSC. However, the simplified BSC lacks many details. Without quantitative data, some experts may argue that the simplified BSC may be of little value to the management. This possibility should be further studied. |
V. Conclusion The participatory approach to the evaluation of the biotechnology research units enabled a wider adoption of the recommendations arising from the evaluation. Having taken part in the development of the recommendation through active participation, the researchers were more willing to cooperate with the management of the research units in making changes and adjustments. A simplified BSC framework allowed the evaluation team to make key recommendations that would improve the management of the units. |
| VI. References: |
Guideline to Balanced Scorecard Performance Management
Methodology |
| Eldis: Participatory Monitoring and Evaluation. (www.Eldis-Participation Manuals.htm). |
National Center for Genetic Engineering
and Biotechnology. The Sixth Annual Meeting. Biotec Research
Units (2003). |
Reports on the Evaluation of Research Units of BIOTEC (2003). |
Thailand and Life Science. An Assessment
and Recommended Policies, Report to the Royal Thai Government,
Hogan & Hartson, L.L.P. (2003). |
The 9th Asia-Pacific S&T
Management Seminar, Tokyo, Japan, 17-19 March, 2004. |
Further details on my interest and responsibilities can
be contact: |