Contribution for the IV European Evaluation Society Conference, Lausanne, Switzerland, 12-14 October, 2000, theme "Partnerships in public policy: implications for evaluation"

This study considers three variants of partnership in the European context involving an aspect of evaluation. The first is "social partnership" in the European Structural Funds programmes. The second prevails in development co-operation between the developing and the developed partners. The third variant derives from the concept of "subsidiarity", which contributes to making "partnership" a goal to achieve in relations between the national central government and self-governing sub-central governments in each Member State as well.

As a concept "partnership" is not immune to the risk that the ensuing evaluations involve simplistic and unsatisfactory background theories (Ahonen 1999, Gregory 2000). However, as such the concept of "partnership" is not superior nor inferior to other comparable concepts used in the context of European integration and elsewhere.

At its face value, the translation "partnership" is relatively easy between the leading EU languages – German, Partnerschaft, and French, parténariat. The beginning part- of a Latin origin is still present in the common English word "part", which makes "partnership" something in which several actors or units have at least some part. From the viewpoint of connotations, "partnership" has since earlier times been used as a business term referring to alliances between people or organisations joining each other to pursue shared commercial interests. "Partnership" continues to be a legal term of the English signifying a given type of co-operation between business actors.

In the study to follow it will be necessary to investigate what partnerships involve in a few different contexts, and what can be said of them on the basis of an analysis of concrete cases of programmes, their evaluation and their metaevaluation. It will be necessary to take into account as well the different meanings, nuances of meanings and uses of the concept of "partnership" in the different contexts. This should help reduce the burden of jargon and loose talk in regard to the concept of "partnership" on the one hand, and on the other hand help distil a reasonable core of that concepts and its application.

A "linguistic turn" or a "discursive turn" has belatedly affected evaluation (Fischer and Forester 1993, Fischer 1995), although the influence could have come long before (Ahonen 1983). Many an evaluation researcher and many an enlightened evaluator nowadays wishes to proceed as a non-positivist or post-positivist with respect to the complex phenomena encountered in evaluation, "partnership" included. The non- or post-positivist stance frequently involves that the researcher comes to regard knowledge in and on society, and no less society itself, to be always already constructed by the social actors themselves and for those actors as they come across with the phenomena on inlive them. Respective constructivist approaches have not remained alien in the other field of this study over and above evaluation, namely European studies (JEP 1999).

From the viewpoint of a moderate version of philosophical and theoretical constructivism , social phenomena involve the engagement of the social actors in so-called modality judgments in regard to competing representations of some conceivable "social reality" (Hodge & Kress 1988; Kress & v. Leeuwen 1996). The modalities to judge involve, for instance, plausibility, credibility, accuracy or factual status of such representations and within them.

Any determinate action by social subjects is thick of modality judgments. However, as opposed to the homogenous degree of being constructed as naïve, extreme constructivism would assume, different actions involve different degrees of judgmental thickness. Manual labour tends to involve a lesser thickness than the work of the handicraftsman. The work of the professional specialist tends to involve the highest thickness of all. Planning of social and economic development programmes and their implementation by the means of management, resource allocation and other steps involve a very high thickness of modality judgments. As an activity imposed upon such programmes and their implementation evaluation is just one thicket of such judgments. In this sense, evaluation involves the comparison of ex ante representations such as programme and project plans or proclamations of key political principles that ground social and economic action programmes with ex post representations of the programmes and projects (cf. Khakee 2000). The latter representations are rendered by the implementing actors or the beneficiaries or constructed by the evaluators themselves separately from the actual programme stakeholders or in co-operation with them. Even ex ante evaluations, common in the European context, can be seen as ex post representations despite that they evolve before the actions do.

From the point of view of moderate constructivism, evaluation is a genre made up of different subgenres – as revealed in the differentiation of evaluation by its background philosophies, theoretical approaches, techniques and target sectors. Seen from this perspective, evaluation involves a repository of interpretative models for evaluators to carry out the said comparisons between representations. Judgments that are conceivable in evaluation are not voluntaristic nor matters of sheer formalism but historically and structurally conditioned. They are, first, conditioned by evaluation’s and its each subtype’s history as a genre or sub-genre. Second, they are conditioned by evaluation’s particular social and political standing as a genre or sub-genre wherever it is being carried out. As a part of this, the judgments made in evaluation are conditioned as well by the particulars of the overall institutionalisation of evaluation and its particular institutionalisation in each incident of evaluation. Interestingly, very practical experiences in building what is known as evaluation capacity render support to these views on the conditions of evaluation (Boyle and Lemaire 1999). Without due attention to its proper conditions, evaluation is likely to fail.

For guidance among the complexities of the planning, implementation and evaluation of social and economic development programmes it is useful to build first a simple categorisation of types of development policies (Table 1). To nurture the empirical relevance of the categorisation in this study, it is useful to build in a reference to two types of programmes. Let us take into attention, first, those taking place within a single nation state between a federal or a central government on the one hand and on the other regional and local governments. Let us take into attention as well, second, those that involve a multilateral or bilateral donor country or donor organisation and a recipient such as a country or a regional international organisation. Indeed, the concept of a "development programme" is not more alien in "regional development policies" or "local (economic) development" often abbreviated as "LED" than in the field of national development policies and development co-operation in regard to developing countries.

In Table 1 four types of partnership ensue:

1. Centre- or donor-driven development without partnership
2. Centre- or donor-driven partial development partnership, that is, with a role but a secondary one for the recipient
3. Recipient-driven partial development partnership, that is, with a role but a secondary one for the centre or for the donor
4. Full development partnership between the two interacting partners

Table 1. Categorisation of Programmes of Development with Variable Degrees of Partnership.

Evaluation is in a certain sense a derivative activity. It is derivative with respect to the programmes and projects being evaluated; it is an activity as if imposed upon the latter frequently ex post but frequently ex nunc and, as the case many be in European contexts, ex ante as well. There are several options how programmes and projects on the one hand and evaluation on the other may be interrelated. With stimulation from Heron (1996) as elaborated by the author previously (Ahonen 1999a), it is possible to build upon the four-fold categorisation of programme types introduced above and extend it to cover programmes and their evaluation simultaneously. The expansion involves the dimension of evaluation categorised in an analogous way as the programme types were categorised (Table 2).

As is commonplace in cross-classification like this, the diagonal depicts combinations to be seen as congruous; in this case a given type of development programme predicts an analogous type of evaluation. Only two extreme combinations are deemed to be conceptually unfeasible or very unlikely (types 14 and 41). However, Table 2 is merely an expression of research hypotheses to be tested empirically, that is, there is no ex ante ruling out of those two combinations, either. Observing that the second categorisation indeed deals with the simultaneous aspect of partnership in programmes and evaluation, altogether six further of the total of sixteen combinations are seen as hard to conceive of. Although they may come into question it derives from research economy that they are not be considered here any further. Sixteen minus the four congruous types minus the two unfeasible types minus the six hard to conceive of types leaves four types worth a brief consideration:

Table 2. Cross-classification of types of development programmes and their evaluation.

Combination 21 might come into question, for instance, where the central actor or the donor on the one hand and on the other the recipient have been able to agree that the latter would play a certain active role in the evaluation unlike it has done in the programme planning, design and management. Combination 12 might come into question where the recipient has scant capacity to engage in a full partnership in evaluation.

Given the state-of-the-art of recipient-driven development programmes, combination 13 might come into question in several cases. Less than one might first imagine, such programmes need involve unilateral devolution of influence first wielded by the centre or the donor to the recipient but a move towards increasingly general types of state support or general development aid. Examples include general subsidisation by a federal or central government towards the fiscal capacity of lower level governments and general development aid to support the balance of payments and general budget aid. There are the sector-wide approaches ("SWAPs") as well, involving targeting of aid to given sectors without presupposing donor control of the ultimate project identification, planning and implementation. There is the further aspect that general forms of aid tend to be de facto conditional upon the recipient’s firm, credible and sustainable commitment to strict macro-economic policy conditions and solid planning and programming of its own to ensure that the general aid will not be wasted. Should the recipient not have the capacity to engage in an evaluation partnership or unwilling to do so, unilateral evaluation by the centre or the donor might well come into question.

Combination 43 must be seen as even more feasible than combination 13. The recipient-drivenness in a programme in no way precludes an equal partnership at the evaluation stage should the recipient have the capacity for that.

This study considers three variants of partnership in the European context related to evaluation. The study will certainly not cover all conceivable reasonable uses of the word "partnership" in European contexts. The sense in which "partnership" is understood in this study involves a certain convergence deriving from European law and its norms, rules and technical principles. An example of the many conceivable meanings of "partnership" not covered by this study consists of the public-private partnerships despite that they have been noted as well in a context of European law (Arrowsmith 2000; cf. OECD 1999; OECD 2000a). Neither does this study address to questions typical of "university-industry partnerships" (Hellström and Jacob 1999). There are certainly very many other types of European or other partnerships not covered by this study, either.

The first instance of partnership studied here is "social partnership" in the domain of the European Structural Funds programmes. The second instance prevails in development co-operation between the developing and the developed partners. The third instance derives from one of the key concepts of European Union policy making, namely "subsidiarity" and its implications. This concept might be found to make "partnership" a goal to achieve in relations between a national central government and self-governing sub-central governments in each Member State in an analogous way as the concept implies the same in the European regional development policies. The first and the second partnership cases studied involved evaluations, whereas the third involved a metaevaluation of previous evaluations.

The empirical research strategy of this study involves that the case studies are presented only by drawing out their background and outline. The overall findings in the two evaluations and the one metaevaluation are not systematically repeated as such because they are not the very topic of this study. However, the evaluation results do come forth in their way, because it is the evaluations including their evaluative conclusions that provide the information to have write the case studies in the first place. The section in this study following the cases will spell out the analysis proper made of this study in regard to the two evaluations and the one metaevaluation.

The definition of "partnership" in the context of European regional development policies and programmes pursued in the recent years has been covered by two regulations the latter of which has lately replaced the former. The former one, valid during the implementation of the programme and its evaluation analysed here, is the European Council of Ministers Regulation of 1988 on the European Regional Development Fund (ERDF; Regulation (EEC) No 2052/1988). The latter is the new overall Council Regulation of 1999 laying down the general provisions on the Structural Funds (1260/1999).

"Partnership" is defined in both the above regulations (the former, Art. 4(1), the latter, Art. 8(1)) as "establishing" or, in the new regulation, "drawing up" of the respective Community actions to complement or to contribute to the respective national operations in close consultation between given partners. Proceeding from the earlier to the later regulation, there is thus a trend from joint decision-making on the programmes to joint making of the programmes themselves. The partners mentioned in the regulations include the European Commission and the respective Member State together with bodies designated by the Member State. The other bodies include, as specified in the new regulation, in particular, regional and local authorities and other competent public authorities, and partners called "economic and social partners" as well.

It is both interesting and very relevant for this study that beside the programme activities themselves, the regulations have presupposed and continue to presuppose partnership in the evaluation of those activities. The reference to the inclusion of the evaluations under partnership principle was the regulation 4253/1988. However, the basic ERDF regulation (2052/1988, Art. 4(1)) included as well the ruling that "(t)he partnership shall cover the preparation, financing, monitoring and assessment of operations." The new Structural Funds regulation (1260/1999, Art. 8(2)) differs from its predecessor in replacing the word "assessment" with "evaluation". Although nowadays repealed by the said regulation 1260/1999, the 4253/1988 was still in force when the programme being evaluated was being implemented. The European Court of Auditors (ECA 1998), in its thorough assessment of the Structural Funds interventions in periods 1989-1993 and 1994-1999, fixed considerable attention to the realisation of the partnership principle in evaluations. The ECA pointed out, and the Commission in its replies to a considerable extent agreed, that the evaluations had not, at least at first, well fulfilled the partnership requirement. However, both the ECA and the Commission agreed that the evaluation partnerships had strengthened over the years.

As prescribed by the programme funding terms, the regional implementing authority of the programme at hand commissioned independent evaluators, in this case two, to evaluate the programme’s Finnish part. The team leader had wide experience of evaluation, whereas the other evaluator had first-hand experience in the promotion of handicrafts entrepreneurship. The evaluators interviewed the core programme personnel, crafts entrepreneurs, the regional authority and other important regional stakeholders, analysed the project documentation, investigated the material project achievements, and visited the project sites. The features of the programme and its evaluation stem largely from the evaluation report (Ahonen & af Ursin 1999).

What is in this study called a "programme" indeed was called so in the original project documentation despite the limited size of the funding and the limited outreach of the activities. The programme was carried out in three Member States under Art. 10 of the European Council of Ministers regulation 2083/93 on implementing the Council regulation on the European Regional Development Fund (ERDF; 2052/88, repealed in 1999). In the Call that the Commission issued in 1995, the ERDF objectives 1, 2, 5b and 6 were to have the forehand. In the Finnish region in question the objective 6 – the one regarding the sparsely populated areas of Sweden and Finland – was prominent, but the objective 5b on rural development played a role as well. In terms of the Art. 10, the programme can be seen as a pilot scheme with aspects of creating incentives for investment in enterprises and pooling experiences for co-operation in development between different Community regions. The programme in the particular Finnish region fulfilled as well the requirement of a border region – in this case between Finland, the other Member State of Sweden, the European Economic Area country of Norway, and Russia’s extreme northwest. In the particular Call the Commission had expressed three aims:

1. Creation of partnerships between public organisations, businesses and associations in culture-related projects of inter-regional economic co-operation for development
2. Exchange of experiences and expertise related to culture between regional and local authorities to enhance cohesion
3. Promotion and use of new information and communication technologies

The Commission accepted the programme plan in April, 1997, and the project activities ended in June, 1999. The consortium expressed the aim of applying for continuation funding. The programme had the following objectives:

1. To develop a model for the creation of economically viable craft centres to provide sustainable jobs in the field of craft heritage
2. To showcase and development the richness and diversity of local, regional and minority cultures, while emphasising the contribution of each to European culture and history
3. To strengthen economic and social cohesion
4. To promote the use of new information technologies and exchange of experience and know-how
5. To engage in local job creation

The funding of the Commission for the programme share carried out in the Finnish region was about one half. Almost all of the other half came from the regional authority, this mostly meaning funds allocated to that authority by the Finnish national government although for the regional authority to decide in an independent way.

The evaluation report (Ahonen & af Ursin 1999) pointed out that to reach its first objective, the programme had had a survey made on craft centres in Finland, supported an MA thesis on the export maturity of a craft company in the region, but not accomplished a best practice guide for craft centres. In regard to the second objective, the project had worked to strengthen two existing craft centres and, albeit indirectly, to get a third one going. The separable effect of the project on the centre creation was all in all negligible. Brochures, fairs, exhibitions, seminars and news coverage fell under the second objective as well.

To enhance economic and social cohesion had been difficult in the project. The region is large, about 100 000 square kilometres and with a maximum length of over 500 kilometres, and project activities had taken place in the Western rim of the region only and only in three locations. The local artisans were a diffuse group of people, many working only part-time. Computer training and several of the other activities aimed at the craftspeople had met with difficulties due to a low participation rate. Other activities under the third objective were minor.

The activities related to the information and the communication technologies and exchange of experiences had included the construction of a website that had stayed modest, a video conference and an effort to accomplish a common video covering all programme partners in the participating countries. The Finnish video remained modest. The project implementors had met in meetings unlike the crafts entrepreneurs themselves. Exchange of experiences had been enhanced by allocating funding for a published study on craft entrepreneurship in the target region and another Finnish region.

Job creation had been the most difficult objective to achieve. Depending on the basis of calculation, only three to ten per cent of the intended effect had been accomplished.

OECD (1996, 1998, 2000b) had given considerable attention to partnerships. An OECD definition is to an extent implicit but no less clear as to its intentions (OECD 1988, section II B):

Note that in OECD parlance "local authorities" often, as it evidently does here, refer to the national and the other governments of a developing country, not municipal governments as one might imagine. The OECD document then goes on to elaborating its concept of partnership by defining a set of joint responsibilities of the donors and the developing countries and the separate responsibilities of both partners.

In the context of European Union policies, "partnership" has appearances that are less specific than those in OECD parlance. Frequently the EU concept of "partnership" refers more than one type of phenomena even where the partnerships in question are limited to those between the Union and various third, that is, non-Union countries. For instance, the announcement of the French Presidency of the Union in the second half of 2000 (Council 2000) includes that the Presidency is seeking "strategic partnership" with Russia and "partnership" in reforming the rules of the EU programme directed in the Mediterranean region with the southern and the eastern Mediterranean countries.

The evaluation team for evaluating programming of Namibian-Finnish development co-operation was selected by the Finnish Ministry for Foreign Affairs through open tendering. The team was made up of four people, including one expert heading a think tank in Windhoek, Namibia. The terms of reference of the evaluation included five objectives:

1. To review critically the current goals and practices in the bilateral country programming system of the Finnish development co-operation
2. Within the extent possible, assess the efficiency, effectiveness, relevance and sustainability of the Namibian-Finnish development co-operation yet without engaging in impact evaluation proper nor engaging in detailed project evaluation
3. To assess the relevance and rationale of the present country programming system
4. To analyse the effects of the aid co-ordination mechanisms upon the Finnish country programming, such as the co-ordination by the UNDP and the European Commission’s representation in Namibia
5. To prepare a separate proposal for a more comprehensive analysis of the country programming system and make recommendations for the impact analysis of the country programmes.

The evaluation involved research of the existing documentation including archival studies in Helsinki, a field mission in September, 1999, and a seminar mission in January, 2000, to receive the first feedback on the very first draft version of the evaluation report. The report was published in June, 2000 (Ahonen et al. 2000).

Evaluation of development co-operation is quite a large and specialised field of expertise. One way of dividing the actual evaluation projects distinguishes evaluations of impact on the one hand, and on the other evaluations of procedures of the so-called programming of co-operation. Impact evaluations form the majority, and frequently evaluations of programming and impact coincide in evaluation projects. (Cox et al. 1997; Convay and Maxwell 1999.) In these terms the evaluation (Ahonen et al. 2000) was one of the proramming, with only implications for impact evaluation to be covered.

The evaluation took forth the fact that Finland is present a bilateral donor in about ten first-priority developing countries and in a number of other countries, over and above donating a large share of its overall aid through multi-lateral channels such as the United Nations system and multi-lateral European Union channels. Evaluation is an established function in the Finnish Ministry for Foreign Affairs, with a clear emphasis upon evaluating development co-operation. The first large Finnish country evaluation regarded Tanzania in the mid-1990s with a focus on impacts of the aid, the other was focused on the programming of Namibian-Finnish development co-operation (Ahonen et al. 2000), and a third will regard Egyptian-Finnish development co-operation from the viewpoints of both programming and impacts. Moreover, a sector evaluation of Finnish aid in the water and sanitation field will be carried out.

Finland keenly observes international decisions on development co-operation, being an active member in the UN and OECD, and now harmonises its development co-operation in regard to its responsibilities as an EU Member State as well. There have as well been features of an implicit division of development co-operation targets between the Nordic countries up to deciding in which regions in some countries such as Tanzania each donor will concentrate its activities. Namibia has since its independence – and even before, during the period that the international community prepared the national liberation organisation SWAPO to take over – been one of the constant main countries of Finland’s development co-operation. As a matter of fact, there has been Finnish missionary presence in the present-day Namibia for about 130 years, which makes that the interactions between the Finnish and the Namibian peoples really have a long history. The very history of Finland’s development co-operation with the independent Namibia is a matter of about one decade.

Development co-operation between Namibia and Finland is based, first, on the general diplomatic agreement between the two governments on the general procedures of the co-operation. In actual practice, second, there have been almost annual bilateral country negotiations to keep stock on the accomplishments of projects, to consider proposals for new projects and the revisions of the existing ones, and to consider the question of closing some of the projects. Third, bilateral agreements are concluded on particular projects and programmes. There has been a tendency of the co-operation to move from diverse projects towards more comprehensive programmes. Another trend has focused a noteworthy part of the Finnish funding towards activities carried out by the Namibian non-governmental organisations. It is relevant that the Finnish Lutheran missions in Namibia focus mostly upon health care, education and social welfare, and belong to recipients of Finnish government subsidies to Finnish NGOs pursuing development co-operation in Namibia. However, sources of the Finnish Lutheran Church are traditionally the leading sources.

The evaluation took forth the fact that Namibia is, together with its large neighbour South Africa, situated as if in a glasshouse of the international community. Despite that Namibia has considerably more features of a one-party state than its big neighbour, part of its press is completely free of government influence and critical of the government if needed. The international news media have no difficulties of knowing even small infringements upon human rights in Namibia if these are known within the country itself. Namibia is a so-called upper-medium income country among the world’s countries. Indeed, were the distribution of income and wealth even moderately equal in the country, destitution and poverty would not exist. However, those distributions in Namibia are probably the world’s most unequal. Together with the prevalence of HIV among up to one third of Namibians of the working age, the development prospects of the country are not all that bright. Should Namibia’s military involvement in helping countries of the same large African region, namely in the Democratic Republic of Congo and Angola, lead to imminent withdrawal of donors, Namibia would turn out to be increasingly worse off.

It has been known of Namibia, and it was confirmed in the evaluation as well that many national Namibian institutions have problems of frailty. In part this continues to derive from the heritage that Namibia resembled a province of South Africa until its independence. However, since independence features of a one-party state, the rise of an indigenous upper class to supplement the traditional white one and tendencies to corruption together with the practice of using public sector employment for social pacification have worked against rapid improvements in institutional capacity.

The evaluation confirmed findings recorded five years earlier in a published Swedish evaluation (Melber et al. 1994). The role of the National Planning Commission had not become clear nor had the Commission received the muscle to do its co-ordination job well. A large share of the aid continued to flow without its inclusion in the Namibian state budget. Many ministries continued to collude with long-time donors without a strong role for either the NPC nor the Namibian Ministry of Finance. The evaluation contested the view that the moderate share of external aid vis-à-vis Namibia’s GDP – some six per cent – would preclude aid dependency. The share of aid of public expenditure is about twice that high, and the share is considerably higher in the expenditures of the key social programmes. Receipt of aid tends to give the incentive to continue to receive aid.

Rather few people turned out to be actually involved in the programming of the Namibian-Finnish development co-operation. A small donor country cannot afford to allocate but part of the work of a single desk officer in the national Ministry for another small country, and a small donor country tends to have small embassies as well. Namibian government actors have to deal with many donors without a possibility to focus on any donor that does not constantly allocate very substantial sums to given fields of co-operation. The inclusion of NGOs both in Finland and in Namibia to the actual decision-making process on the co-operation had been hard despite that the NGOs of the both ends are important recipients of the Finnish governments funds for development co-operation pursued in Namibia.

According to the evaluation, programming of development co-operation between Namibia and Finland is not an activity based upon strict written guidelines and documents. The basic Finnish Ministry for Foreign Affairs guideline regards project and programme design, management and evaluation only (FMFA 1998). Aspect of written documentation exist and have existed during the years in the country programming, but, for instance, the contemporary practice, common in many donor countries, rather to confirm co-operation principles regarding large multi-country regions than single countries does not help very much in the case of single countries. There is that aspect as well that the nature of the programming remains to an extent a matter of contestation in the donor’s key Ministry. It is true that a confusion is likely given the fact that programming of development takes place in each developing country. Where challenges to program the co-operation with the donor arises, this may give rise to the question if the donor engages in programming what the target country should do instead.

The evaluation suggested that many donors have resembling problems. Co-ordination between donors is difficult. In Namibia UNPD, playing in principle the co-ordination role, finds the European Commission’s Namibian representation to engage in a resembling role as well. The donor countries are independent countries irrespective of their EU membership, and perfect division of labour is hard to achieve. It has been particularly hard to ensure that each donor has a workable exit strategy for the situation that the need for aid is relieved or the circumstances change to preclude further aid.

The core concept that regards relations between central and sub-central political actors in European law and the respective political theory is "subsidiarity". This concept has long roots. The concept has long roots (Bruchs 2000, but the contemporary source is the political theory of the Catholic Church is beyond contestation:

"Subsidiarity" is directly present in the German version. "Jedwede Gesellschaftstätigkeit ist ja ihrem Wesen und Begriff nach subsidiär; sie soll die Glieder des Sozialkörpers unterstützen, darf sie aber niemals zerschlagen oder aufsaugen." (Quadragesimo Anno 1931, Art. 80, in German translation.)

Schilling (1995) takes up the two different wordings of the important European integration principle of "subsidiarity". The crucial place in European law was originally the Art. 3b (2) in the Treaty on the European Communities ("ECT", replacing the previous EEC Treaty; the definition is now in Art. 5 ECT):

This definition concerns only relationship between the Community and the Member States. A different definition is given in the preamble of the Maastricht Treaty (Treaty on the European Union, "TEU"): "(D)ecisions are taken as closely as possible to the citizen". This definition is wider than the Art. 3b (2) ECT definition, which becomes part of the wider definition. The wider definition covers relationships prevailing within each Member State as well.

According to Schilling (1995), "subsidiarity" might mean no fewer than four things:

1. As a concept of European integration subsidiarity is, first, equal to the "subsidiarity norm" of European legislation.
2. In European integration subsidiarity is equal to the "subsidiarity rule" instead of a mere political subsidiarity principle in that it is applicable in an all-or-nothing fashion without necessarily having to yield to any superordinate princples.
3. Subsidarity is equal to a legal, technical "subsidiarity principle" with the implication that the Community – and mutatis mutandis, a national government as well – should act when and insofar as a decision taken closer to the citizen is ruled out.
4. Subsidiarity in the European integration context is not a mere "political subsidiarity principle", which would receive its content in each case through political wheeling and dealing.

In the context of the preparatory process leading to the Amsterdam Treaty of 1997 the Finnish government expressed as the viewpoint that "subsidiarity" is above all a political principle (IGC 1997). However, this is problematic given the nature of the principle as a norm, a rule and a technical principle. Even if the preamble of the TEU does not force the Finnish government to apply the principle to the extreme in the Finnish relations between the national government and the municipal self-governments, the principle is, first, more than a mere political principle open to full political contestation. Second, Finland’s implementation of European policies as a Member States since 1995 involves that she is forced to putting "subsidiarity" into effect within her national borders and between her national central state and her sub-national governments, the prime field being the Structural Funds policies. This involves both de jure and de facto influences of subsidiarity. A further contributing factor consists of the position of the Finnish regional authorities as municipal federations (kuntayhtymät) without a direct legitimation by the voters but controlled by their member municipalities. Third, long before Finland could in any way consider membership in the Union or its predecessor, the subsidiarity principle was present in Finnish legal discussion on relationships between the national central government on the one hand and on the other the local municipal self-governments. One of the key issues was the legal inalienability of the latter’s rights. (Holopainen 1969.) The winning standpoint was in accordance with subsidiarity.

"Partnership" is present in the same source of normative Catholic political theory as "subsidiarity": "…in the present condition of human society that, so far as is possible, the work-contract be somewhat modified by a partnership-contract" (Quadragesimo Anno 1931, Art. 65). The partners in question are the employers on the one hand and on the other the employees. Interestingly, these are the very same partners as the so-called socioeconomic partners in the European Structural Funds regulations. There, the Community actions shall complement or to contribute to the respective national operations in close consultation between given partners (Regulation (EEC) No 2052/1988 on the ERDF, Art 4(1)). It is worth mentioning that local economic partnerships, established in the name of local economic development with workplace creation, are widely proclaimed contemporary goals of the OECD (1999; 2000b).

The metaevaluation was based upon a contract concluded by the Ministry of the Interior with a single expert evaluator. The metaevaluation regarded two sets of evaluations of basic public service provision, carried out and published by the Finnish national government’s Provincial Administrations in 1997-8 and 1998-9, respectively. First, a metaevaluation regarded the evaluations and their reports published in 1998. An unpublished report was prepared in the spring of 1999. Because the next evaluation reports were becoming ready, the Ministry had the same evaluator metaevaluate them as well later the same spring. The joint metaevaluation report was published (Ahonen 1999).

The metaevaluation was based upon, first, an analysis of research literature and experiences of previous evaluations on similar themes. Second, the metaevaluation had a empirical grounding based upon a form that the evaluator used to spell explicitly out the results of his evaluations. The form was built on the basis of available standards to evaluate evaluations (e.g., AEA 1997) and other sources. The form covered, using a five-point assessment scale in each question. Questions 4-6 were separately answered regarding each of the sectors covered by each evaluation report being metaevaluated, but the other questions regarded only each evaluation and its report as a whole. There were about 220 items to assess each of the evaluation reports and the respective evaluations. The metaevaluation question groups were:

2. Questions of method in regard to the background, design and data of the evaluation
3. Questions of method in regard to the analysis methods applied in the evaluation
4. Questions in regard to the conclusions and the recommendations made in the evaluation
5. Questions regarding the reporting of the evaluation
6. Further general questions regarding the evaluation
7. A Summary on the strengths and weaknesses of the evaluation
8. A general numerical evaluation of the evaluation’s utility and usability
9. Final verbal conclusions
10. An ultimate numerical and verbal assessment of the evaluation as "exemplary", "good", "normal", "mediocre", or "weak", not following mechanically out the evaluations of the various earlier items

The metaevaluation took up that the evaluation function of the Finnish national government’s provincial administrations stems from two directions. First, before a fundamental reforms of administration at the provincial and regional levels to prepare Finland for European Union membership, the Provincial Administrations had played a noteworthy role in sub-national planning and therefore had analytic capacity even after they had largely lost that role. Second, the said reforms consolidated the previous twelve provinces and their administrations in the Finnish mainland – excluding the autonomous province of Åland-Ahvenanmaa – into mere five in the same as the tasks of the Provincial Administrations were generally reduced. To fulfil the European Union requirement important in the Structural Funds policy-making field, in particular, functions previously carried out by the Provincial Administrations were moved to the new regional authorities. The latter, in turn, were organised following the unique Finnish version of indirect municipal self-government, this meaning that the regional authorities are federations of their member municipalities. The Finnish Ministry of the Interior, pursuing the overall supervisory function over the Provincial Administrations, wished to enable them and their knowledgeable personnel to have meaningful functions over and above those staying in those administrations. Therefore the Ministry included evaluation of basic services as a further function that the Provincial Administrations should pursue. The evaluation function was confirmed in the result contracts concluded between the Ministry and the Provincial Administrations.

The first trial evaluations were carried out 1996, or at the stage that the merger of the previous provinces was still under way. The first evaluations proper by the new and generally large provinces were published in 1998 and 1999, respectively. The provinces were not generally supposed to approach the municipalities to collect information. The actual evaluation topics were determined in such a way that each ministry having a stake in the affairs taken care by the Provincial Administration made their suggestions and proposals. This derived from the fact that the Provincial Administrations are in reality under the multiple rule of the Ministry of the Interior pursuing overall supervision and each sector ministry that has functions taken care of by the Provincial Administrations in the latter’s special divisions. The guidance letters issued by the Ministry of Interior in 1997 and 1998 for the ensuing evaluations stressed the perspective of the citizen yet without – somewhat paradoxically – presupposing that the citizens themselves be asked on their views on the basic services. The evaluation topics turned out to become rather similar in the evaluations of 1997-8 and 1998-9, and even did the evaluation criteria of reachability, economy, impact and quality stay the same:

• In the large welfare sector of education only special topics, namely social rights aspects of adult education and the so-called youth workshops preventing drop-outing and increasing employability, were evaluated.
• In the other large welfare sector of social welfare certain special services were evaluated: the so-called special services (child protection measures, drug user and alcoholic rehabilitation, services for the disabled, and mental health services), sufficiency of social workers, and social and welfare services rendered in the homes of senior citizens. These were exactly the same services that the National Research and Development Centre for Welfare and Health (STAKES) had singled out in its report of 1997.
• The national government policies of promotion and regulation of transportation in the public interest in the province were likewise evaluated.
• The activities of the police in the region (the police being a function managed and provided by the national government) were another target of evaluation.
• Services of public safety (e.g., fire brigades whether public or private and voluntary, ambulances, precautions for catastrophes, technical screening of radiation and environmental dangers) were evaluated as well.
• Policies of combat against drug abuse were an inter-sector evaluation topic.
• Only in 1997-8, the utilisation of information and communication technologies in the supply of public services was evaluated. However, the topic was re-introduced in the evaluations of 1999-2000, not covered by the metaevaluation.
• Only in 1998-9, the organisation of pre-schooling of young children was evaluated.

Most of the services evaluated are provided by the municipalities in Finland. Some are provided by municipalities or by voluntary private citizens, the local fire brigades being a case of point. In some cases there is targeted government subsidisation and not only the general and sector subsidisation paid in the bulk of the basic services. The former targets include parts of the adult education and the youth workshops. The transportation services evaluated involved sectors under government regulation and sectors receiving government subsidies for servicing sparsely populated areas. The police is the only public service organised and funded entirely by the central government. It is noteworthy that health care, one of the large welfare sectors, was hardly covered by the original evaluations despite that evaluation activities are well-developed in that domain.

The evaluation reports to be metaevaluated were generally rather bulky and complex documents, annually five in number. The complexity was at its highest in the three provinces that had been created by merging three to four previous provinces into each of the new provinces.

As said, the evaluations were first based only on the result contracts between the Ministry of the Interior and the provincial administrations. An effort by the Ministry to make the evaluation a statutory function of the Provincial Administrations first failed in that the proposal prepared did not lead to new legislation being passed by Parliament. Finally, in the year 2000, such legislation indeed came into effect, amending the Act on the Provincial Administrations with a clause making the evaluation function a permanent part of the functions in those administrations.

The metaevaluation (Ahonen 1999) was indeed able to distinguish the altogether ten reports published by the Provincial Administrations in 1998-9 into different categories of quality. There, "normal" let alone a mixture of "normal" and "mediocre" was a sign of needs of improvement, and only one report each year – but not by the same Provincial Administration both years – was seen to reach the level of being a combination of "exemplary" and "good". In general, the cohesion in the quality of the reports and the respective evaluations clearly increased between the two volume sets of evaluation reports. The size and the complexity of the three large provinces, each merging 3-4 previous provinces seemed to be a factor precluding rapid reaching of high evaluation quality. The highest quality single evaluation passage was deemed to be the evaluation of the information and communication technology utilisation in public services by the Province of Oulu, in its Western parts knows as a high technology Mecca with operations by such leading technology firms as Nokia.

There was variation in the quality of the evaluations qua evaluations. The evaluations of the police services were generally monotonous with their emphasis on operational criteria of response timeliness, although wider criteria were making their approach in the 1999 reports. The evaluations of the public safety services included problems from the viewpoint of evaluation. There was a tendency in several of the provincial evaluations on this function to turn towards auditing and control of legality, with a focus on the success or the failure of the municipalities in the province’s region to have fulfilled the service criteria written in legislation. Necessary though such auditing and control is, the metaevaluator found the practice to jeopardise the possibilities of evaluation as an activity based upon, involving and strengthening partnerships.

The evaluations of the social welfare services and the two types of education services were generally of a reasonable quality. However, their authors seemed to have had difficulties in adding to what was already known in the 1997 report referred to. In a few cases the evaluation of the sufficiency of social workers suffered from an excessive stress of the social workers’ own point of view as a professional group.

The first case, the Structural Funds programme evaluation, hinted to problems of credibility in the original programme albeit one gone undetected from the financiers and other decision-makers before the programme start. This was at its clearest in regard to the workplace creation through the programme. The programme funding was in no way of the size that it could even in the best circumstances have created the tens of workplaces indicated in the original programme plan. However, the ultimate number of workplaces credibly created was not lesser than it should have been given the approximately known cost of workplace creation in the region in question. These are problems not badly known in evaluations of Structural Funds Programmes (ECA 1999).

The problems in the programme implementation – especially the frequently low participation rate of the ultimate beneficiaries themselves in many of the programme activities and the low utilisation rate of the some of the facilities and equipment – referred to a deeper credibility and plausibility problem. The programme in question was in effect two-bottomed. At its face value it involved discernible features of the programme type 3 in Table 1 of this study, that is, a recipient-driven partial development partnership. By far and large, this was in accordance with the Structural Funds regulations. However, within its region of implementation, and accounting for the relationship between the stakeholders represented in the programme governance vis-à-vis the ultimate beneficiaries, the programme was a centre-driven one. The representation of the ultimate beneficiaries by given organised stakeholders operating in the region was part of the standard statutory practice in Structural Fund policies, programmes and projects. However, this type of shallow representation was problematic from the viewpoint of reaching good programme results that the tie between the stakeholders and the beneficiaries was largely severed. Partnership there indeed was, but it failed to be a strong partnership that would have involved the ultimate beneficiaries. This is ultimately a message to the programme designers and those who bear the responsibility for the further development of the Structural Funds programmes. It should be separately investigated to what extent problems and aspects of the democratic deficit of the European Union prevail there.

In the second case with a challenge of a partnership between a developing country and a donor country the partnership was to a certain extent lacking in terms of feasibility, efficiency and effectiveness. This was so despite that the general trend in the co-operation between the two countries had gone from donor-driven albeit partial partnership with the developing country towards a full partnership and also features of recipient-drivenness in some programmes. That is, the trend had gone from partnership type 2 of Figure 1 towards types 4 and 3. In certain programmes and projects, in particular, partnership type 3 had been becoming increasingly prominent. This was especially so in programmes and projects with a community development or governance improvement aspect, and in programmes where non-governmental organisations were enhanced. The type 3 was the best prominent where the NGOs receiving support were operative in the fields of democracy, equality, human rights and good governance.

The limits to the overall width and breadth of the partnership derived from features of frail and in certain respects chronically frail governance and lacking institutional solidity and capacity in organisations of the developing country partner. This was so despite that there is no reason to doubt the best intentions of both the developing country and the donor country and despite that in very many developing countries comparable situations are considerably more serious. There were features of clientism as well in the sense of long-term close relationships between given ministries in the developing country and certain donors. It was possible to check that these relationships not rarely went beyond mere good, solid and sustainable partnerships, and instead involved features of perpetuation of co-operation even where more productive co-operation targets might have surfaced. There was long-term bypassing of the receiving country’s official public financial administration through special donor-recipient schemes, both by the donor country at stake and many other donors.

In the home country headquarters of the Ministry representing the donor there was some conceptual confusion about what the focus of the evaluation itself, "country programming", involved. There were certain tendencies to mix two views. The former was the defensible view that a donor country is well allowed to engage in programming its own policies with respect to its developing partner countries both in regard to the expansion and contraction of its country presence and its entry and exit in respect to the partner countries. The latter was the non-sustainable view that the country programming should not be pursued in the first place, because it would involve programming which the developing partner country should do itself. This gives rise to the question to what extent should each donor really be a partner with the entire developing world instead of just certain countries, especially as far as no reasonable and productive division of co-operation countries and sectors between donors seems to be feasible.

Finally, the evaluation was able to a certain extent to contest the received view that the moderate volume of external aid with respect to the recipient country’s GDP – some six per cent – would have precluded aid dependency. On the contrary, the expectation of aid continued to maintain features of aid dependency built upon the expectations of established partners in the developing country that they could continue to support their activities by the means of donor funding. This, further, reduced the viability of the partnership between the two respective governments.

The last case was unlike the others one of metaevaluation: certain evaluation policies and activities were the target of the evaluation. The metaevaluation implies that the potential partnership between the national government evaluating the provision of basic public services mostly rendered by municipalities was jeopardised by adverse tendencies. They involved at least in one of the sectors of public service provision engagement in criticism under the name of evaluation but with a de facto content of auditing and control of legality. The latter involves comparison of the municipalities service infrastructure to norms issued by the national government. More than perhaps realised by the national government, this was eating ground under workable evaluation partnerships in regard to the services in question and, by extension, any services. However, this was only one of the reasons for the prevailing state of affairs, surpassed in importance by the national government’s policy to save both the municipalities and the citizens from rendering novel information for the evaluations. In terms of the programme types of Table 2 the metaevaluation was mostly of the type 1, or centre-driven, as far as we can consider the original evaluations to have bee a programme in the specific sense of an "evaluation programme".

The centre-driven nature of the original evaluations being metaevaluated was mellowed down by an important feature of two large and well-established sectors of basic social services: health care and social welfare. There, partnerships up to a solid collusion based upon established welfare service professions turned out to have transcended the boundary line between the national central government and the municipal self-governments. This brought in features transcending the categorisation given in Table 1 and making it difficult to categorise the "evaluation programme" that was in reality carried out by the respective professionals.

The third case involved a further problematic feature as a programme in the sense of a programme of evaluation. In the documentation of the national government whose provincial administrations were carrying out the evaluations the principal motivation given for the evaluation effort regarding the basic services was the importance of those services to the citizens. However, to a first and not so minor astonishment of the evaluator, it turned out that evaluation did not involve any approach to the citizens citizens. This adds a further feature of centre-drivenness, tending the evaluation metaevaluated further towards evaluation type 1 of Table 1.

The evaluation of the Structural Funds programme in the first case turns out to be of the evaluation type 13 of Figure 2, that is, a centre- or donor-driven evaluation of a recipient-driven partial partnership. Although the Structural Funds regulations put the regional authority that is independent from the respective national government into the leading position in the programme implementation, the same regulations have a converse effect in evaluation. Given that the funds available for the regional authority largely stemmed from the Communities budget and the national government budget, the accountability emphasis of the evaluation was understandable as well. From the viewpoint of the ultimate beneficiaries – the handicrafts entrepreneurs in the region – the evaluation was of the type 11.

The second case, the one in the field of development co-operation, can be characterised as a congruous case. Equally as the overall partnership between the donor and the developing country was of the type 2 of Table 1, so was the evaluation of the type 22 of Table 2. That is, the evaluation involved a donor-driven albeit partial partnership in evaluation. The overall tone of the evaluation is critical towards the national central government of the recipient country, somewhat but technically critical towards the headquarters of the donor’s Ministry in the home country, and not critical at all towards non-governmental organisations in both countries. Regarding programmes targeted to NGO enhancement the evaluation received features of evaluation type 33, that is, recipient-driven evaluation of recipient-driven development programmes.

If in the third case the evaluation activity that was metaevaluated can be seen as an "evaluation programme", then the metaevaluation can be seen to represent two different types of evaluation in terms of Table 2. From the viewpoint of the citizens – and certain other stakeholders such as companies and local and regional NGOs – the metaevaluation was definitely a centre-driven one and therefore of the type 11. The same holds as to most fields of basic service provision where the municipalities are the providers. However, there are a few exceptions where the municipalities had been willing to provide fresh information had the central government’s provincial administration decided to ask for it contrary to the guidelines of the ministry supervising the evaluation. In regard to the provinces where that had happened the metaevaluation can be seen to be of type 12 as to its type as an evaluation. On one occasion the metaevaluation received features of the evaluation type 14, namely where a true evaluation partnership seemed to prevailbetween the national government’s given Provincial Administration and the bulk of the municipalities situated in its area. The minority of public services are rendered in Finland by local and regional branches of national government authorities en lieu of the municipal self-governments. The metaevaluation regarding those services comes close to the congruous type 22: centre-driven partial partnership in evaluation.

In general, a large part of success in the art of Structural Funds evaluation hinges upon the acknowledgement of the degree of organisation of the ultimate beneficiaries and the depth that the organised ones among the partners represent the said beneficiaries. The experiences recorded above have, first, their message for the art of programme planning and implementation. There, the likelihood of programme success increases where the organised partners in the roles they occupy in the actual programme implementation indeed are able to commit the ultimate beneficiaries to engage in the programme activities. Second, it may turn out in ex post evaluation that the ultimate beneficiaries have been so diffuse that the evaluators end up with difficulties in finding them to analyse their experiences in the first place. There, it is not ruled out that there have been faults in the original programme design including the role that the organised partners were assumed to bear. Perhaps the right partners were not singled out at the outset to represent the ultimate beneficiaries. The case not being so, the needs and demands among the beneficiaries may indeed have been too diffuse to allow their workable representation by any conceivable, organised programme partners.

The Structural Funds evaluation advises the science of evaluation not to turn its back to the venerable yet today somewhat marginalised knowledge of theoretical policy analysis (cf. Geva-May and Pal 1999). In policy analysis attention has been given to the role that the high vs. the low structuration of policy fields plays in predicting high vs. low engagement, commitment and participation rate of ultimate beneficiaries and a low vs. medium vs. high consequent success of programmes in accomplishing their objectives and goals. First, where the organised partners indeed keenly aggregate and articulate the interests of the ultimate beneficiaries both programme implementation and evaluation are likely to be at their easiest. Second, there may be instances of the mixed case where non-homogeneous degrees of organisation prevail in different parts of the total beneficiary population in that in some parts the organised partners indeed aggregate and articulate interests well, whereas in other parts this is hardly so. Third, both programme implementation and evaluation meet difficulties where the ultimate beneficiaries are as if atomistic and perhaps so heterogeneous as not to be well represented by organisations acting in their name and badly available as programme participants and sources of evaluation information.

The evaluation case on development co-operation suggests for the art of evaluation that the well-known question of aid conditionality in development co-operation is more complex than might seem. Explicitly, all members of the global donor community have made the promise of not making any of their aid conditional upon their own values, policies or interests. Unfortunately, this sounds too good to be quite true. At the face value, all members of the donor community are committed to doing their utmost to nurture the primary responsibility of the developing countries for their own development even where a substantial part of the funding derives from the donors. However, in the same the donors render support to the developing countries to ensure that the latter evolve as full members of the global market economy. However, there companies from the donor countries are the major players. In general, the donors work to maintain and impose the values generally accepted by a major part of the global community, including most of the developing countries. However, we must ask if each and every one of those values let alone each of their nuances leave the developing countries in the "driver’s seat" of their own development and the true "owners" of their development programmes.

The evaluation of development co-operation leads to the further, ethical question if all developing countries are treated equally in evaluations. For instance, are small, weak developing countries that are not hopelessly far from a virtuous cycle of development placed as if in a glass house in comparison to large, powerful or otherwise important developing countries?

Another question for the art of evaluation concerns the non-governmental organisations in developing countries. It is not ruled out that some of them would not exist without constant and substantial donor contributions. Let us assume that at least a few of the values promoted by the donor community let alone some of the nuances of those values are problematic in the sense that their universal validity can be put into question. How should the donor contributions to the maintenance of NGOs pursuing such problematic values and value nuances be evaluated?

For the science of evaluation the evaluation case of development co-operation suggests, not differently from the Structural Funds evaluation case, that the structuration of the supply vs. the demand sides of programme implementation and the respective sides of the evaluation effort are well worth a consideration. Part of this thematic has become an established dimension of research on and practice of development co-operation. Project and programme failures have continued to witness of the important contributing role that the recipient country’s institutions – its national and sub-national governments included – play in making aid effective and its effects sustainable. Where the government is frail, the development co-operation efforts are more in vain than where those institutions thrive. The latest large-scale and in the same very expensive experiment has been the Western support to creating a market economy in Russia in Western terms, driven by a Western belief that healthy markets evolve even without a solid support from strong public sector institutions in a transitional country setting (Wedel 1998).

The structuration of the supply vs. the demand of aid paid in development co-operation poses a serious and in the same a classical problem. The problem prevails between two actors the one of which more than the other is supposed to render help whereas the position of the other is more characterised to be that of a recipient. From the viewpoint of the philosophy of social science the question goes: to what extent and down to what depth may the actor acting in the role of the helper render support to the other partner without jeopardising its status as a fully entitled subject? Here, development co-operation may not have become during the decades as innocent as it might first seem. It is true that the thrust has changed from technical assistance rendered by high-earning expatriate technical experts who tried to create key technical infrastructures to "softer" methods such as capacity building reaching well beyond sheer training and training-of-trainers in the recipient country and its organisations. A conceivable object of comparison is therapy. At which point is the support-rendering actor manipulating and indoctrinating the receiving actor to adopt the former’s mindset and value set? In the same, the problematic leads us to critical questions of cultural relativism.

Finally, questions of equality between nation states and other actors in the international system, including the field of international development co-operation, deserves attention in the science of evaluation. Should evaluation start from the assumption that there are certain values that are absolute, whereas certain other values are such as to put in relation to the circumstances prevailing in a developing country? If so, how flexibly should values – and which values and their nuances – be applied for evaluation to preserve a minimum degree of consistency and morale?

The case suggests for the art of evaluation that centre-driven evaluation may on occasion be too cautious. Even if in interaction between a central national government and self-governing sub-national governments caution may be a virtue as such, caution precluding partnership in evaluation is a problem as well. Moreover, exclusion of citizens on the motivation of saving in data collection expenses and the citizen’s time is problematic given that the very definition of "basic services" refers to the importance of those services for the citizens themselves. In the case analysed the citizens were not represented by any organised actor, either. The analysis of the case hints to serious problems of distrust and suspicion between the two types of governments in question – the national government and its Provincial Administrations on the one hand, and on the other the self-governing municipalities. Given that both derive most or their revenue from taxes paid by the citizens, the latter have the right-in-principle to demand that the governments they funds engage in a workable partnership.

From the viewpoint of the art of evaluation, the metaevaluation brings forward good albeit limited experiences in two of the evaluation-pursuing provinces of the national government. Transcending and even transgressing the guidelines that precluded collection of novel information for the evaluation instead of relying upon existing data, those provinces reached good results. As a matter of fact, the single evaluation with the best outstanding quality, namely the evaluation of information and communication technologies in public service provision, was the highest quality evaluation of any of the sector evaluations carried out by any province. To a large extent this derives from the rich data collected directly from municipalities and their respective partnership in the evaluation effort.

Solutions to problems in evaluating basic services in Finland are appearing from a somewhat unexpected direction. The Finnish Evaluation Society (FES 2000) was founded in late 1999 after two full and two preliminary rounds of evaluation of the basic services had lately finished. It is important that the board involves a strong representation both from the national government and the municipal sector.

The last case being of a different order than the two first cases as a metaevaluation gives different messages for the science of evaluation than the two previous cases. Evaluating evaluation is a different pursuit than evaluating what most evaluations evaluate themselves. However, within limits there are similarities up to a justification of evaluating evaluations as programmes of a specific type namely "evaluation programmes". The last case sends a further message in regard to the field of evaluation research. First, we definitely need the bulk of evaluation research and practical evaluation that involve a research component or that strongly build upon research. This type of evaluation can be called "research in evaluation". However, second, we need research on the very activities of evaluation, their results, their preconditions and the use of their results as well. The latter type of research can be named "research on evaluation".

PERTTI AHONEN, DSocSc, MScEcon, is Professor of Public Management and Policy at the European Institute of Public Administration (EIPA), Maastricht, the Netherlands, and Professor of Public and Financial Management at the University of Tampere, Finland and an active professional evaluator. He is one of the founders of the Finnish Evaluation Society in 2000. Please address correspondence to EIPA, O.L. Vrouweplein 22, 6201 BE Maastricht, the Netherlands [email: [email protected] or [email protected]]