Elsevier

Research Policy

Volume 41, Issue 2, March 2012, Pages 251-261
Research Policy

Performance-based university research funding systems

https://doi.org/10.1016/j.respol.2011.09.007Get rights and content

Abstract

The university research environment has been undergoing profound change in recent decades and performance-based research funding systems (PRFSs) are one of the many novelties introduced. This paper seeks to find general lessons in the accumulated experience with PRFSs that can serve to enrich our understanding of how research policy and innovation systems are evolving. The paper also links the PRFS experience with the public management literature, particularly new public management, and understanding of public sector performance evaluation systems. PRFSs were found to be complex, dynamic systems, balancing peer review and metrics, accommodating differences between fields, and involving lengthy consultation with the academic community and transparency in data and results. Although the importance of PRFSs seems based on their distribution of universities’ research funding, this is something of an illusion, and the literature agrees that it is the competition for prestige created by a PRSF that creates powerful incentives within university systems. The literature suggests that under the right circumstances a PRFS will enhance control by professional elites. PRFSs since they aim for excellence, may compromise other important values such as equity or diversity. They will not serve the goal of enhancing the economic relevance of research.

Highlights

► Performance-based research funding systems (PRFS) are national systems of research output evaluation used to distribute research funding to universities. ► PRFSs have emerged in response to the knowledge economy, new public management and a desire for research excellence. ► PRFSs create powerful incentives within university systems less by redistributing funding than by creating a public competition for prestige. ► Effects on institutional autonomy are ambiguous, but under the right circumstances a PRFS will enhance control by professional elites. ► PRFSs will not increase equity or diversity, nor enhance economic relevance.

Introduction

The Research Assessment Exercise (RAE) in the UK was launched in 1986, since then many countries have followed suit and introduced performance-based research funding systems (PRFSs). At least fourteen such systems were found in 2010. Such widespread adoption represents a shift in national research policies of some significance since universities are so central in national innovation systems. Going forward, to understand the evolution of national innovation systems one must address the design and execution of PRFSs. To advance our understanding of research policy and its effect on national innovation systems, we must build an understanding of PRFSs in general. This necessitates stepping back from the intricacies of individual systems and taking a broad perspective on PRFS characteristics and the challenges they face as instruments of research policy.

PRFSs are highly intricate, dynamic and embedded in national research systems; therefore any general understanding will only be possible through the collective effort of scholars who understand each system in depth. As experience with PRFSs has grown, along with the salience of the exercises for funding and success, a rich tradition of critique and discussion has emerged, particularly in the UK and Australia. This literature is nation-specific and usually discipline-specific. Building on the national literature, Geuna and Martin (2003), Dolan (2007) and the European Commission (2010) took an international perspective offering comparative descriptions of various systems. This analytical review attempts to further this collective effort by identifying common themes emerging from in-depth analyses of individual PRFSs.

Note that understanding the literature on PRFSs requires acknowledging the dual identity of university research. On the one hand, university research is part of the larger enterprise of the university and is shaped by university governance and university-related policy making. On the other hand, university research is a substantial element of every national innovation system, and so is of concern to scholars of innovation and to governments seeking to enhance the innovativeness of their economies. These two perspectives on PRFSs differ somewhat. The higher education literature treats the research mission of universities in a somewhat sketchy fashion towards the end of documents that are mainly concerned with accreditation, completion rates, harmonisation, etc. The innovation literature tends to ignore the educational mission of universities and the changes under way in the allocation of the teaching component of university funding. Although this paper takes the innovation perspective for the most part, the analysis draws from the higher education literature a framework for understanding the introduction of PRFS as part of new public management, a movement to reshape government.

This overview of PRFSs begins by defining and identifying existing systems, even systems that have not yet been discussed in the English language literature. The rationales underpinning PRFSs will be explored from the scholarly perspectives of the innovation and higher education literatures as well as from the perspective of policy makers. A typology of PRFSs is developed and then common challenges are identified. Issues of autonomy and control reoccur regularly in PRFS discussion and these are discussed before final thoughts on the future of these systems are offered.1 I am guided throughout by the question: Are there common lessons for innovation policy to be drawn by stepping back from the details of individual systems to frame university performance based research funding systems more generally? The basis of the analysis is a literature review conducted in early 2010 that sought to identify all known PRFSs and draw out as much detail on their operation as possible in order to compare the rationale and design of PRFSs and their relation to funding systems.

Section snippets

Performance based research funding systems defined and listed

Research is evaluated in many places, by many types of organizations, for many different purposes. This heterogeneity can confound attempts to produce a coherent review. To avoid this, I begin by narrowly defining the PRFS and listing systems that meet or will meet the narrow definition. Systems included in this review met the following criteria:

  • Research must be evaluated. Evaluations of the quality of degree programmes and teaching are excluded.

  • Research evaluation must be ex post. Evaluations

Rationales

The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input. (Herbst, 2007, p. 90)

Although seemingly straightforward, the motivations behind PRFSs do contain a few nuances. Broadly speaking, different

Typology and timelines

Performance-based research funding systems vary in their unit of analysis, methods of measurement, frequency and census period. Individuals, research groups, departments and universities are all possible targets for evaluation. Research groups are the unit of evaluation with the best theoretical support because research is conducted by groups, not by individuals or departments. Departmental level PRFSs are routinely criticised because of this (see for example Herbst, 2007, p. 91). The recent

Challenges

PRFSs exhibit similarities in design because they face similar challenges. Peer review is held in high esteem as it is well accepted by the academic community in every country. However, being expensive and time consuming, it is used irregularly. As a result, funding decisions may be based on out-of-date information. Arguably departmental or field level PRFS using peer judgment based on indicators represents the state of the art. ERA is implementing this method.

University-level evaluation is

The impossibility of assessing costs and benefits

An analytical comparison of PRFSs would assess costs and benefits of these systems. At present, this is impossible for two reasons. First, costs are not discussed. Second, there are no standards for reporting the amount of university funding dependent on the PRFS.

The costliness of the PRFS should be a concern for any government. PRFSs incur indirect costs for universities that compile submissions and direct costs for the evaluation of those submissions. The RAE and VTR incur heavy costs in

Autonomy versus control

Autonomy, self-governance and competition are sensitive issues for the academic community and the introduction of a PRFS creates tensions between autonomy and control. British and Australian commentators on PRFSs are sensitive to the subtleties of rhetoric and reality in relation to university autonomy so this theme is prominent in English-language commentary (Marginson, 1997). Tapper and Salter argue that: “Ironically, such a model of governance may constrain higher education institutions more

Thoughts on the future

The long-term future of a PRFS will depend on how well it meets the government's goals. Many governments have articulated a clear goal for their PRFS: the enhancement of research excellence. A competition for prestige among universities based on research performance will likely achieve that goal. Problems may arise however if governments realise that their goals, or values, are broader. First, a PRFS will not be a good way to encourage interaction with industry and application of research,

Conclusions

Performance-based research funding systems (PRFS) are national systems of research output evaluation used to distribute research funding to universities. Fourteen such systems have been implemented as of 2010 in response to the knowledge economy, new public management and a universal desire for research excellence on the part of governments. This paper sought to step back from the intricacies of particular systems to draw out general lessons that can serve to enrich our understanding of how

Acknowledgements

The OECD Directorate for Science, Technology and Industry funded the research underpinning this paper. The comments, feedback and editing contributed by Sarah Box at OECD and an anonymous reviewer were most helpful in improving the paper. The first version of this paper was presented at the June 2010 OECD-Norway Workshop on Performance-based Funding for Public Research in Tertiary Education Institutions, Paris, France.

References (54)

  • M. Corsi et al.

    Pluralism at risk? Heterodox economic approaches and the evaluation of economic research in Italy

    American Journal of Economics and Sociology

    (2010)
  • K. Debackere et al.

    Using a bibliometric approach to support research policy making: the case of the Flemish BOF-key

    Scientometrics

    (2004)
  • Department of Trade and Industry (DTI) and Office of Science and Innovation

    PSA Target Metrics 2006

    (2007)
  • C. Dolan

    Feasibility Study: the Evaluation and Benchmarking of Humanities Research in Europe. HERA D4.2.1

    (2007)
  • European Commission

    Assessing Europe's University-Based Research. EUR 24187 EN, Science in Society 2008 Capacities, 1.4.1

    (2010)
  • H. Fosse Hansen

    Research Evaluation: Methods, Practice, and Experience

    (2009)
  • M. Franceschet et al.

    The first Italian Research Assessment Exercise: a bibliometric perspective, Research Policy

    Journal of Informetrics

    (2009)
  • N.J. French et al.

    Research assessment in Hong Kong

    Higher Education

    (2001)
  • Frölich, N., 2008. The politics of steering by numbers: debating performance-based funding in Europe. Studier av...
  • A. Geuna et al.

    University research evaluation and funding: an international comparison

    Minerva

    (2003)
  • S. Harley et al.

    Research selectivity, managerialism, and the academic labor process: the future of Nonmainstream Economics in U.K. Universities

    Human Relations

    (1997)
  • G. Harman

    Allocating research infrastructure grants in post-binary higher education systems: British and Australian approaches

    Journal of Higher Education Policy and Management

    (2000)
  • C.J. Heinrich et al.

    Incentives and their dynamics in public sector performance management systems

    Journal of Policy Analysis and Management

    (2010)
  • M. Herbst
    (2007)
  • D. Hicks

    The four literatures of social science

  • D. Hicks

    Evolving regimes of multi-university research evaluation

    Higher Education

    (2008)
  • Hicks, D., Wang, J., 2009. Towards a Bibliometric Database for the Social Sciences and Humanities—A European Scoping...
  • Cited by (732)

    View all citing articles on Scopus

    This article is based on “Overview of models of performance-based research funding systems” originally published by OECD in OECD (2010), Performance-based Funding of Public Research in Tertiary Education Institutions, OECD Publishing, http://dx.doi.org/10.1787/9789264094611-en.

    View full text