Performance-based university research funding systems☆
Highlights
► Performance-based research funding systems (PRFS) are national systems of research output evaluation used to distribute research funding to universities. ► PRFSs have emerged in response to the knowledge economy, new public management and a desire for research excellence. ► PRFSs create powerful incentives within university systems less by redistributing funding than by creating a public competition for prestige. ► Effects on institutional autonomy are ambiguous, but under the right circumstances a PRFS will enhance control by professional elites. ► PRFSs will not increase equity or diversity, nor enhance economic relevance.
Introduction
The Research Assessment Exercise (RAE) in the UK was launched in 1986, since then many countries have followed suit and introduced performance-based research funding systems (PRFSs). At least fourteen such systems were found in 2010. Such widespread adoption represents a shift in national research policies of some significance since universities are so central in national innovation systems. Going forward, to understand the evolution of national innovation systems one must address the design and execution of PRFSs. To advance our understanding of research policy and its effect on national innovation systems, we must build an understanding of PRFSs in general. This necessitates stepping back from the intricacies of individual systems and taking a broad perspective on PRFS characteristics and the challenges they face as instruments of research policy.
PRFSs are highly intricate, dynamic and embedded in national research systems; therefore any general understanding will only be possible through the collective effort of scholars who understand each system in depth. As experience with PRFSs has grown, along with the salience of the exercises for funding and success, a rich tradition of critique and discussion has emerged, particularly in the UK and Australia. This literature is nation-specific and usually discipline-specific. Building on the national literature, Geuna and Martin (2003), Dolan (2007) and the European Commission (2010) took an international perspective offering comparative descriptions of various systems. This analytical review attempts to further this collective effort by identifying common themes emerging from in-depth analyses of individual PRFSs.
Note that understanding the literature on PRFSs requires acknowledging the dual identity of university research. On the one hand, university research is part of the larger enterprise of the university and is shaped by university governance and university-related policy making. On the other hand, university research is a substantial element of every national innovation system, and so is of concern to scholars of innovation and to governments seeking to enhance the innovativeness of their economies. These two perspectives on PRFSs differ somewhat. The higher education literature treats the research mission of universities in a somewhat sketchy fashion towards the end of documents that are mainly concerned with accreditation, completion rates, harmonisation, etc. The innovation literature tends to ignore the educational mission of universities and the changes under way in the allocation of the teaching component of university funding. Although this paper takes the innovation perspective for the most part, the analysis draws from the higher education literature a framework for understanding the introduction of PRFS as part of new public management, a movement to reshape government.
This overview of PRFSs begins by defining and identifying existing systems, even systems that have not yet been discussed in the English language literature. The rationales underpinning PRFSs will be explored from the scholarly perspectives of the innovation and higher education literatures as well as from the perspective of policy makers. A typology of PRFSs is developed and then common challenges are identified. Issues of autonomy and control reoccur regularly in PRFS discussion and these are discussed before final thoughts on the future of these systems are offered.1 I am guided throughout by the question: Are there common lessons for innovation policy to be drawn by stepping back from the details of individual systems to frame university performance based research funding systems more generally? The basis of the analysis is a literature review conducted in early 2010 that sought to identify all known PRFSs and draw out as much detail on their operation as possible in order to compare the rationale and design of PRFSs and their relation to funding systems.
Section snippets
Performance based research funding systems defined and listed
Research is evaluated in many places, by many types of organizations, for many different purposes. This heterogeneity can confound attempts to produce a coherent review. To avoid this, I begin by narrowly defining the PRFS and listing systems that meet or will meet the narrow definition. Systems included in this review met the following criteria:
- •
Research must be evaluated. Evaluations of the quality of degree programmes and teaching are excluded.
- •
Research evaluation must be ex post. Evaluations
Rationales
The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input. (Herbst, 2007, p. 90)
Although seemingly straightforward, the motivations behind PRFSs do contain a few nuances. Broadly speaking, different
Typology and timelines
Performance-based research funding systems vary in their unit of analysis, methods of measurement, frequency and census period. Individuals, research groups, departments and universities are all possible targets for evaluation. Research groups are the unit of evaluation with the best theoretical support because research is conducted by groups, not by individuals or departments. Departmental level PRFSs are routinely criticised because of this (see for example Herbst, 2007, p. 91). The recent
Challenges
PRFSs exhibit similarities in design because they face similar challenges. Peer review is held in high esteem as it is well accepted by the academic community in every country. However, being expensive and time consuming, it is used irregularly. As a result, funding decisions may be based on out-of-date information. Arguably departmental or field level PRFS using peer judgment based on indicators represents the state of the art. ERA is implementing this method.
University-level evaluation is
The impossibility of assessing costs and benefits
An analytical comparison of PRFSs would assess costs and benefits of these systems. At present, this is impossible for two reasons. First, costs are not discussed. Second, there are no standards for reporting the amount of university funding dependent on the PRFS.
The costliness of the PRFS should be a concern for any government. PRFSs incur indirect costs for universities that compile submissions and direct costs for the evaluation of those submissions. The RAE and VTR incur heavy costs in
Autonomy versus control
Autonomy, self-governance and competition are sensitive issues for the academic community and the introduction of a PRFS creates tensions between autonomy and control. British and Australian commentators on PRFSs are sensitive to the subtleties of rhetoric and reality in relation to university autonomy so this theme is prominent in English-language commentary (Marginson, 1997). Tapper and Salter argue that: “Ironically, such a model of governance may constrain higher education institutions more
Thoughts on the future
The long-term future of a PRFS will depend on how well it meets the government's goals. Many governments have articulated a clear goal for their PRFS: the enhancement of research excellence. A competition for prestige among universities based on research performance will likely achieve that goal. Problems may arise however if governments realise that their goals, or values, are broader. First, a PRFS will not be a good way to encourage interaction with industry and application of research,
Conclusions
Performance-based research funding systems (PRFS) are national systems of research output evaluation used to distribute research funding to universities. Fourteen such systems have been implemented as of 2010 in response to the knowledge economy, new public management and a universal desire for research excellence on the part of governments. This paper sought to step back from the intricacies of particular systems to draw out general lessons that can serve to enrich our understanding of how
Acknowledgements
The OECD Directorate for Science, Technology and Industry funded the research underpinning this paper. The comments, feedback and editing contributed by Sarah Box at OECD and an anonymous reviewer were most helpful in improving the paper. The first version of this paper was presented at the June 2010 OECD-Norway Workshop on Performance-based Funding for Public Research in Tertiary Education Institutions, Paris, France.
References (54)
Explaining Australia's increased share of ISI publications – the effects of a funding formula based on publication counts
Research Policy
(2003)- et al.
The evolution of research activity in Spain – The impact of the National Commission for the Evaluation of Research Activity (CNEAI)
Research Policy
(2003) The ratings game
Nature
(2010)ERA 2010 Submission Guidelines
(2009)- Australian Government Department of Education, Employment and Workplace Relations, 2009. Finance 2008, Financial...
University rankings smarten up
Nature
(2010)Impacts of performance-based research funding systems: a review of the concerns and the evidence
- Carr, K., 2011. Improvements to Excellence in Research for Australia. Press release, Senator the Hon Kim Carr, Minister...
Higher Education in Germany: Country Report
(2007)2004–2008 VQR Guidelines
(2011)
Pluralism at risk? Heterodox economic approaches and the evaluation of economic research in Italy
American Journal of Economics and Sociology
Using a bibliometric approach to support research policy making: the case of the Flemish BOF-key
Scientometrics
PSA Target Metrics 2006
Feasibility Study: the Evaluation and Benchmarking of Humanities Research in Europe. HERA D4.2.1
Assessing Europe's University-Based Research. EUR 24187 EN, Science in Society 2008 Capacities, 1.4.1
Research Evaluation: Methods, Practice, and Experience
The first Italian Research Assessment Exercise: a bibliometric perspective, Research Policy
Journal of Informetrics
Research assessment in Hong Kong
Higher Education
University research evaluation and funding: an international comparison
Minerva
Research selectivity, managerialism, and the academic labor process: the future of Nonmainstream Economics in U.K. Universities
Human Relations
Allocating research infrastructure grants in post-binary higher education systems: British and Australian approaches
Journal of Higher Education Policy and Management
Incentives and their dynamics in public sector performance management systems
Journal of Policy Analysis and Management
The four literatures of social science
Evolving regimes of multi-university research evaluation
Higher Education
Cited by (732)
Can the Locked-In Be Unlocked? University Stratification in China Under State-Led Quest for World-Class Universities
2024, Higher Education PolicyConstructing an effective evaluation system to identify doctors’ research capabilities
2024, Health Care ScienceMetrics in research impact assessment and grant funding: Insights from researchers in the “Reviewer 2 Must Be Stopped!” Facebook group
2024, Journal of Applied Learning and Teaching
- ☆
This article is based on “Overview of models of performance-based research funding systems” originally published by OECD in OECD (2010), Performance-based Funding of Public Research in Tertiary Education Institutions, OECD Publishing, http://dx.doi.org/10.1787/9789264094611-en.