Under the heading "not everything that counts can be counted",
the League of European Research Universities has opened up the
discussion on the pros and cons of evaluating the fruits of
university research. Under auspices of Dr. Mary Philips (former
director of research planning at University College London), LERU
is currently working on a paper that will definitely stir up a
healthy debate.
Growth in research evaluation
Governments have by and large understood the need to invest in
research because it is a vital engine driving innovative,
knowledge-driven economies. It goes without saying that they - and
funders of research in general - want to evaluate the fruits of
their investment. But the growth in research evaluation regimes
risks to create (has already done so, one may argue) a sometimes
unhealthy obsession with monitoring and measuring research.
This potentially has some undesirable consequences: demands on
universities to produce excessive amounts of data straining finite
human and financial resources, unhelpful or conflicting duplication
of various assessment exercises, a short-sighted "bean counting"
culture and other practices that detract from rather than support
high quality research.
Return on investment
Of course universities themselves assess the research performed
within their walls for a variety of reasons. Along with governments
and research funders they want to gauge research output,
quality and impact, improve performance and maximise return on
investment.
Universities are also interested in research assessment as a way
to inform strategic planning and positioning of the university, to
invest in areas of research strengths or new directions, to expose
weaknesses, to identify and track individual accomplishments, to
recruit, retain or reward top performers, to find and foster
productive research collaborations, etc.
Evidence of usefulness
Even when it's clear who wants to assess research for what
purpose, further challenges stem from the fact that assessment can
be performed in different ways. Peer review (basically asking other
researchers to evaluate the research) is a wide-spread method, but
costly, time-consuming and open to subjectivity or bias.
Another method is to collect bibliometric data including number
of publications, citation frequency etc. While it is less costly,
the bibliometric approach also has drawbacks. Moreover, a new trend
is to show research impact, i.e. evidence that a piece of research
is in a broad sense useful to society. While impact has its place,
it should be understood that it is not the driving force of
research. Whatever the method, clearly there is a need for
sophisticated tools, but it is equally necessary to understand
their limits.
Central databases
What can be done in practice? Universities for example need to
have ample human expertise and sophisticated research assessment
tools suited to the task of assessing universities' research
strengths and weaknesses. They should maintain central databases
capable of producing fine-grained, accurate and up-to-date HR and
research data. They can also support the emerging practice of using
unique personal identifiers to avoid ambiguities about researchers'
correct names.
External agencies should avoid creating perverse incentives for
universities and researchers and should ensure consistency for
reliable comparisons locally and internationally. Above all, in
assessing university research they need to appreciate that research
often has a long term outlook rather than a concern with immediate
return on investment.
Sense and sensibility
Our main point is to call for a sensible approach to research
assessment. Governments, universities and others should "assess
assessment", carefully looking at what works in different research
environments and building on good practice where there is valid
evidence that the process leads to demonstrable improvements in
productivity and impact.
The paper will be presented during a breakfast launch event in
Brussels on 19 June 2012. More information at www.leru.org.