Research assessment and recognized excellence: simple bibliometrics for more efficient academic research evaluations

Abstract

Many countries perform research assessment of universities, although the methods differ widely. Significant resources are invested in these exercises. Moving to a more mechanical, metrics-based system could therefore create very significant savings. We evaluate a set of simple, readily accessible metrics by comparing real Economics departments to three possible benchmarks of research excellence: a fictitious department composed exclusively of former Nobel Prize winners, actual world-leading departments, and reputation-based rankings of real departments. We examine two types of metrics: publications weighted by the quality of the outlet and citations received. The publication-based metric performs better at distinguishing the benchmarks if it requires at least four publications over a six year period and allows for a top rate for a very small set of elite reviews. Cumulative citations received over two six-year review periods appear to be somewhat more consistent with our three benchmarks than within-period citations, although within-period citations still distinguish quality. We propose a simple evaluation process relying on a composite index with a journal-based and a citations-based component. We also provide rough estimates of the cost: assuming that all fields of research would be amenable to a similar approach, we obtain a total cost of about £12M per review period.