Evaluation reports steer away from ‘automated’ UK research assessment – UKRI

The findings of three reports initiated as part of the Future Research Assessment Programme (FRAP) have been published today (12 December 2022).

Informing future decisions

Together the reports caution against moving to a fully metricised system for the next UK research assessment exercise.

The reports do suggest ways in which metrics, artificial intelligence (AI) or machine learning might be used to support or inform low-risk areas of the exercise, from allocating outputs to enabling post-assessment analysis.

The UK’s four higher education funding bodies, which jointly oversee the FRAP, commissioned the reports as part of their broader evidence gathering process. This included sector-wide consultation to inform decisions on the future of research assessment in the UK.

The three reports

Harnessing the metric tide: indicators, infrastructures and priorities for responsible research assessment in the UK

A review of the role of metrics in research management and assessment, led by Professor Stephen Curry, Dr Elizabeth Gadd and Professor James Wilsdon.

The report plots the development of the responsible research assessment agenda since the 2015 metric tide report.

It revisits the potential use of indicators in any future Research Excellence Framework (REF) exercise, proposing an increased use of ‘data for good’. It also considers opportunities to further support the roll-out of responsible research assessment practice across the UK higher education sector.

Responsible use of technology in research assessment

In early 2022, the UK’s four higher education funding bodies commissioned the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton.

They were asked to investigate possible approaches to using technologies such as AI and machine learning to support or streamline research assessment in future exercises.

Overall, the study finds that such technologies could only be employed in future exercises in relatively low-risk areas of the assessment, such as allocation of research outputs to reviewers ahead of assessment, or for analytical purposes.

REF outputs analysis: maximising the use of REF data

This report investigates how insights generated by the REF (or by future research assessment exercises) can be maximised to better understand the health of UK research.

A series of analytical ‘experiments’ explore the feasibility of more granular analysis of research assessment data to gain insight into disciplinary and interdisciplinary strengths.

The report finds that future exercises could incorporate some automation in post-assessment data analysis, but that such approaches should be complemented by peer review to validate any conclusions.

Supporting a healthy research system

Professor Dame Jessica Corner, Executive Chair at Research England, welcomed the reports’ findings and said:

Given the significant role of the REF for UK universities, it is imperative that any changes are built on sound evidence and close consultation with the sector.

The insights from today’s reports will be considered alongside the findings from our other FRAP workstreams to allow us to develop a system that supports our vision for a healthy, inclusive and impactful research system.

The assessment process

The findings of these reports will be considered by the four funding bodies, along with a broader suite of evaluation and consultation activities. The funding bodies will announce their initial decisions on the next assessment exercise in late spring 2023.

FRAP was initiated by the four UK higher education funding bodies:

• Research England
• Scottish Funding Council
• Higher Education Funding Council for Wales
• the Department for the Economy, Northern Ireland

The FRAP team is based at Research England.

FRAP aims to explore possible approaches to the assessment of UK higher education research performance. For more information on FRAP, please visit the Jisc FRAP web pages.