When performing a national research assessment, some countries rely on
citation metrics whereas others, such as the UK, primarily use peer review. In
the influential Metric Tide report, a low agreement between metrics and peer
review in the UK Research Excellence Framework (REF) was found. However,
earlier studies observed much higher agreement between metrics and peer review
in the REF and argued in favour of using metrics. This shows that there is
considerable ambiguity in the discussion on agreement between metrics and peer
review. We provide clarity in this discussion by considering four important
points: (1) the level of aggregation of the analysis; (2) the use of either a
size-dependent or a size-independent perspective; (3) the suitability of
different measures of agreement; and (4) the uncertainty in peer review. In the
context of the REF, we argue that agreement between metrics and peer review
should be assessed at the institutional level rather than at the publication
level. Both a size-dependent and a size-independent perspective are relevant in
the REF. The interpretation of correlations may be problematic and as an
alternative we therefore use measures of agreement that are based on the
absolute or relative differences between metrics and peer review. To get an
idea of the uncertainty in peer review, we rely on a model to bootstrap peer
review outcomes. We conclude that particularly in Physics, Clinical Medicine,
and Public Health, metrics agree quite well with peer review and may offer an
alternative to peer review