Towards a fair and inclusive next-generation of academic evaluation

Academia is in dire need of more openness and transparency, inclusivity and fairness (see Crameri 2023). While other branches of academia, such as publishing, are being lift onto a next level by the passionate early-career community, academic evaluation is lagging behind and stuck with its outdated focus on quantitative publication output.

Despite some whitewash, the ruling metric, the h-index, is still misused widely to rank us based upon numbers of papers published and number of received citations: Who published the most adequately-cited papers (only papers) wins permanent jobs, project funding, and awards (e.g., “publish or perish”). Something that is most easily achieved by an academic, who blends in, and works in, an established research entity and along an established line of research, and does not share methodologies. The impact is clear: huge competition between peers (often culminating in entire research groups that stop communicating with each another), self-centred (instead of community-centred) research focus, little attention to methodological correctness, and an disproportional brain-drain of valuable academics (see Crameri 2023).

Scientific evaluation built upon numeric metrics is advantageous: It’s time effective (saving precious research time), fair (directly comparable and less biased by subjective reviewer opinions), and does not require, and cannot be altered by, individual linguistic or other skills that are only tangential to actual research (unlike written CVs, for example). By taking a step back from the prejudiced search for a single-number metric, we realise that a multi-metric profile would serve us better: Our aim is to characterise, and not rank, academics.

A multi-metric profile, in contrast to traditional academic practices, has the potential to reshape academic incentives at large. When carefully designed, a numeric profile can prioritise the quality of research over its quantity and represent (and thereby foster) the openness of methodologies and tools through single, candid metrics. It offers a unique opportunity to promote academic diversity, encourage disruptive science, and enhance communication with the general public. Today, an evaluation based on a multi-metric profile allows for a nuanced assessment of research quality. This approach recognises pivotal contributions in method and tool development, teaching, and outreach, providing a comprehensive view of an academic's achievements. Instead of relying on a simplistic ranking, a multi-metric profile highlights individual strengths and weaknesses, facilitating the assembly of effective research teams. This not only improves the likelihood of research success but also enables supervisors and individual academics to leverage strengths and address areas for improvement.

ProAc 1.0.0 (www.fabiocrameri.ch/proac), the first and ready-to-be-used version of the academic profile, is geared towards making academic evaluation fairer and more time-effective, and science the best it can be: diverse, collaborative, disruptive. ProAc is neither perfect nor complete – it never will be. This is why it is designed for continuous improvements and adjustments. ProAc has been crafted truly independently of any traditional academic bounds, but with all your gain in mind. Despite this radical approach, I do hope for your expert feedback and support down along its exciting roadmap.

Crameri, Fabio (2023), Multi-metric academic profiling with ProAc (1.0.0), Zenodo, https://doi.org/10.5281/zenodo.4899015