Quality assessment of meta-analyses on soil organic carbon research

The number of meta-analyses published in the field of agriculture is continuously rising. As a consequence of this rising popularity, more and more publications refer to their synthesis work as a meta-analysis, despite applying less than rigorous methodologies. All this gives reason to assume that core criteria, necessary in producing meta-analyses, are not clear to many researchers. As a result, poor quality meta-analyses are published, which might report questionable conclusions and recommendations to policymakers and farmers. This study is therefore aiming to provide fellow soil and agricultural researchers with an easy-to-use set of criteria on how to produce high quality meta-analyses. Alongside, the incorporated scoring scheme supports researchers and policy makers in evaluating the quality of existing agricultural meta-analyses.

We analyzed 31 meta-analyses studying the effects of different management practices on SOC between the years 2005-2020. Moreover, the retrieved meta-analyses were structured according to eleven management categories which allowed us to analyze and assess the quality of the state-of-knowledge on these categories. We found that, although overall quality was rising, meta-analyses on SOC still do not reach sufficient quality and a maximum score may be reached only by the year 2032.

Especially for the reporting of literature search, application of standard metrics for effect size calculation, correct weighting, extraction of independent effect sizes and database presentation, major deficiencies were found. In some cases, the term “meta-analysis” is still falsely used to describe quantitative syntheses of any style. Only one out of 31 meta-analyses in the category “tillage” applied a rigorous meta-analytical methodology and received a high overall quality score.

We conclude that, in order for the scientific community to provide high quality synthesis work and to push forward the sustainable management of agricultural soils, we need to adapt rigorous methodologies of meta-analysis as quickly as possible.