Publishing BUGS: Insights from the Journal of Trial and Error

Addressing positive publication bias and clearing out the file drawer has been at the core of the Journal of Trial and Error since its conception. Publishing the trial-and-error components of science is advantageous in numerous ways, as already pointed out in the description of this panel: errors can lead to unexpected insights and warning others about dead ends can prevent wasted time and other resources. Besides those advantages, publishing negative and null results facilitates conducting robust meta-analyses. In addition, predictive machine learning models benefit from training on data from all types of research rather than just data from studies with positive, exciting results; already researchers are reporting that models trained on published data are overly optimistic.

Besides publishing negative and null results as well as methodological failures, the Journal of Trial and Error couples each published study with a reflection article. The purpose of these reflection articles is to have a philosopher, sociologist or domain expert reflect on what exactly went wrong. This helps contextualize the failure, helping to pinpoint the systematic factors at play as well as helping the authors and other scientists to draw lessons from the reported research struggles which can be applied to improve future research.

Publishing failure brings with it some practical challenges: convincing authors to submit manuscripts detailing their trial-and-error; instructing peer reviewers on how to conduct peer review for the types of articles; differentiating between interesting … and uninformative, sloppy science; and determining the best formats to publish various failure-related outcomes in. Authors are still hesitant to publish their research struggles due to reputational concerns and time constraints. In addition, authors often fear that peer reviewers will be more critical of articles describing research failures compared to articles reporting positive results. To counteract this (perceived) tendency of peer reviewers to be more critical of research without positive results, we provide specific instructions to peer reviewers to only assess the quality of the study without taking into account the outcome. This then also ensures that we only publish research that adheres to the standards of the field rather than sloppy science. Whether submitted research provides informative insights is assed by the editor-in-chief and the handling editor.

Finally, we are constantly evaluating and innovating the types of articles we publish. Various types of errors and failures benefit from differing ways of reporting. For example, recently we introduced serendipity anecdotes, a format where scientists can anecdotally describe instances serendipity which occurred during their research. This format allows researchers to focus on the conditions which allowed for the serendipitous discovery rather than the research itself.