Evolving Scientific Software in Long-Running Observatories: Lessons from the TERENO Sensor Management Migration

Abstract:

Scientific software in geosciences often grows organically: initial solutions
are developed within small teams to meet immediate research needs, and over time
they evolve into critical infrastructure. While this organic growth can be
highly effective, it frequently leads to challenges in maintainability,
documentation, and reuse when systems are expected to support larger communities
or integrate with new platforms. In this contribution, we share lessons learned
from evolving the software infrastructure of the TERENO environmental observatories.

For more than a decade, TERENO relied on tightly coupled systems in which
observational data and sensor metadata were managed together. This data
infrastructure proved robust in daily operations but gradually accumulated
inconsistencies, implicit conventions, and project-specific extensions that were
insufficiently documented. As TERENO is now being integrated into the Earth &
Environment DataHub, these limitations became visible and required a systematic
rethinking of how sensor and measurement metadata are managed.

As part of the infrastructure redesign within the Earth & Environment DataHub
initiative, we adopted the Helmholtz Sensor Management System (SMS), an open,
community-driven software platform. To support the transition, we developed and
extended the Python tool ODM2SMS, which enables reproducible and configurable
migration of metadata from the legacy system into SMS. This process exposed
several common pitfalls in scientific software development: hidden assumptions
in data structures, incomplete documentation, and software that worked well for
its original developers but was hard to adapt for new use cases.

We addressed these challenges by applying a set of pragmatic good practices.
These included increasing modularity and configurability in ODM2SMS, explicitly
documenting previously implicit rules, and combining automated migration steps
with manual review where scientific context was required. A particularly
instructive example is the migration of complex lysimeter installations,
involving hundreds of interconnected devices. This case highlighted the
importance of clear abstractions, shared terminology, and close interaction
between users and developers.

Our contribution reflects on how community engagement, open development, and
incremental refactoring can improve long-lived scientific software without
disrupting ongoing research. We conclude by discussing transferable lessons for
researchers facing similar challenges: balancing rapid development with
sustainability, making software usable beyond its original context, and turning
legacy systems into maintainable, future-ready tools.