DC_OCEAN: An open-source algorithm for identification of duplicates in ocean databases

A high-quality hydrographic observational database is essential for ocean and climate studies and operational applications. Because there are numerous global and regional ocean databases, duplicate data continues to be an issue in data management, data processing and database merging, posing a challenge on effectively and accurately using oceanographic data to derive robust statistics and reliable data products. This study aims to provide an algorithm to identify the duplicates and assign labels to them. We propose first the definition of exact duplicates and possible duplicates; and second, an open-source and semi-automatic system (named DC_OCEAN) based on crude screening and target screening, which is followed by a manual expert check to review the identified duplicates to detect duplicate data and erroneous metadata. The robustness of the system is then evaluated with a subset of the World Ocean Database (WOD18) with over 600,000 in-situ temperature and salinity profiles. This system is an open-source Python package allowing users to effectively use the software. Users can customize their settings. The application result from the WOD18 subset also forms a benchmark dataset, which is available to support future studies on duplicate checking, metadata error identification, and machine learning applications. This duplicate checking system will be incorporated into the International Quality-controlled Ocean Database (IQuOD) data quality control system to guarantee the uniqueness of ocean observation data in this product.