Toward Robust Evaluations of Flood Inundation Predictions Using Remote Sensing Derived Benchmark Maps

Remote Sensing-derived Flood Inundation Maps (RS-FIM) are an attractive and commonly used source of evaluation benchmarks. Errors in model-predicted FIM (M-FIM) evaluation results due to biases in RS-FIM benchmarking are quantified by introducing a high-confidence benchmark FIM, which was manually delineated from ultra-resolution imagery, as Ground Truth. The evaluation results show considerable differences in M-FIM accuracy assessment when using lower-quality benchmarks. A RS-FIM enhancement (gap-filling) procedure is presented and its effect on FIM evaluation results is analyzed. The results show that the enhancement is insufficient for significantly improving the robustness of the evaluation. The impact of including/excluding Permanent Water Bodies (PWB) on FIM evaluation results is analyzed. The results show that including PWB in FIM evaluation can significantly inflate the model accuracy. A novel evaluation strategy is proposed and analyzed. The proposed evaluation strategy is based on excluding low-confidence grid cells and PWB from the M-FIM evaluation analysis. Low-confidence grid cells are those that were estimated to be flooded by the gap-filling procedure, but were not classified as such by the remote sensing analysis. The results show that the proposed evaluation strategy can dramatically improve the robustness of the evaluation, except when a considerable number of false positives exist in the RS-FIM. The analyses showcase the many challenges in FIM evaluation. We provide an in-depth discussion about the need for standards, user-centric evaluation, the use of secondary sources, and qualitative evaluation.