Colorisation of archival aerial imagery using deep learning

Archival imagery dating back to the mid-twentieth century holds information that pre-dates urban expansion and the worst impacts of climate change.  In this research, we examine deep learning colorisation methods applied to historical aerial images in Japan.  Specifically, we attempt to colorize monochrome images of river basins by applying the method of Neural Style Transfer (NST).    First, we created RGB orthomosaics (1m) for reaches of 3 Japanese rivers, the Kurobe, Ishikari, and Kinu rivers.  From the orthomosaics, we extract 60 thousand image tiles of `100 x100` pixels in order to train the CNN used in NST.  The Image tiles were classified into 6 classes: urban, river, forest, tree, grass, and paddy field.  Second, we use the VGG16 model pre-trained on ImageNet data in a transfer learning approach where we freeze a variable number of layers.  We fine-tuned the training epochs, learning rate, and frozen layers in VGG16 in order to derive the optimal CNN used in NST.  The fine tuning resulted in the F-measure accuracy of 0.961, 0.947, and 0.917 for the freeze layer in 7,11,15, respectively.  Third, we colorize monochrome aerial images by the NST with the retrained model weights.  Here used RGB images for 7 Japanese rivers and the corresponding grayscale versions to evaluate the present NST colorization performance.  The RMSE between the RGB and resultant colorized images showed the best performance with the model parameters of lower content layer (6), shallower freeze layer (7), and larger style/content weighting ratio (1.0 x10⁵).  The NST hyperparameter analysis indicated that the colorized images became rougher when the content layer selected deeper in the VGG model.  This is because the deeper the layer, the more features were extracted from the original image.  It was also confirmed that the Kurobe and Ishikari rivers indicated higher accuracy in colorisation.  It might come from the fact that the training dataset of the fine tuning was extracted from these river images.  Finally, we colorized historical monochrome images of Kurobe river with the best NST parameters, resulting in quality high enough compared with the RGB images.  The result indicated that the fine tuning of the NST model could achieve high performance to proceed further land cover classification in future research work.