Enhancing Accessibility to High-Resolution Satellite Imagery: A Novel Deep Learning-Based Super-Resolution Approach
Abstract
The increasing availability of open access in space remote sensing has democratized access to satellite imagery. However, high-resolution imagery remains limited to those with advanced space technology expertise. To address this limitation, this research paper introduces a novel approach for enhancing the quality of Sentinel-2 satellite images by leveraging deep learning techniques for super-resolution. This approach offers a comprehensive solution that significantly improves the spatial resolution (scaling factor 8), considering the volumetric constraints and spectral band dependencies inherent in satellite imagery. The proposed model harnesses the power of deep convolutional networks (CNN) and incorporates cutting-edge concepts such as Network In Network, end-to-end learning, multi-scale fusion, neural network optimization, acceleration, and filter transfer. In addition to the advanced model architecture, an efficient mosaicking technique is employed to further enhance the super-resolution of satellite images. The model also accounts for inter-spectral dependencies and carefully selects training data to optimize performance. Experimental results demonstrate that the proposed algorithm rapidly and effectively restores intricate details in satellite images, surpassing several state-of-the-art methods. Thorough benchmarking against various neural networks and extensive experimentation on a meticulously curated dataset validate the superior performance of the proposed solution. It delivers impressive visual and perceptual quality and exhibits enhanced inference speed. This research opens new avenues for improved accessibility and utilization of high-resolution satellite imagery.
Downloads
References
A. S. Belward and J. O. Skøien, “Who launched what, when and why; trends in Global Land-cover observation capacity from civilian Earth Observation satellites,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 103, pp. 115–128, 2015, DOI: 10.1016/j.isprsjprs.2014.03.009.
K. Ose, T. Corpetti, and L. Demagistri, “Multispectral Satellite Image Processing,” Optical Remote Sensing of Land Surface, pp. 57–124, 2016. 10.1016/B978-1-78548-102-4.50002-8
Dumoulin, Vincent, and Francesco Visin. "A guide to convolution arithmetic for deep learning." arXiv preprint arXiv:1603.07285 (2016).
C. Dong, C. Change Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” Proceedings of the European Conference on Computer Vision, 2016, DOI: 10.48550/arXiv.1608.00367.
L. Liebel and M. Körner, “Single-image super-resolution for multispectral remote sensing data using Convolutional Neural Networks,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLI-B3, pp. 883–890, 2016, doi:10.5194/isprs-archives-XLI-B3-883-2016.
M. U. Müller, N. Ekhtiari, R. M. Almeida, and C. Rieke, “Super-resolution of multispectral satellite images using Convolutional Neural Networks,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. V-1-2020, pp. 33–40, 2020, DOI: 10.5194/isprs-annals-V-1-2020-33-2020.
C. Lanaras, J. Bioucas-Dias, S. Galliani, E. Baltsavias, and K. Schindler, “Super-resolution of sentinel-2 images: Learning a globally applicable deep neural network,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 146, pp. 305–319, 2018, DOI: 10.1016/j.isprsjprs.2018.09.018.
Ç. Tuna, ‘Morphological Hierarchies for Analysing Satellite Image Time Series’, Ph.D. Thesis, 2020.
D. Pouliot, R. Latifovic, J. Pasher, and J. Duffe, “Landsat super-resolution enhancement using convolution neural networks and sentinel-2 for training,” Remote Sensing, vol. 10, no. 3, p. 394, 2018, DOI: 10.3390/rs10030394.
J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, DOI: 10.1109/cvpr.2016.182.
B. Lim, S. Son, H. Kim, S. Nah, K.M. “Lee Enhanced deep residual networks for single image super-resolution” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, IEEE (2017), pp. 1132-1140.
Xueyun Chen, Shiming Xiang, Cheng-Lin Liu, and Chun-Hong Pan, “Vehicle detection in satellite images by hybrid deep convolutional neural networks,” IEEE Geoscience and Remote Sensing Letters, vol. 11, no. 10, pp. 1797–1801, 2014, DOI: 10.1109/LGRS.2014.2309695.
M. Kawulok, S. Piechaczek, K. Hrynczenko, P. Benecki, D. Kostrzewa, and J. Nalepa, “On Training Deep Networks for satellite image Super-Resolution,” IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium, 2019, DOI: 10.48550/arXiv.1906.06697.
T. Lu, K. Zhao, Y. Wu, Z. Wang, and Y. Zhang, “Structure-texture parallel embedding for remote sensing image Super-Resolution,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022, DOI: 10.1109/LGRS.2022.3206348.
C. Liu and D. Liu, “Hyperspectral image super-resolution employing nonlocal block and hybrid multiscale three-dimensional convolution,” Journal of Applied Remote Sensing, vol. 16, no. 03, 2022, DOI: 10.1117/1.JRS.16.036518.
J. Cornebise, I. Oršolić and F. Kalaitzis, “Open High-Resolution Satellite Imagery: The WorldStrat Dataset--With Application to Super-Resolution”. arXiv preprint arXiv:2207.06418 [eess.IV], DOI: 10.48550/arXiv.2207.06418.
F. Palsson, J. R. Sveinsson and M. O. Ulfarsson, “Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network,” IEEE Trans. Geosci. Remote Sens. Let., 14 (5), 639 –643, 2017, DOI: 10.1109/LGRS.2017.2668299.
C. Liu and D. Liu, “Hyperspectral image super-resolution employing nonlocal block and hybrid multiscale three-dimensional convolution,” Journal of Applied Remote Sensing, vol. 16, no. 03, 2022.
J. Yamanaka, S. Kuwashima, and T. Kurita, “Fast and accurate image super-resolution by deep CNN with skip connection and network in Network,” Neural Information Processing, pp. 217–225, 2017, doi:10.48550/arXiv.1707.05425.
“Sentinel-2 operations,” ESA. [Online]. Available: https://www.esa.int/Enabling_Support/Operations/Sentinel-2_operations. [Accessed: 26-Feb-2022].
Chen, Yu, et al. "Fsrnet: End-to-end learning face super-resolution with facial priors." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
Pouliot, D., Latifovic, R., Pasher, J., Duffe, J., 2018, “Landsat Super-Resolution Enhancement Using Convolution Neural Networks and Sentinel-2 for Training,” Remote Sensing 10, 394.
M. Lin, Q. Chen and S. Yan, “Network In Network,” arxiv: 1312.4400 [cs.NE], 2014, DOI: 10.48550/arXiv.1312.4400.
Sun, Ke, et al. "Deep high-resolution representation learning for human pose estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, DOI: 10.1109/CVPR.2019.00584.
M. Al-Qizwini, I. Barjasteh, H. Al-Qassab, and H. Radha, “Deep learning algorithm for autonomous driving using googlenet,” 2017 IEEE Intelligent Vehicles Symposium (IV), 2017.
C. Guillemot and A. Roumy, ‘De Fourier à la compression d’images et de vidéos’, Oct. 2019.
Andrea Bordone Molini, Diego Valsesia, Giulia Fracastoro, Enrico Magli “DeepSUM: Deep neural network for Super-resolution of Unregistered Multitemporal images” (2020)
X. Delaunay, ‘Compression d'images satellite par post-transformées dans le domaine ondelettes’. (2008).
S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, ‘How Does Batch Normalization Help Optimization?’, in Advances in Neural Information Processing Systems, 2018, vol. 31. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/905056c1ac1dad141560467e0a99e1cf-Paper.pdf
Z. He, H. Wu, and G. Wu, “Spectral-spatial classification of hyperspectral images using label dependence,” IEEE Access, vol. 9, pp. 119219–119231, 2021, DOI: 10.1109/ACCESS.2021.3107976.
Z. Zhang, “Improved adam optimizer for Deep Neural Networks,” 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), 2018.
L. Zhipeng , Z. Changming, Z. Haiyang, Z. Zi and W. Xuan, “Application of the multi-scale fusion super-resolution algorithm to UAV detection,” OpticsJournal, 2021, DOI: 10.5768/JAO202142.0302003.
CHOLLET, François. Xception: Deep learning with depthwise separable convolutions. In : Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 1251-1258.
DANG, Lanxue, PANG, Peidong, et LEE, Jay. Depth-wise separable convolution neural network with residual connection for hyperspectral image classification. Remote Sensing, 2020, vol. 12, no 20, p. 3408.
M. A. Ghaffar, A. McKinstry, T. Maul, and T. T. Vu, “Data augmentation approaches for satellite image super-resolution,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-2/W7, pp. 47–54, 2019.
A. Hore and D. Ziou, “Image quality metrics: PSNR vs. SSIM,” 20th International Conference on Pattern Recognition, 2010, DOI: 10.1109/ICPR.2010.579.
Y. Li, W. Xie, and H. Li, “Hyperspectral image reconstruction journal-of-applied-remote-sensing by deep convolutional neural network for classification,” Pattern Recognition, vol. 63, pp. 371–383, 2017, DOI: 10.1016/j.patcog.2016.10.019.
B. Pang, E. Nijkamp, and Y. N. Wu, “Deep learning with tensorflow: A Review,” Journal of Educational and Behavioral Statistics, vol. 45, no. 2, pp. 227–248, 2019, DOI: 10.3102/1076998619872761.
N. Ketkar, “Introduction to keras,” Deep Learning with Python, pp. 97–111, 2017, DOI: 10.3390/rs12203408.
D. C. Vogelsang and B. J. Erickson, “Magician’s Corner: 6. tensorflow and TensorBoard,” Radiology: Artificial Intelligence, vol. 2, no. 3, 2020, DOI: 10.1148/ryai.2020200012.
MÜLLER, Markus U., EKHTIARI, Nikoo, ALMEIDA, Rodrigo M., et al. Super-resolution of multispectral satellite images using convolutional neural networks. arXiv preprint arXiv:2002.00580, 2020.
M. Zabalza and A. Bernardini, “Super-resolution of sentinel-2 images using a spectral attention mechanism,” Remote Sensing, vol. 14, no. 12, p. 2890, 2022. DOI: 10.3390/rs14122890
Cong Liu and Dan Liu "Hyperspectral image super-resolution employing nonlocal block and hybrid multiscale three-dimensional convolution," Journal of Applied Remote Sensing 16(3), 036518 (15 September 2022). DOI: 10.1117/1.JRS.16.036518
J. Jiang et al., “Learning spatial–spectral prior for super-resolution of hyperspectral imagery,” IEEE Trans. Comput. Imag., 6 1082 –1096, 2020. DOI: 10.1109/TCI.2020.2996075 (2020)
Q. Li, Q. Wang and X. Li, “Mixed 2D/3D convolutional network for hyperspectral image super-resolution,” Remote Sens., 12 (10), 1660 https://doi.org/10.3390/rs12101660 (2020)
S. Mei et al., “Hyperspectral image spatial super-resolution via 3D full convolutional neural network,” Remote Sens., 9 1139, 2017. DOI: 10.3390/rs9111139
Y. Li et al., “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in BigMM2018, 1 –4 (2018)
J. Kim, J. K. Lee and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE Conf. Comput. Vision and Pattern Recognit., 1646 –1654 (2016). DOI: 10.1109/CVPR.2016.182
B. Lim et al., “Enhanced deep residual networks for single image super-resolution,” in IEEE Conf. Comput. Vision and Pattern Recognit. Workshops, 1132 –1140 (2017).
C. Dong et al., “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell., 38 (2), 295 –307 DOI: 10.1109/TPAMI.2015.2439281 ITPIDJ 0162-8828 (2016).
P. Liu, Y. Hong, and Y. Liu, “Deep differential convolutional network for Single Image Super-resolution,” IEEE Access, vol. 7, pp. DOI: 37555–37564, 2019. 10.1109/ACCESS.2019.2903528
Y. Zhang and K. Li, “Image super-resolution using very deep residual channel attention networks,” in Proc. Eur. Conf. Comput. Vision., 294 –310 (2018)
R. H. Yuhas, A. F. H. Goetz and J. W. Boardman, “Discrimination among semi-arid landscape endmembres using the spectral angle mapper (SAM) algorithm,” 147 –149 (1992).
T. Chai and R. R. Draxler, “Root mean square error (RMSE) or mean absolute error (mae)? – arguments against avoiding RMSE in the literature,” Geoscientific Model Development, vol. 7, no. 3, pp. 1247–1250, 2014. 10.5194/gmd-7-1247-2014
L. Wald, “Quality of high-resolution synthesised images: is there a simple criterion?” in Proc. Int. Conf. Fusion of Earth Data, 81 –84 (2002).
J.-C. Yoo and T. H. Han, “Fast normalized cross-correlation,” Circuits, Systems and Signal Processing, vol. 28, no. 6, pp. 819–843, 2009. Doi: 10.1007/s00034-009-9130-7
J. Singh, “Computational complexity and analysis of supervised machine learning algorithms,” Lecture Notes in Networks and Systems, pp. 195–206, 2022. DOI: 10.1007/978-981-19-1412-6_16.
Copyright (c) 2023 Journal of Environmental Treatment Techniques
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
