Deep learning method for large scene DSM generation of GF-7 imagery

doi:10.11947/j.AGCS.2023.20220567

Abstract

Abstract: Stereo matching is an important step to generate DSM from satellite imageries. Recently, studies have shown that deep learning-based methods have better performance. However, due to the fixed and limited disparity range predicted by models and the lack of training data, deep learning is rarely directly applied to the stereo matching of satellite images in large scenes. In this paper, a hierarchical dynamic matching strategy is proposed to dynamically determine the region of image blocks of the current level according to the matching results of the previous level, so that the disparity between the left and right epipolar image blocks is relatively small, which is conducive to the prediction of deep learning models. Besides, a scheme for the production of samples is introduced, and a GF-7 dataset is constructed by using manually edited DSM or LiDAR point clouds to obtain ground truth disparity values. In the experiment, this dataset, together with an existing dataset, is used to train Stereo-Net and DSM-Net, and based on the hierarchical matching strategy, the generation of high-quality DSM from Gaofen-7 imagery combined with deep learning technology is achieved for the first time. Experiments in imageries from three cities show that the average endpoint error is about 1 pixel, and the fraction of erroneous pixels is less than 3.8%. The quality of the generated DSMs is better than that of the traditional method.

Key words: satellite imagery, stereo matching, DSM, deep learning, GF-7

CLC Number:

P236

HE Sheng, ZHANG Jiankai, CHEN Feng, LI Shenhong, JIANG Wanshou. Deep learning method for large scene DSM generation of GF-7 imagery[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(12): 2103-2114.

References

[1] 季顺平, 罗冲, 刘瑾. 基于深度学习的立体影像密集匹配方法综述[J]. 武汉大学学报(信息科学版), 2021, 46(2): 193-202. JI Shunping, LUO Chong, LIU Jin. A review of dense stereo image matching methods based on deep learning[J]. Geomatics and Information Science of Wuhan University, 2021, 46(2): 193-202.
[2] SCHARSTEIN D, SZELISKI R, ZABIH R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47(1): 7-42.
[3] 岳庆兴, 高小明, 唐新明. 基于半全局优化的资源三号卫星影像DSM提取方法[J]. 武汉大学学报(信息科学版), 2016, 41(10): 1279-1285. YUE Qingxing, GAO Xiaoming, TANG Xinming. ZY-3 DSM generation method based on semi-global optimization[J]. Geomatics and Information Science of Wuhan University, 2016, 41(10): 1279-1285.
[4] HIRSCHMVLLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341.
[5] GEHRKE S, MORIN K, DOWNEY M, et al. Semi-global matching: an alternative to LiDAR for DSM generation? [EB/OL].[2022-10-04]. https://www.isprs.org/proceedings/XXXVIII/part1/11/11_01_Paper_121.pdf.
[6] D'ANGELO P, REINARTZ P. Semiglobal matching results on the ISPRS stereo matching benchmark[EB/OL]. [2022-10-04]. https://pdfs.semanticscholar.org/63a1/166a69bb236bf0c07c4d0c421aa8ecad3e81.pdf.
[7] 张力, 张继贤. 基于多基线影像匹配的高分辨率遥感影像DEM的自动生成[J]. 武汉大学学报(信息科学版), 2008, 33(9): 943-946. ZHANG Li, ZHANG Jixian. Automatic DEM generation from high-resolution satellite imagery based on multiple-baseline image matching[J]. Geomatics and Information Science of Wuhan University, 2008, 33(9): 943-946.
[8] 刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9): 1141-1150. DOI: 10.11947/j.AGCS.2019.20180247. LIU Jin, JI Shunping. Deep learning based dense matching for aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9): 1141-1150. DOI: 10.11947/j.AGCS.2019.20180247.
[9] POGGI M, TOSI F, BATSOS K, et al. On the synergies between machine learning and binocular stereo for depth estimation from images: a survey[EB/OL].[2022-10-04]. https://arxiv.org/abs/2004.08566.
[10] ZBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[EB/OL].[2022-10-04]. https://arxiv.org/pdf/1409.4326.pdf.
[11] SEKI A, POLLEFEYS M. SGM-nets: semi-global matching with neural networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017:6640-6649.
[12] SHAKED A, WOLF L. Improved stereo matching with constant highway networks and reflective confidence learning[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017:6901-6910.
[13] MAYER N, ILG E, HAUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016:4040-4048.
[14] CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego:IEEE, 2005: 539-546.
[15] KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 66-75.
[16] CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 5410-5418.
[17] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[18] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE, 2012: 3354-3361.
[19] MENZE M, GEIGER A. Object scene flow for autonomous vehicles[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3061-3070.
[20] SCHARSTEIN D, HIRSCHMVLLER H, KITAJIMA Y, et al. High-resolution stereo datasets with subpixel-accurate ground truth[M]//Lecture Notes in Computer Science. Cham: Springer International Publishing, 2014: 31-42.
[21] MAYER N, ILG E, FISCHER P, et al. What makes good synthetic training data for learning disparity and optical flow estimation?[J]. International Journal of Computer Vision, 2018, 126(9): 942-960.
[22] TAO Rongshu, XIANG Yuming, YOU Hongjian. An edge-sense bidirectional pyramid network for stereo matching of VHR remote sensing images[J]. Remote Sensing, 2020, 12(24): 4025.
[23] HE Sheng, LI Shenhong, JIANG San, et al. HMSM-Net: hierarchical multi-scale matching network for disparity estimation of high-resolution satellite stereo images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 188: 314-330.
[24] RAO Zhibo, HE Mingyi, ZHU Zhidong, et al. Bidirectional guided attention network for 3D semantic detection of remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 6138-6153.
[25] KHAMIS S, FANELLO S, RHEMANN C, et al. StereoNet: guided hierarchical refinement for real-time edge-aware depth prediction[C]//Proceedings of 2018 ECCV. New York: ACM Press, 2018: 596-613.
[26] HE Sheng, ZHOU Ruqin, LI Shenhong, et al. Disparity estimation of high-resolution remote sensing images with dual-scale matching network[J]. Remote Sensing, 2021, 13(24): 5050.
[27] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-780.
[28] WU T, VALLET B, PIERROT-DESEILLIGNY M, et al. A new stereo dense matching benchmark dataset for deep learning[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2021, XLⅢ-B2-2021: 405-412.
[29] BOSCH M, FOSTER K, CHRISTIE G, et al. Semantic stereo for incidental satellite images[C]//Proceedings of 2019 IEEE Winter Conference on Applications of Computer Vision.Waikoloa Village: IEEE, 2019: 1524-1532.
[30] PATIL S, COMANDUR B, PRAKASH T, et al. A new stereo benchmarking dataset for satellite images[EB/OL]. [2022-10-04]. https://arxiv.org/abs/1907.04404.pdf.
[31] LI Shenhong, HE Sheng, JIANG San, et al. WHU-stereo: a challenging benchmark for stereo matching of high-resolution satellite images[EB/OL]. [2022-10-04]. https://arxiv.org/abs/2206.02342.pdf
[32] 杨幸彬, 吕京国, 江珊, 等. 高分辨率遥感影像DSM的改进半全局匹配生成方法[J]. 测绘学报, 2018, 47(10): 1372-1384. DOI: 10.11947/j.AGCS.2018.20180091. YANG Xingbin, LÜ Jingguo, JIANG Shan, et al. Digital surface model generation for high resolution satellite stereo image based on modified semi-global matching[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(10): 1372-1384. DOI: 10.11947/j.AGCS.2018.20180091.
[33] 袁修孝, 曹金山. 高分辨率卫星遥感精确对地目标定位理论与方法[M]. 北京: 科学出版社, 2012: 182-183. YUAN Xiuxiao, CAO Jinshan. Theory and method of precise object positioning of high resolution satellite imagery[M]. Beijing: Science Press, 2012: 182-183.