无监督密集匹配特征提取网络性能分析

doi:10.11947/j.AGCS.2022.20200503

摘要/Abstract

摘要： 随着人工智能的发展，基于深度学习的有监督密集匹配方法在虚拟、室内及驾驶等近景数据集上取得了不错的表现。针对航空影像密集匹配标签数据获取困难的问题，本文在无监督密集匹配框架下，借鉴多个有监督网络结构，分别在航空影像数据集和作为参照的近景数据集上测试了匹配精度，实现了网络结构模块与精度关系的定性分析，为进一步探索深度学习在测绘领域的实用化提供了重要的参考。试验在相同损失函数条件下，分别采用DispNetS、DispNetC、iResNet、GCNet、PSMNetB及PSMNetS网络结构进行测试。经分析，得出如下结论：①测试的网络结构中，PSMNetS在航空影像数据集和近景数据集上表现稳定，且精度最高，训练整体耗时少，具有实用化的潜力；②在监督方法中效果更好的网络结构在无监督方法中效果不一定更好，其精度不仅取决于网络自身的匹配能力，同时也依赖于网络与损失函数的兼容性；③孪生网络模块、相关信息融合模块、金字塔池化模块和堆叠沙漏模块与无监督损失函数兼容性良好，可提升网络精度，而iResNet的图像重构迭代精化模块与重构损失函数重复约束，会产生“负优化”的作用。

关键词: 密集匹配, 深度学习, 无监督, 特征提取, 航空影像

Abstract: With the development of artificial intelligence, supervised dense matching method based on deep learning has achieved good performance in virtual, indoor and driving data sets. In view of the difficulty in obtaining aerial image dense matching tag data, we use unsupervised dense matching framework for reference, and test the matching accuracy on aerial image data set and referential close range data set respectively, and realize the qualitative analysis of the relationship between network structure module and precision, which provides a further exploration of the practical application of deep learning in the field of surveying and mapping, and has important reference value. Under the same loss function condition, DispNetS, DispNetC, iResNet, GCNet, PSMNetB and PSMNetS network structures are used to test. Through analysis, the following conclusions are obtained:① Among the tested network structures, PSMNetS has the highest accuracy in aerial image data set and close range data set, and has the potential of practical application; ② The network with better performance in the supervised method may not have better performance in the unsupervised method. Its accuracy depends not only on the matching ability of the network itself, but also on the compatibility between the network and the loss function; ③ The twin network module, related information fusion module, pyramid pooling module and stacked hourglass module have good compatibility with unsupervised loss function, which can improve the network accuracy, while iResNet's image reconstruction iterative refinement module and reconstruction loss function repeat constraints, which will produce a "negative optimization" effect.

Key words: dense matching, deep learning, unsupervised learning, feature extraction, aerial imaging

中图分类号:

P237

金飞, 官恺, 刘智, 韩佳容, 芮杰, 李庆高. 无监督密集匹配特征提取网络性能分析[J]. 测绘学报, 2022, 51(3): 426-436.

JIN Fei, GUAN Kai, LIU Zhi, HAN Jiarong, RUI Jie, LI Qinggao. A performing analysis of unsupervised dense matching feature extraction networks[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(3): 426-436.

参考文献

[1] JANAI J, GVNEY F, BEHL A, et al. Computer vision for autonomous vehicles:problems, datasets and state of the art[J]. Foundations and Trends^® in Computer Graphics and Vision, 2020, 12(1-3):1-308.
[2] 纪艳华. 顾及边缘信息的航摄影像半全局密集匹配算法研究[D]. 武汉:武汉大学, 2020. JI Yanhua. Research on semi-global dense matching algorithm for aerial images using edge information[D]. Wuhan:Wuhan University, 2020.
[3] GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018(1):1-15.
[4] SCHARSTEIN D, SZELISKI R, ZABIH R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[C]//Proceedings of IEEE Workshop on Stereo and Multi-Baseline Vision. Kauai, HI, USA:IEEE, 2001:131-140.
[5] ZABIH R, WOODFILL J. Non-parametric local transforms for computing visual correspondence[J]. Lecture Notes in Computer Science, 1994, 801(1):151-158.
[6] BIRCHFIELD S, TOMASI C. Depth discontinuities by pixel-to-pixel stereo[C]//Proceedings of 6th International Conference on Computer Vision. Bombay, India:IEEE, 1998:1073-1080.
[7] BOYKOV Y, VEKSLER O, ZABIH R. Fast approximate energy minimization via graph cuts[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra, Greece:IEEE, 1999:377-384.
[8] HIRSCHMVLLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2):328-341.
[9] ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston, MA, USA:IEEE, 2015:1592-1599.
[10] MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA:IEEE, 2016:4040-4048.
[11] DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet:learning optical flow with convolutional networks[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:2758-2766.
[12] LIANG Z, FENG Y, GUO Y, et al. Learning deep correspondence through prior and posterior feature constancy[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA:IEEE,2017:2403-2411.
[13] KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy:IEEE, 2017:66-75.
[14] NIBALI A, HE Z, MORGAN S, et al. Numerical coordinate regression with convolutional neural networks[EB/OL].[2020-08-25].https://arxiv.org/abs/1801.07372.
[15] CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA:IEEE, 2018:5410-5418.
[16] ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA:IEEE, 2017:6230-6239.
[17] NEWELL A, YANG Kaiyu, DENG Jia. Stacked hourglass networks for human pose estimation[C]//Proceedings of 2016 Computer Vision. Amsterdam,The Netherlands:ECCV, 2016:483-499.
[18] XIE Junyuan, GIRSHICK R, FARHADI A. Deep3D:fully automatic 2D-to-3D video conversion with deep convolutional neural networks[C]//Proceedings of 2016 Computer Vision.Amsterdam,The Netherlands:ECCV, 2016:842-857.
[19] GARG R, VIJAY KUMAR B G, CARNEIRO G, et al. Unsupervised CNN for single view depth estimation:geometry to the rescue[C]//Proceedings of 2016 Computer Vision.Amsterdam,The Netherlands:ECCV 2016:740-756.
[20] GODARD C, AODHA O M, BROSTOW G J. Unsupervised monocular depth estimation with left-right consistency[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA:IEEE, 2017:6602-6611.
[21] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[EB/OL] [2020-08-25]. http://cn.arxiv.org/abs/1506.02025.
[22] ZHONG Y, DAI Y, LI H. Self-supervised learning for stereo matching with self-improving ability[EB/OL] [2020-08-25]. http://cn.arxiv.org/abs/1709.00930.
[23] 王玉锋, 王宏伟, 吴晨, 等. 基于共同视域的自监督立体匹配算法[J]. 光学学报, 2019, 39(2):287-296. WANG Yufeng, WANG Hongwei, WU Chen, et al. Self-supervised stereo matching algorithm based on common view[J]. Acta Optica Sinica, 2019, 39(2):287-296.
[24] CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.San Diego, CA, USA:IEEE, 2005:539-546.
[25] WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment:from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612.
[26] 刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9):1141-1150.DOI:10.11947/j.AGCS.2019.20180247. LIU Jin, JI Shunping. Deep learning based dense matching for aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9):1141-1150.DOI:10.11947/j.AGCS.2019.20180247.
[27] LIU Jin, JI Shunping. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020:6049-6058.
[28] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA:IEEE, 2012:3354-3361.
[29] MENZE M, GEIGER A. Object scene flow for autonomous vehicles[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015:3061-3070.
[30] PASZKE A, GROSS S, CHINTALA S, et al. Automatic differentiation in pytorch[EB/OL].[2020-08-25]. https://openreview.net/pdf?id=BJJsrmfCZ.
[31] KINGMA D, BA J. Adam:a method for stochastic optimization[EB/OL].[2020-08-25]. http://arxiv.org/abs/1412.6980.