Learned local features for SfM reconstruction of UAV images

doi:10.11947/j.AGCS.2024.20220636

Abstract

Abstract: Reliable feature matching plays an essential role in SfM (structure from motion) for UAV (unmanned aerial vehicle) images. Recently, deep learning-based methods have been used for feature detection and matching, which outperforms traditional handcrafted methods, e.g., SIFT, on benchmark datasets. However, few studies have reported their performance on UAV images as these models are trained and tested using internet photos. By using UAV datasets with varying features, this study evaluated both handcrafted and learned methods in terms of feature matching and SfM-based image orientation. The experimental results show that even with the pretrained public-available models, more accurate and complete feature matching can be obtained through the combination of high-precision localization of handcrafted detectors and the high representation ability of learned descriptors, which has competitive or better performance in SfM-based image orientation when compared with SIFT-like handcrafted methods.

Key words: photogrammetry, 3D reconstruction, structure from motion, learned feature, convolutional neural network

CLC Number:

P237

JIANG San, LIU Kai, LI Qingquan, JIANG Wanshou. Learned local features for SfM reconstruction of UAV images[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(2): 321-331.

References

[1] JIANG San, JIANG Wanshou, WANG Lizhe. Unmanned aerial vehicle-based photogrammetric 3D mapping:a survey of techniques, applications, and challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2022, 10(2):135-171.
[2] 陈武, 姜三, 李清泉, 等. 无人机影像增量式运动恢复结构研究进展[J]. 武汉大学学报(信息科学版), 2022, 47(10):1662-1674. CHEN Wu, JIANG San, LI Qingquan,et al. Recent research of incremental structure from motion for unmanned aerial vehicle images[J]. Geomatics and Information Science of Wuhan University, 2022, 47(10):1662-1674.
[3] JIANG San, JIANG Wanshou, HUANG Wei, et al. UAV-based oblique photogrammetry for outdoor data acquisition and offsite visual inspection of transmission line[J]. Remote Sensing, 2017, 9(3):278.
[4] ZHENG J, FU H, LI W, et al. Growing status observation for oil palm trees using unmanned aerial vehicle (UAV) images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 173:95-121.
[5] 姜三, 许志海, 张峰, 等. 面向无人机倾斜影像的高效SfM重建方案[J]. 武汉大学学报(信息科学版), 2019, 44(8):1153-1161. JIANG San, XU Zhihai, ZHANG Feng, et al. Solution for efficient SfM reconstruction of oblique UAV images[J]. Geomatics and Information Science of Wuhan University, 2019, 44(8):1153-1161.
[6] 张力, 刘玉轩, 孙洋杰, 等. 数字航空摄影三维重建理论与技术发展综述[J]. 测绘学报, 2022, 51(7):1437-1457.DOI:10.11947/J.AGCS.2022.20220130. ZHANG Li, LIU Yuxuan, SUN Yangjie, et al. A review of developments in the theory and technology of three-dimensional reconstruction in digital aerial photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7):1437-1457.DOI:10.11947/J.AGCS.2022.20220130.
[7] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[8] ARANDJELOVIC R, ZISSERMAN A. Three things everyone should know to improve object retrieval[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. New York:ACM Press, 2012:2911-2918.
[9] DONG Jingming, SOATTO S. Domain-size pooling in local descriptors:DSP-SIFT[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston:IEEE, 2015:5097-5106.
[10] SUN Yanbiao, ZHAO Liang, HUANG Shoudong, et al. 2-SIFT:sift feature extraction and matching for large images in large-scale aerial photogrammetry[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 91:1-16.
[11] SEDAGHAT A, EBADI H. Remote sensing image matching based on adaptive binning SIFT descriptor[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(10):5283-5293.
[12] 范大昭, 董杨, 张永生. 卫星影像匹配的深度卷积神经网络方法[J]. 测绘学报, 2018, 47(6):844-853. DOI:10.11947/J.AGCS.2018.20170627. FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolution neural network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):844-853. DOI:10.11947/J.AGCS.2018.20170627.
[13] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202.DOI:10.11947/J.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming,et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202.DOI:10.11947/J.AGCS.2021.20200048.
[14] JIN Yuhe, MISHKIN D, MISHCHUK A, et al. Image matching across wide baselines:from paper to practice[J]. International Journal of Computer Vision, 2021, 129(2):517-547.
[15] BALNTAS V, RIBA E, PONSA D, et al. Learning local feature descriptors with triplets and shallow convolutional neural networks[C]//Proceedings of 2016 British Machine Vision Conference 2016. York:British Machine Vision Association, 2016:3.
[16] TIAN Yurun, FAN Bin, WU Fuchao. L2-net:deep learning of discriminative patch descriptor in euclidean space[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu:IEEE, 2017:661-669.
[17] MISHCHUK A, MISHKIN D,RADENOVIĆ F, et al. Working hard to know your neighbor's margins:local descriptor learning loss[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York:ACM Press, 2017:4829-4840.
[18] LUO Zixin, SHEN Tianwei, ZHOU Lei, et al. GeoDesc:learning local descriptors by integrating geometry constraints[EB/OL].[2023-12-30]. https://arxiv.org/abs/1807.06294.pdf.
[19] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint:self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Salt Lake City:IEEE, 2018:224-236.
[20] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net:a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach:IEEE, 2019:8092-8101.
[21] LUO Zixin, ZHOU Lei, BAI Xuyang, et al. ASLFeat:learning local features of accurate shape and localization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE, 2020:6589-6598.
[22] SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue:learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE, 2020:4938-4947.
[23] SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR:detector-free local feature matching with transformers[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville:IEEE, 2021:8922-8931.
[24] BALNTAS V, LENC K, VEDALDI A, et al. HPatches:a benchmark and evaluation of handcrafted and learned local descriptors[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu:IEEE, 2017:5173-5182.
[25] PHILBIN J, CHUM O, ISARD M, et al. Object retrieval with large vocabularies and fast spatial matching[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition.Minneapolis:IEEE, 2007:1-8.
[26] SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas:IEEE, 2016:4104-4113.