Affine invariant feature matching of oblique images based on multi-branch network

doi:10.11947/j.AGCS.2021.20200506

Abstract

Abstract: The available wide-baseline image matching algorithms have been prone to failure or only producing few matches, due to the complex affine deformation and perspective distortion. On this basis, we proposed a novel affine invariant feature matching algorithm for oblique stereo images based on multivariate network. In our method, we applied the Hessian algorithm to extract initial feature regions, then we constructed triplet network (TN) model, and obtained affine invariant feature regions through deep learning. To improve the matching performance of similar features, we proposed multilateral constraint loss function to train multi-branch descriptor network (MDN) model, and then generated deep learning descriptors with higher discrimination for image features. Afterwards, the conjugate features were produced by the matching metric of nearest/next distance ratio (NNDR), and eliminated possible mismatch points through random sampling consistency (RANSAC) algorithm. Finally, experiments on oblique stereo images acquired by unmanned aerial vehicle verified the effectiveness of the proposed approach.

Key words: oblique stereo images, convolutional neural network, affine invariant feature, deep learning, image matching

CLC Number:

P236

ZHANG Chuanhui, YAO Guobiao, ZHANG Li, AI Haibin, MAN Xiaocheng, Huang Pengfei. Affine invariant feature matching of oblique images based on multi-branch network[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(5): 641-651.

References

[1] MIKOLAJCZYK K, SCHMID C. Scale & affine invariant interest point detectors[J]. International Journal of Computer Vision, 2004, 60(1):63-86.
[2] 姚国标, 邓喀中, 张力, 等. 基于Harris-Affine的宽基线立体影像LSM匹配方法[J]. 中南大学学报(自然科学版), 2014, 45(8):2661-2668. YAO Guobiao, DENG Kazhong, ZHANG Li, et al. Least square matching method for wide baseline stereo images based on Harris-Affine features[J]. Journal of Central South University (Science and Technology), 2014, 45(8):2661-2668.
[3] MIKOLAJCZYK K, TUYTELAARS T, SCHMID C, et al. A comparison of affine region detectors[J]. International Journal of Computer Vision, 2005, 65(1-2):43-72.
[4] MATAS J, CHUM O, URBAN M, et al. Robust wide-baseline stereo from maximally stable extremal regions[J]. Image and Vision Computing, 2004, 22(10):761-767.
[5] MOREL J M, YU Guoshen. ASIFT:a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2):438-469.
[6] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[7] FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolutional neural network[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):90-100. DOI:10.11947/j.JGGS.2019.0210.
[8] GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1):1-15. DOI:10.11947/j.JGGS.2018.0101.
[9] 刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9):1141-1150. DOI:10.11947/j.AGCS.2019.20180247. LIU Jin, JI Shunping. Deep learning based dense matchingfor aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9):1141-1150. DOI:10.11947/j.AGCS.2019.20180247.
[10] 袁修孝, 袁巍, 许殊, 等. 航摄影像密集匹配的研究进展与展望[J]. 测绘学报, 2019, 48(12):1542-1550. DOI:10.11947/j.AGCS.2019.20190453. YUAN Xiuxiao, YUAN Wei, XU Shu, et al. Research developments and prospects on dense image matching in photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1542-1550. DOI:10.11947/j.AGCS.2019.20190453.
[11] YI K M, TRULLS E, LEPETIT V, et al. LIFT:learned invariant feature transform[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands:Springer, 2016:467-483.
[12] VERDIE Y, YI K M, FUA P, et al. TILDE:a temporally invariant learned DEtector[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015.
[13] TIAN Yurun, FAN Bin, WU Fuchao. L2-Net:deep learning of discriminative patch descriptor in Euclidean space[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:661-669.
[14] MISHCHUK A, MISHKIN D, RADENOVIC' F, et al. Working hard to know your neighbor's margins:local descriptor learning loss[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, CA:Curran Associates Inc., 2017:4826-4837
[15] HOFFER E, AILON N. Deep metric learning using tripletnetwork[C]//Proceedings of the 3rd International Workshop on Similarity-Based Pattern Recognition. Copenhagen, Denmark:Springer, 2015.
[16] IOFFE S, SZEGEDY C. Batch normalization:Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of 2015 International Conference on Machine Learning.[S.l.]:GMLR, 2015.
[17] GLOROT X, BORDES A, BENGIO Y, et al. Deep sparse rectifier neural networks[C]//Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. Fort Lauderdale, FL:W&CP, 2011:315-323.
[18] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE, 2017.
[19] YI K M, VERDIE Y, FUA P, et al. Learning to assign orientations to feature points[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:107-116.
[20] 陈敏, 朱庆, 何海清, 等. 面向城区宽基线立体像对视角变化的结构自适应特征点匹配[J]. 测绘学报, 2019, 48(9):1129-1140. DOI:10.11947/j.AGCS.2019.20180266. CHEN Min, ZHU Qing, HE Haiqing, et al. Structureadaptive feature point matching for urban area wide-baseline images with viewpoint variation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9):1129-1140. DOI:10.11947/j.AGCS.2019.20180266.
[21] BROWN M, HUA Gang, WINDER S. Discriminative learning of local image descriptors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1):43-57.
[22] 姚国标, 邓喀中, 张力, 等. 融合互补仿射不变特征的倾斜立体影像高精度自动配准方法[J]. 测绘学报, 2013, 42(6):869-876, 883. YAO Guobiao, DENG Kazhong, ZHANG Li, et al. An automated registration method with high-accuracy for oblique stereo images based on complementary affine invariant features[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(6):869-876, 883.
[23] 李欣, 杨宇辉, 杨博, 等. 利用方向相位特征进行多源遥感影像匹配[J]. 武汉大学学报(信息科学版), 2020, 45(4):488-494. LI Xin, YANG Yuhui, YANG Bo, et al. A multi-source remote sensing image matching method using directional phase feature[J]. Geomatics and Information Science of Wuhan University, 2020, 45(4):488-494.
[24] 袁修孝, 袁巍, 陈时雨. 基于图论的遥感影像误匹配点自动探测方法[J]. 武汉大学学报(信息科学版), 2018, 43(12):1854-1860. YUAN Xiuxiao, YUAN Wei, CHEN Shiyu. An automatic detection method of mismatching points in remote sensing images based on graph theory[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):1854-1860.
[25] TIAN Yurun, YU Xin, FAN Bin, et al. SOSNet:second order similarity regularization for local descriptor learning[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA:IEEE, 2019:11016-11025.
[26] LI S, HE F, DU B, et al. Fast spatio-temporal residual network for video super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:GMLR, 2019.