Spatial relation reasoning and representation for image matching

doi:10.11947/j.AGCS.2021.20190497

Abstract

Abstract: The spatial relationship in the paper means the spatial adjacency relation between the objects in word space, and the paper proposes a novel image matching method based on analyzing the spatial adjacency relation between the object contained in the image pair. In the method, the feature extraction network is first trained based on comparison mechanism, and the well-trained model could produce the deep features for the objects, which could effectively match the consistent objects across images. Then the priori images are employed to deduce the spatial adjacency information between different objects, which is further represented by the spatial adjacency graph. Finally, the spatial relation matching between the image pair is conducted by calculating the spatial adjacency score based on the spatial adjacency graph. The experimental results demonstrate that the spatial adjacency scores of the non-matched image pairs generally equal to 0, and those of the matched pairs are generally greater than 0. As several objects are involved, the proposed spatial relation matching method could achieve high robustness, it outperforms other methods in the comparison experiments, and it could effectively complete the image matching task with high efficiency.

Key words: deep feature, spatial adjacency graph, image matching, corresponding node of object

CLC Number:

P208

LI Qin, YOU Xiong, LI Ke, WANG Weiqi. Spatial relation reasoning and representation for image matching[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(1): 117-131.

References

[1] CUMMINS M, NEWMAN P. FAB-MAP:appearance-based place recognition and mapping using a learned visual vocabulary model[C]//Proceedings of the 27th International Conference on Machine Learning (ICML-10).Haifa, Israel:Omnipress, 2010.
[2] QIU Kaichang, WAN Wenhui, ZHAO Hongying, et al. Progress and applications of visual SLAM[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):38-49.
[3] 李向阳, 庄越挺, 潘云鹤. 基于内容的图像检索技术与系统[J]. 计算机研究与发展, 2001, 38(3):344-354. LI Xiangyang, ZHUANG Yueting, PAN Yunhe. The technique and systems of content-based image retrieval[J]. Journal of Computer Research and Development, 2001, 38(3):344-354.
[4] 杨珂, 李从敏, 周维勋, 等. 卷积神经网络多层特征联合的遥感图像检索[J]. 测绘科学, 2019, 44(7):9-15, 34. YANG Ke, LI Congmin, ZHOU Weixun, et al. Remote sensing image retrieval based on multi-layer feature integration of convolution neural networks[J]. Science of Surveying and Mapping, 2019, 44(7):9-15, 34.
[5] 袁一, 程亮, 宗雯雯, 等. 互联网众源照片的三维重建定位技术[J]. 测绘学报,2018, 47(5):631-643. DOI:10.11947/j.AGCS.2018.20170365. YUAN Yi, CHENG Liang, ZONG Wenwen, et al. Crowd-sourced pictures geo-localization method based on 3D reconstruction[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5):631-643. DOI:10.11947/j.AGCS.2018.20170365.
[6] BVRKI M, CADENA C, GILITSCHENSKI I, et al. Appearance-based landmark selection for visual localization[J]. Journal of Field Robotics, 2019, 36(6):1041-1073.
[7] LI Qin, LI Ke, YOU Xiong, et al. Place recognition based on deep feature and adaptive weighting of similarity matrix[J]. Neurocomputing, 2016, 199:114-127.
[8] LOWE D G. Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision.Kerkyra, Greece:IEEE,1999.
[9] BAY H, TUYTELAARS T, VAN GOOL L. SURF:speeded up robust features[C]//Proceedings of the 9th European Conference on Computer Vision. Graz, Austria:Springer,2006.
[10] BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF)[J]. Computer Vision and Image Understanding, 2008, 110(3):346-359.
[11] RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB:An efficient alternative to SIFT or SURF[C]//Proceedings of International Conference on Computer Vision. Barcelona:IEEE, 2012.
[12] SIVIC J, ZISSERMAN A. Video Google:A text retrieval approach to object matching in videos[C]//Proceedings of the 9th IEEE International Conference on Computer Vision. Nice, France:IEEE, 2003.
[13] CAO Song, SNAVELY N. Learning to match images in large-scale collections[C]//Proceedings of 2012 European Conference on Computer Vision. Florence, Italy:Springer, 2012.
[14] JÉGOU H, DOUZE M, SCHMID C, et al. Aggregating local descriptors into a compact image representation[C]//Proceedings of 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA:IEEE, 2010.
[15] JAAKKOLA T S, HAUSSLER D. Exploiting generative models in discriminative classifiers[C]//Proceedings of the 11th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 1998:487-493.
[16] PERRONNIN F, DANCE C. Fisher kernels on visual vocabularies for image categorization[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN:IEEE, 2007:1-8.
[17] ARANDJELOVIĆ R, GRONAT P, TORII A, et al. NetVLAD:CNN architecture for weakly supervised place recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6):1437-1451.
[18] 李钦, 游雄, 李科, 等. 图像深度层次特征提取算法[J]. 模式识别与人工智能, 2017, 30(2):127-136. LI Qin, YOU Xiong, LI Ke, et al. Deep hierarchical feature extraction algorithm[J]. Pattern Recognition and Artificial Intelligence, 2017, 30(2):127-136.
[19] KRIZHEVSKYA, SUTSKEVERI, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. London:MIT Press,2012:1097-1105.
[20] SVNDERHAUF N, SHIRAZI S, DAYOUB F, et al. On the performance of convNet features for place recognition[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany:IEEE, 2015:4297-4304.
[21] MELEKHOV I, KANNALA J, RAHTU E. Siamese network features for image matching[C]//Proceedings of the 23rd International Conference on Pattern Recognition (ICPR). Cancun, Mexico:IEEE, 2016:378-383.
[22] CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). San Diego, CA:IEEE, 2005:539-546.
[23] FINMAN R, PAULL L, LEONARD J. Toward object-based place recognition in dense RGB-D maps[C]//Proceedings of ICRA Workshop Visual Place Recognition in Changing Environments. Seattle, WA:[s.n.], 2015.
[24] PEPPERELL E, CORKE P I, MILFORD M. Routed roads:Probabilistic vision-based place recognition for changing conditions, split streets and varied viewpoints[J]. The International Journal of Robotics Research, 2016, 35(9):1057-1179.
[25] CASCIANELLI S, COSTANTE G, BELLOCCHIO E, et al. Robust visual semi-semantic loop closure detection by a covisibility graph and CNN features[J]. Robotics and Autonomous Systems, 2017, 92:53-65.
[26] ZITNICK C L, DOLLÁR P. Edge boxes:locating object proposals from edges[C]//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland:Springer, 2014.
[27] DOLLÁR P, ZITNICK C L. Structured forests for fast edge detection[C]//Proceedings of 2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia:IEEE, 2014.
[28] BROWN M, HUA Gang, WINDER S. Discriminative learning of local image descriptors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(1):43-57.
[29] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? thekitti vision benchmark suite[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2012:3354-3361.
[30] HUANG Xinyu, WANG Peng, CHENG Xinjing, et al. The apolloscape open dataset for autonomous driving and its application[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10):2702-2719.
[31] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016:2818-282.
[32] CHRISTIAN S, LIU Wei, JIA Yangqing, et al. Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA:IEEE, 2015:1-9.