Instance object localization based on semantic information and geo-registration

doi:10.11947/j.AGCS.2023.20220008

Abstract

Abstract: With the rapid development of geospatial science, there are more and more researches towards geospatial intelligent perception. Based on geospatial 3D model as priori data, an instance object perception and localization method based on semantic information and geo-registration is proposed. First, GNSS and IMU (inertial measurement unit) are used to obtain its initial position and orientation, which are then used to render the images from 3D models. Second, the panoptic segmentation and the match network are together used to segment and match the 3D model rendered image and the truth image. Third, the matching results between two images are used to restore the camera motion, which then is used to refine the position of the camera with the geographic coordinate information of the 3D models. Finally, the objects can be detected and localized using the results of instance segmentation, depth information and the refined camera position. The method was tested on different types of 3D models and images. The results demonstrate that the proposed method can improve the matching and localization accuracy, and can detect and localize objects effectively.

Key words: geospatial intelligence, geo-registration, panoptic segmentation, object localization

CLC Number:

P208

Lü Kefeng, ZHANG Yongsheng, YU Ying, MIN Jie. Instance object localization based on semantic information and geo-registration[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(8): 1375-1386.

References

[1] 张永生, 张振超,童晓冲, 等. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9): 1137-1146. DOI: 10.11947/j.AGCS.2021.20200420. ZHANG Yongsheng, ZHANG Zhenchao, TONG Xiaochong, et al. Progress and challenges of geospatial artificial intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(9): 1137-1146. DOI: 10.11947/j.AGCS.2021.20200420.
[2] 李德仁, 洪勇, 王密, 等. 测绘遥感能为智能驾驶做什么?[J]. 测绘学报, 2021, 50(11):1421-1431. DOI: 10.11947/j.AGCS.2021.20210280. LI Deren, HONG Yong, WANG Mi, et al. What can surveying and remote sensing do for intelligent driving?[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11)1421-1431. DOI: 10.11947/j.AGCS.2021.20210280.
[3] 邓晨, 游雄, 张威巍, 等. 基于2D地图的城市户外ARGIS视觉辅助地理配准技术[J]. 测绘学报, 2019, 48(10):1305-1319. DOI: 10.11947/j.AGCS.2019.20190007. DENG Chen, YOU Xiong, ZHANG Weiwei, et al. A vision-aided geo-registration method for outdoor ARGIS in urban envi ronments based on 2D maps[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10):1305-1319. DOI: 10.11947/j.AGCS.2019.20190007.
[4] KIRILLOV A, HE Kaiming, GIRSHICK R, et al. Panoptic segmentation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2020: 9396-9405.
[5] 徐旺, 李爱光. 纸地图增强现实对准技术研究[J]. 测绘科学技术学报, 2016, 33(2): 185-190. XU Wang, LI Aiguang. Research on paper map augmented reality registration technology[J]. Journal of Geomatics Science and Technology, 2016, 33(2): 185-190.
[6] LIU Haomin, ZHANG Guofeng, BAO Hujun. Robust keyframe-based monocular SLAM for augmented reality[C]//Proceedings of 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Merida: IEEE, 2016: 1-10.
[7] YOU Xiong, ZHANG Weiwei, MA Meng, et al. Survey on urban warfare augmented reality[J]. ISPRS International Journal of Geo-Information, 2018, 7(2): 46.
[8] 邓晨, 游雄,智梅霞. 户外增强现实地理配准精度测试评估[J]. 系统仿真学报, 2020, 32(9): 1693-1704. DENG Chen, YOU Xiong, ZHI Meixia. Evaluation on geo-registration accuracy of outdoor augmented reality[J]. Journal of System Simulation, 2020, 32(9): 1693-1704.
[9] BAATZ G, SAURER O, KÖSER K, et al. Large scale visual geo-localization of images in mountainous terrain[C]//Proceedings of 2012 ECC. Berlin: Springer, 2012: 517-530.
[10] MATEI B C, VANDER VALK N, ZHU Zhiwei, et al. Image to LIDAR matching for geotagging in urban environments[C]//Proceedings of 2013 IEEE Workshop on Applications of Computer Vision (WACV). Clearwater Beach: IEEE, 2013: 413-420.
[11] MITHUN N C, SIKKA K, CHIU H P, et al. RGB2LIDAR: towards solving large-scale cross-modal visual localization[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle: ACM Press, 2020: 934-943.
[12] CHIU H P, MURALI V, VILLAMIL R, et al. Augmented reality driving using semantic geo-registration[C]//Proceedings of 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Tuebingen: IEEE, 2018: 423-430.
[13] LIU Weiquan, WANG Cheng, ZANG Yu, et al. Ground camera images and UAV 3D model registration for outdoor augmented reality[C]//Proceedings of 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka:IEEE, 2019: 1050-1051.
[14] LOWE D G. Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision, 2004, 60(2): 91-110.
[15] 叶沅鑫, 单杰, 熊金鑫, 等. 一种结合SIFT和边缘信息的多源遥感影像匹配方法[J]. 武汉大学学报(信息科学版), 2013, 38(10): 1148-1151, 1260. YE Yuanxin, SHAN Jie, XIONG Jinxin, et al. A matching method combining SIFT and edge information for multi-source remote sensing images[J]. Geomatics and Information Science of Wuhan University, 2013, 38(10): 1148-1151, 1260.
[16] DELLINGER F, DELON J, GOUSSEAU Y, et al. SAR-SIFT: a SIFT-like algorithm for SAR images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(1): 453-466.
[17] SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle:IEEE, 2020: 4937-4946.
[18] EFE U, INCE K G, AYDIN ALATAN A. DFM: a performance baseline for deep feature matching[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville:IEEE, 2021: 4279-4288.
[19] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202. DOI: 10.11947/j.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202. DOI: 10.11947/j.AGCS.2021.20200048.
[20] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net: a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2020: 8084-8093.
[21] CHENG B, SCHWING A G, KIRILLOV A. Per-Pixel classification is not all you need for semantic segmentation[EB/OL]. [2022-12-08]. https://arxiv.org/abs/2107.06278.
[22] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]// Proceedings of 2020 ECCV. Cham: Springer International Publishing, 2020: 213-229.
[23] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL].[2022-12-08].https://arxiv.org/abs/1706.03762.
[24] HARRIS C, STEPHENS M. A combined corner and edge detector[C]//Proceedings of 1988 Alvey Vision Conference. Manchester: Alvey Vision Club, 1988.
[25] SHI Jianbo, TOMASI C. Good features to track[C]//Proceedings of 1994 IEEE Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2002: 593-600.
[26] 高翔, 张涛, 刘毅. 视觉SLAM十四讲: 从理论到实践[M]. 2版. 北京: 电子工业出版社, 2019. GAO Xiang, ZHANG Tao, LIU Yi. Fourteen lectures on visual SLAM: from theory to practice[M]. 2nd ed. Beijing: Publishing House of Electronics Industry, 2019.
[27] CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
[28] SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR: detector-free local feature matching with transformers[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville:IEEE, 2021: 8918-8927.
[29] TOFT C, MADDERN W, TORII A, et al. Long-term visual localization revisited[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 2074-2088.
[30] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City:IEEE, 2018: 337-33712.
[31] TOFT C, STENBORG E, HAMMARSTRAND L, et al. Semantic match consistency for long-term visual localization[C]//Proceedings of 2018 ECCV. Cham: Springer International Publishing, 2018: 391-408.