语义信息与地理配准相结合的实例目标定位

doi:10.11947/j.AGCS.2023.20220008

测绘学报 ›› 2023, Vol. 52 ›› Issue (8): 1375-1386.doi: 10.11947/j.AGCS.2023.20220008

语义信息与地理配准相结合的实例目标定位

吕可枫¹, 张永生², 于英², 闵杰³

1. 61290部队, 上海 200442;
2. 信息工程大学地理空间信息学院, 河南郑州 450001;
3. 96863部队, 河南洛阳 471000

收稿日期:2022-01-06 修回日期:2023-02-10 发布日期:2023-09-07
通讯作者: 张永生 E-mail:ysZhang2001@vip.163.com
作者简介:吕可枫(1996-),男,博士,主要研究方向为地理空间动态目标智能感知。E-mail:kflv2014@163.com
基金资助:
国家自然科学基金(42071340);中原学者首席科学家工作室专项(2018007)

Instance object localization based on semantic information and geo-registration

Lü Kefeng¹, ZHANG Yongsheng², YU Ying², MIN Jie³

1. Troops 61290, Shanghai 200422, China;
2. Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China;
3. Troops 96863, Luoyang 471000, China

Received:2022-01-06 Revised:2023-02-10 Published:2023-09-07
Supported by:
The National Natural Science Foundation of China (No. 42071340); The Chief Scientist Studio Program of Central Plain Scholar in Remote Sensing and Geospatial Intelligence System (No. 2018007)

摘要/Abstract

摘要： 随着地理空间科学的迅速发展,面向地理空间智能感知应用的研究日益增多。本文以三维模型为先验数据,提出了一种基于语义信息和地理配准的实例目标智能感知与定位的方法。首先,使用载体上搭载的GNSS和IMU(inertial measurement unit)获取传感器的位姿,并利用初始位姿从先验的三维模型实时渲染图像,同时用深度相机获取当前帧的真实场景图像;其次,对真实图像和渲染图像进行全景分割,使用语义分割结果对两幅图进行匹配,获取匹配点对;然后,使用匹配点信息得到两个图像的运动关系,并根据运动参数和三维模型的地理坐标信息对传感器位姿进行优化;最后,使用更新后的传感器位姿、实例分割结果及对应深度信息对目标进行感知和定位。针对不同类型三维模型,使用不同类型图像数据进行了测试,并与多种匹配算法进行了对比,结果表明本文算法能够提高匹配准确率和定位精度,并能有效对目标进行感知和定位。

关键词: 地理空间智能, 地理配准, 全景分割, 目标定位

Abstract: With the rapid development of geospatial science, there are more and more researches towards geospatial intelligent perception. Based on geospatial 3D model as priori data, an instance object perception and localization method based on semantic information and geo-registration is proposed. First, GNSS and IMU (inertial measurement unit) are used to obtain its initial position and orientation, which are then used to render the images from 3D models. Second, the panoptic segmentation and the match network are together used to segment and match the 3D model rendered image and the truth image. Third, the matching results between two images are used to restore the camera motion, which then is used to refine the position of the camera with the geographic coordinate information of the 3D models. Finally, the objects can be detected and localized using the results of instance segmentation, depth information and the refined camera position. The method was tested on different types of 3D models and images. The results demonstrate that the proposed method can improve the matching and localization accuracy, and can detect and localize objects effectively.

Key words: geospatial intelligence, geo-registration, panoptic segmentation, object localization

中图分类号:

P208

吕可枫, 张永生, 于英, 闵杰. 语义信息与地理配准相结合的实例目标定位[J]. 测绘学报, 2023, 52(8): 1375-1386.

Lü Kefeng, ZHANG Yongsheng, YU Ying, MIN Jie. Instance object localization based on semantic information and geo-registration[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(8): 1375-1386.

参考文献

[1] 张永生, 张振超,童晓冲, 等. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9): 1137-1146. DOI: 10.11947/j.AGCS.2021.20200420. ZHANG Yongsheng, ZHANG Zhenchao, TONG Xiaochong, et al. Progress and challenges of geospatial artificial intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(9): 1137-1146. DOI: 10.11947/j.AGCS.2021.20200420.
[2] 李德仁, 洪勇, 王密, 等. 测绘遥感能为智能驾驶做什么?[J]. 测绘学报, 2021, 50(11):1421-1431. DOI: 10.11947/j.AGCS.2021.20210280. LI Deren, HONG Yong, WANG Mi, et al. What can surveying and remote sensing do for intelligent driving?[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11)1421-1431. DOI: 10.11947/j.AGCS.2021.20210280.
[3] 邓晨, 游雄, 张威巍, 等. 基于2D地图的城市户外ARGIS视觉辅助地理配准技术[J]. 测绘学报, 2019, 48(10):1305-1319. DOI: 10.11947/j.AGCS.2019.20190007. DENG Chen, YOU Xiong, ZHANG Weiwei, et al. A vision-aided geo-registration method for outdoor ARGIS in urban envi ronments based on 2D maps[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10):1305-1319. DOI: 10.11947/j.AGCS.2019.20190007.
[4] KIRILLOV A, HE Kaiming, GIRSHICK R, et al. Panoptic segmentation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2020: 9396-9405.
[5] 徐旺, 李爱光. 纸地图增强现实对准技术研究[J]. 测绘科学技术学报, 2016, 33(2): 185-190. XU Wang, LI Aiguang. Research on paper map augmented reality registration technology[J]. Journal of Geomatics Science and Technology, 2016, 33(2): 185-190.
[6] LIU Haomin, ZHANG Guofeng, BAO Hujun. Robust keyframe-based monocular SLAM for augmented reality[C]//Proceedings of 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Merida: IEEE, 2016: 1-10.
[7] YOU Xiong, ZHANG Weiwei, MA Meng, et al. Survey on urban warfare augmented reality[J]. ISPRS International Journal of Geo-Information, 2018, 7(2): 46.
[8] 邓晨, 游雄,智梅霞. 户外增强现实地理配准精度测试评估[J]. 系统仿真学报, 2020, 32(9): 1693-1704. DENG Chen, YOU Xiong, ZHI Meixia. Evaluation on geo-registration accuracy of outdoor augmented reality[J]. Journal of System Simulation, 2020, 32(9): 1693-1704.
[9] BAATZ G, SAURER O, KÖSER K, et al. Large scale visual geo-localization of images in mountainous terrain[C]//Proceedings of 2012 ECC. Berlin: Springer, 2012: 517-530.
[10] MATEI B C, VANDER VALK N, ZHU Zhiwei, et al. Image to LIDAR matching for geotagging in urban environments[C]//Proceedings of 2013 IEEE Workshop on Applications of Computer Vision (WACV). Clearwater Beach: IEEE, 2013: 413-420.
[11] MITHUN N C, SIKKA K, CHIU H P, et al. RGB2LIDAR: towards solving large-scale cross-modal visual localization[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle: ACM Press, 2020: 934-943.
[12] CHIU H P, MURALI V, VILLAMIL R, et al. Augmented reality driving using semantic geo-registration[C]//Proceedings of 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Tuebingen: IEEE, 2018: 423-430.
[13] LIU Weiquan, WANG Cheng, ZANG Yu, et al. Ground camera images and UAV 3D model registration for outdoor augmented reality[C]//Proceedings of 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka:IEEE, 2019: 1050-1051.
[14] LOWE D G. Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision, 2004, 60(2): 91-110.
[15] 叶沅鑫, 单杰, 熊金鑫, 等. 一种结合SIFT和边缘信息的多源遥感影像匹配方法[J]. 武汉大学学报(信息科学版), 2013, 38(10): 1148-1151, 1260. YE Yuanxin, SHAN Jie, XIONG Jinxin, et al. A matching method combining SIFT and edge information for multi-source remote sensing images[J]. Geomatics and Information Science of Wuhan University, 2013, 38(10): 1148-1151, 1260.
[16] DELLINGER F, DELON J, GOUSSEAU Y, et al. SAR-SIFT: a SIFT-like algorithm for SAR images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(1): 453-466.
[17] SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle:IEEE, 2020: 4937-4946.
[18] EFE U, INCE K G, AYDIN ALATAN A. DFM: a performance baseline for deep feature matching[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville:IEEE, 2021: 4279-4288.
[19] 蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202. DOI: 10.11947/j.AGCS.2021.20200048. LAN Chaozhen, LU Wanjie, YU Junming, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202. DOI: 10.11947/j.AGCS.2021.20200048.
[20] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-net: a trainable CNN for joint description and detection of local features[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2020: 8084-8093.
[21] CHENG B, SCHWING A G, KIRILLOV A. Per-Pixel classification is not all you need for semantic segmentation[EB/OL]. [2022-12-08]. https://arxiv.org/abs/2107.06278.
[22] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]// Proceedings of 2020 ECCV. Cham: Springer International Publishing, 2020: 213-229.
[23] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL].[2022-12-08].https://arxiv.org/abs/1706.03762.
[24] HARRIS C, STEPHENS M. A combined corner and edge detector[C]//Proceedings of 1988 Alvey Vision Conference. Manchester: Alvey Vision Club, 1988.
[25] SHI Jianbo, TOMASI C. Good features to track[C]//Proceedings of 1994 IEEE Conference on Computer Vision and Pattern Recognition. Seattle:IEEE, 2002: 593-600.
[26] 高翔, 张涛, 刘毅. 视觉SLAM十四讲: 从理论到实践[M]. 2版. 北京: 电子工业出版社, 2019. GAO Xiang, ZHANG Tao, LIU Yi. Fourteen lectures on visual SLAM: from theory to practice[M]. 2nd ed. Beijing: Publishing House of Electronics Industry, 2019.
[27] CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
[28] SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR: detector-free local feature matching with transformers[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville:IEEE, 2021: 8918-8927.
[29] TOFT C, MADDERN W, TORII A, et al. Long-term visual localization revisited[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 2074-2088.
[30] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City:IEEE, 2018: 337-33712.
[31] TOFT C, STENBORG E, HAMMARSTRAND L, et al. Semantic match consistency for long-term visual localization[C]//Proceedings of 2018 ECCV. Cham: Springer International Publishing, 2018: 391-408.

语义信息与地理配准相结合的实例目标定位

Instance object localization based on semantic information and geo-registration

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 8

编辑推荐

Metrics

本文评价

[1]	程涛, 张洋, James Haworth. 基于网络和图的时空智能——概念、方法和应用[J]. 测绘学报, 2022, 51(7): 1629-1639.
[2]	朱庆, 张利国, 丁雨淋, 胡翰, 葛旭明, 刘铭崴, 王玮. 从实景三维建模到数字孪生建模[J]. 测绘学报, 2022, 51(6): 1040-1049.
[3]	张永生, 张振超, 童晓冲, 纪松, 于英, 赖广陵. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9): 1137-1146.
[4]	邓晨, 游雄, 张威巍, 智梅霞. 基于2D地图的城市户外ARGIS视觉辅助地理配准技术[J]. 测绘学报, 2019, 48(10): 1305-1319.
[5]	曹斌, 朱述龙, 邱振戈, 曹彬才. 一种更严密的双介质立体摄影测量折射改正算法[J]. 测绘学报, 2017, 46(9): 1182-1192.
[6]	刘颖真, 贾奋励, 万刚, 诸云强, 霍超. 非专业弱关联影像的地理配准及其精度评估[J]. 测绘学报, 2015, 44(9): 1014-1021.
[7]	袁修孝,余翔. 高分辨率卫星遥感影像姿态角系统误差检校[J]. 测绘学报, 2012, 41(3): 0-416.
[8]	邢帅徐青刘军李建胜. 多源卫星遥感影像的光束法区域网平差[J]. 测绘学报, 2009, 38(2): 0-174.