基于深度学习的语义SLAM关键帧图像处理

doi:10.11947/j.AGCS.2021.20210251

测绘学报 ›› 2021, Vol. 50 ›› Issue (11): 1605-1616.doi: 10.11947/j.AGCS.2021.20210251

基于深度学习的语义SLAM关键帧图像处理

邓晨¹, 李宏伟², 张斌², 许智宾¹, 肖志远³

1. 郑州大学信息工程学院, 河南郑州 450052;
2. 郑州大学地球科学与技术学院, 河南郑州 450052;
3. 郑州大学水利科学与工程学院, 河南郑州 450001

收稿日期:2021-05-11 修回日期:2021-11-06 发布日期:2021-12-07
通讯作者: 李宏伟 E-mail:laob_811@sina.com
作者简介:邓晨(1995—),男,硕士,研究方向为软件工程、计算机视觉。
基金资助:
中国工程科技发展战略河南研究院战略咨询研究项目（2020HENZT07）

Research on key frame image processing of semantic SLAM based on deep learning

DENG Chen¹, LI Hongwei², ZHANG Bin², XU Zhibin¹, XIAO Zhiyuan³

1. School of Information Engineering, Zhengzhou University, Zhengzhou 450052, China;
2. School of Geoscience and Technology, Zhengzhou University, Zhengzhou 450052, China;
3. School of Water Conservancy Science and Engineering, Zhengzhou University, Zhengzhou 450001, China

Received:2021-05-11 Revised:2021-11-06 Published:2021-12-07
Supported by:
The Strategic Consulting Research Project of Henan Research Institute of China Engineering Science and Technology Development Strategy-Key Technologies of Intelligent Robot Spatial Perception (No. 2020HENZT07)

摘要/Abstract

摘要： 同时定位和地图构建（SLAM）凭借其高能效和低功耗等特点在诸多领域应用前景广阔。然而，在传统的SLAM系统中仍存在一些问题：传统的视觉里程计中关键帧并不包含语义信息，移动机器人获取的图像信息较为单一，且在实际场景中关键帧总包含大量误匹配点和动态点。针对以上问题，本文提出一种语义SLAM思路。首先，为了能够匹配到正确且对应的特征点，摒弃动态点和误匹配点的干扰，提出了一种基于Lucas-Kanade光流法的相邻帧特征状态判别法，将这项功能作为新的线程加入ORB-SLAM3的视觉里程计部分，完成对部分传统SLAM框架的优化和改进工作。其次，针对传统SLAM系统前端视觉里程计获取的图像帧不包含任何语义信息的问题，使用基于YOLOV4的目标检测算法和融合全连接条件随机场CRF的Mask R-CNN语义分割算法对ORB-SLAM3中的关键帧图像进行处理，有效提高了机器人等智能设备对室内环境的感知能力。

关键词: 同时定位与建图, ORB-SLAM3, 三维语义地图, 深度学习, Mask R-CNN

Abstract: Simultaneous localization and mapping (SLAM) has broad application prospects in many fields due to its high energy efficiency and low power consumption. However, there are still some problems in the traditional SLAM system: the key frame in the traditional visual odometer does not contain semantic information, the image information obtained by the mobile robot is relatively single, and the key frame in the actual scene always contains a large number of mismatched points and dynamic points. In response to the above problems, this paper proposes a new idea of semantic SLAM mapping technology. First, in order to find the correct and corresponding feature points, abandon the interference of dynamic points and mismatched points simultaneously, a method for judging the feature state of adjacent frames based on the Lucas-Kanade optical flow method is proposed, and this function is regarded as a new feature. The thread is added to the visual odometry part of ORB-SLAM3 to complete the optimization and improvement of part of the traditional SLAM framework. Secondly, in view of the problem that the image frame obtained by the front-end visual odometer of the traditional SLAM system does not contain any semantic information, the target detection algorithm based on YOLOV4 and the Mask R-CNN semantic segmentation algorithm fused with fully connected conditional random field CRF are used to compare ORB-SLAM3. The key frame image processing of the robot effectively improves the perception of the indoor environment of smart devices such as robots.

Key words: simultaneous localization and mapping, ORB-SLAM3, three-dimensional semantic map, deep learning, Mask R-CNN

中图分类号:

P227

邓晨, 李宏伟, 张斌, 许智宾, 肖志远. 基于深度学习的语义SLAM关键帧图像处理[J]. 测绘学报, 2021, 50(11): 1605-1616.

DENG Chen, LI Hongwei, ZHANG Bin, XU Zhibin, XIAO Zhiyuan. Research on key frame image processing of semantic SLAM based on deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1605-1616.

参考文献

[1] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//Proceedings of 2005 Computer Vision and Pattern Recognition(CVPR). San Diego, CA, USA: IEEE, 2005: 886-893.
[2] 魏乐麒.基于环境语义信息的SLAM算法研究与实现[D]. 西安:西安电子科技大学,2018. WEI Leqi. Research and implementation of SLAM algorithm based on environmental semantic information[D]. Xi’an:Xidian University, 2018.
[3] 常思雨.基于视觉SLAM的语义地图研究[D]. 哈尔滨:哈尔滨工业大学,2018. CHANG Siyu. Research on semantic map based on visual SLAM[D]. Harbin:Harbin Institute of Technology, 2018.
[4] 白云汉.基于SLAM算法和深度神经网络的语义地图构建研究[J].计算机应用与软件,2018(1):183-190. BAI Yunhan.Study on the construction of semantic map based on SLAM algorithm and deep neural network[J].Computer Applications and Software, 2018(1):183-190.
[5] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J].IEEE Trans Pattern Anal Mach Intell, 2017, 39(6):1137-1149.
[6] SHAFIEE M J, CHYWL B, LI F, et al. Fast YOLO: a fast you only look once system for real-time embedded object detection in video[J]. Journal of Computational Vision and Imaging Systems, 2017, 3(1): 112-123.
[7] HE K, GKIOXARI G, DOLLAR P,et al. Mask R-CNN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017.
[8] RUNZ M, AGAPITO L. Co-fusion: real-time segmentation, tracking and fusion of multiple objects[C]//Proceedings of 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore: IEEE, 2017: 4471-4478.
[9] RUNZ M, BUFFIER M, AGAPITO L. MaskFusion: real-time recognition, tracking and reconstruction of multiple moving objects[C]//Proceedings of 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany: IEEE, 2019.
[10] 杨现辉, 王惠南. ICP算法在3D点云配准中的应用研究[J]. 计算机仿真, 2010, 27(8):235-238. YANG Xianhui, WANG Huinan. Application of ICP algorithm in 3D point cloud registration [J]. Computer simulation, 2010, 27(8): 235-238
[11] MISRA I, MOORTHI S M, DHAR D, et al. An automatic satellite image registration technique based on Harris corner detection and random sample consensus (RANSAC) outlier rejection model[C]//Proceedings of 2012 International Conference on Recent Advances in Information Technology. [S.l.]: IEEE, 2012.
[12] BAKER S, MATTHEWS I. Lucas-Kanade 20 years on: a unifying framework[J]. International Journal of Computer Vision, 2004, 56(3):221-255.
[13] ZHANG Q, WANG Y, ZHU L, et al. Research on real-time reasoning based on jetson TX2 heterogeneous acceleration YOLOv4[C]//Proceedings of 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA). [S.l.]: IEEE, 2021.
[14] KAMNITSAS K, LEDIG C, NEWCOMBE V, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation[J]. Medical Image Analysis, 2016, 36:61.
[15] MUR-ARTAL R, MONTIEL J M M, TARDOS J D. ORB-SLAM: A versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2015, 31(5):1147-1163.
[16] 吴刚, 武春风, 侯晴宇,等. 基于不变矩特征匹配的目标定位方法[J]. 光学精密工程, 2009, 17(2):460-468. WU Gang, WU Chunfeng, HOU Qingyu, et al. Target location method based on invariable moment feature matching[J]. Guangxue Jingmi Gongcheng/Optics and Precision Engineering, 2009, 17(2):460-468.
[17] 胡树楷, 杨洋, 王新梅. 几种LDPC码的最小汉明距离的计算[J]. 北京邮电大学学报, 2012, 35(2):1-4. HU Shukai, YANG Yang, WANG Xinmei. Calculation of minimum hamming distance of several LDPC codes[J]. Journal of Beijing University of Posts and Telecommunications, 2012, 35(2):1-4.
[18] DEHURI S, ROY R, CHO S B, et al. An improved swarm optimized functional link artificial neural network (ISO-FLANN) for classification[J]. Journal of Systems & Software, 2012, 85(6):1333-1345.
[19] CARON F, DUFLOS E, POMORSKI D, et al. GPS/IMU data fusion using multisensor kalman filtering: introduction of contextual aspects[J]. Information Fusion, 2017, 7(2):221-230.
[20] 赵玮, 王强, 何晓晖,等. 基于BA优化核参数的非线性Volterra滤波方法研究[J]. 仪表技术与传感器, 2019(6):77-81. ZHAO Wei, WANG Qiang, HE Xiaohui, et al. Research on nonlinear volterra filtering method based on BA Optimization of kernel parameters[J]. Instrument Technique and Sensor, 2019(6):77-81.
[21] 熊磊, 刘国栋, 刘小勇,等. 数字图像相关中基于非线性灰度改变模型的灰度梯度迭代算法[J]. 机床与液压, 2012, 40(13):70-70. XIONG Lei, LIU Guodong, LIU Xiaoyong, et al. Grayscale gradient iteration algorithm based on nonlinear grayscale change model in digital image correlation[J]. Machine Tool & Hydraulics, 2012, 40(13):70-70.
[22] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of 2014 European Conference on Computer Vision. [S.l.]: Springer International Publishing,2014.
[23] LONG SUN,TAO WU,GUANGCAI SUN,et al.object detection research of SAR image using improved faster regionbased convolutional neural network[J].Journal of Geodesy and Geoinformation Science,2020,3(3):18-28.
[24] HE K, ZHANG X, REN S, et al. Identity mappings in deep residual networks[C]//Proceedings of 2016 European Conference on Computer Vision. [S.l.]: Springer International Publishing, 2016.
[25] JIANG J, FU X, QIN R, et al. High-speed lightweight ship detection algorithm based on YOLO-V4 for three-channels RGB SAR image[J]. Remote Sensing, 2021, 13(10):1909.
[26] JIANYA GONG,SHUNPING JI.Photogrammetry and deep learning[J].Journal of Geodesy and Geoinformation Science,2018(01):1-15.
[27] VINYALS O,TOSHEV A,BENGIO S, et al. Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016, 39(4):652-663.

基于深度学习的语义SLAM关键帧图像处理

Research on key frame image processing of semantic SLAM based on deep learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张良培, 何江, 杨倩倩, 肖屹, 袁强强. 数据驱动的多源遥感信息融合研究进展[J]. 测绘学报, 2022, 51(7): 1317-1337.
[2]	程涛, 张洋, James Haworth. 基于网络和图的时空智能——概念、方法和应用[J]. 测绘学报, 2022, 51(7): 1629-1639.
[3]	龚健雅, 宦麟茜, 郑先伟. 影像解译中的深度学习可解释性分析方法[J]. 测绘学报, 2022, 51(6): 873-884.
[4]	张勤, 赵超英, 陈雪蓉. 多源遥感地质灾害早期识别技术进展与发展趋势[J]. 测绘学报, 2022, 51(6): 885-896.
[5]	龚健雅, 张觅, 胡翔云, 张展, 李彦胜, 姜良存. 智能遥感深度学习框架与模型设计[J]. 测绘学报, 2022, 51(4): 475-487.
[6]	毛文婧, 王卫林, 焦利民, 刘安宝. 基于深度学习的中国连续空间覆盖PM_2.5浓度预报[J]. 测绘学报, 2022, 51(3): 361-372.
[7]	金飞, 官恺, 刘智, 韩佳容, 芮杰, 李庆高. 无监督密集匹配特征提取网络性能分析[J]. 测绘学报, 2022, 51(3): 426-436.
[8]	何直蒙, 丁海勇, 安炳琪. 高分辨率遥感影像建筑物提取的空洞卷积E-Unet算法[J]. 测绘学报, 2022, 51(3): 457-467.
[9]	张永生, 张振超, 童晓冲, 纪松, 于英, 赖广陵. 地理空间智能研究进展和面临的若干挑战[J]. 测绘学报, 2021, 50(9): 1137-1146.
[10]	艾廷华. 深度学习赋能地图制图的若干思考[J]. 测绘学报, 2021, 50(9): 1170-1182.
[11]	龚健雅, 许越, 胡翔云, 姜良存, 张觅. 遥感影像智能解译样本库现状与研究[J]. 测绘学报, 2021, 50(8): 1013-1022.
[12]	杨必胜, 韩旭, 董震. 适用于城市场景大规模点云语义标识的深度学习网络[J]. 测绘学报, 2021, 50(8): 1059-1067.
[13]	陶超, 阴紫薇, 朱庆, 李海峰. 遥感影像智能解译:从监督学习到自监督学习[J]. 测绘学报, 2021, 50(8): 1122-1134.
[14]	晏雄锋, 艾廷华, 杨敏, 郑建滨. 地图空间形状认知的自编码器深度学习方法[J]. 测绘学报, 2021, 50(6): 757-765.
[15]	张传辉, 姚国标, 张力, 艾海滨, 满孝成, 黄鹏飞. 多分支网络联合的倾斜影像仿射不变特征匹配[J]. 测绘学报, 2021, 50(5): 641-651.