Research on key frame image processing of semantic SLAM based on deep learning

doi:10.11947/j.AGCS.2021.20210251

Abstract

Abstract: Simultaneous localization and mapping (SLAM) has broad application prospects in many fields due to its high energy efficiency and low power consumption. However, there are still some problems in the traditional SLAM system: the key frame in the traditional visual odometer does not contain semantic information, the image information obtained by the mobile robot is relatively single, and the key frame in the actual scene always contains a large number of mismatched points and dynamic points. In response to the above problems, this paper proposes a new idea of semantic SLAM mapping technology. First, in order to find the correct and corresponding feature points, abandon the interference of dynamic points and mismatched points simultaneously, a method for judging the feature state of adjacent frames based on the Lucas-Kanade optical flow method is proposed, and this function is regarded as a new feature. The thread is added to the visual odometry part of ORB-SLAM3 to complete the optimization and improvement of part of the traditional SLAM framework. Secondly, in view of the problem that the image frame obtained by the front-end visual odometer of the traditional SLAM system does not contain any semantic information, the target detection algorithm based on YOLOV4 and the Mask R-CNN semantic segmentation algorithm fused with fully connected conditional random field CRF are used to compare ORB-SLAM3. The key frame image processing of the robot effectively improves the perception of the indoor environment of smart devices such as robots.

Key words: simultaneous localization and mapping, ORB-SLAM3, three-dimensional semantic map, deep learning, Mask R-CNN

CLC Number:

P227

DENG Chen, LI Hongwei, ZHANG Bin, XU Zhibin, XIAO Zhiyuan. Research on key frame image processing of semantic SLAM based on deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1605-1616.

References

[1] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//Proceedings of 2005 Computer Vision and Pattern Recognition(CVPR). San Diego, CA, USA: IEEE, 2005: 886-893.
[2] 魏乐麒.基于环境语义信息的SLAM算法研究与实现[D]. 西安:西安电子科技大学,2018. WEI Leqi. Research and implementation of SLAM algorithm based on environmental semantic information[D]. Xi’an:Xidian University, 2018.
[3] 常思雨.基于视觉SLAM的语义地图研究[D]. 哈尔滨:哈尔滨工业大学,2018. CHANG Siyu. Research on semantic map based on visual SLAM[D]. Harbin:Harbin Institute of Technology, 2018.
[4] 白云汉.基于SLAM算法和深度神经网络的语义地图构建研究[J].计算机应用与软件,2018(1):183-190. BAI Yunhan.Study on the construction of semantic map based on SLAM algorithm and deep neural network[J].Computer Applications and Software, 2018(1):183-190.
[5] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J].IEEE Trans Pattern Anal Mach Intell, 2017, 39(6):1137-1149.
[6] SHAFIEE M J, CHYWL B, LI F, et al. Fast YOLO: a fast you only look once system for real-time embedded object detection in video[J]. Journal of Computational Vision and Imaging Systems, 2017, 3(1): 112-123.
[7] HE K, GKIOXARI G, DOLLAR P,et al. Mask R-CNN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017.
[8] RUNZ M, AGAPITO L. Co-fusion: real-time segmentation, tracking and fusion of multiple objects[C]//Proceedings of 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore: IEEE, 2017: 4471-4478.
[9] RUNZ M, BUFFIER M, AGAPITO L. MaskFusion: real-time recognition, tracking and reconstruction of multiple moving objects[C]//Proceedings of 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany: IEEE, 2019.
[10] 杨现辉, 王惠南. ICP算法在3D点云配准中的应用研究[J]. 计算机仿真, 2010, 27(8):235-238. YANG Xianhui, WANG Huinan. Application of ICP algorithm in 3D point cloud registration [J]. Computer simulation, 2010, 27(8): 235-238
[11] MISRA I, MOORTHI S M, DHAR D, et al. An automatic satellite image registration technique based on Harris corner detection and random sample consensus (RANSAC) outlier rejection model[C]//Proceedings of 2012 International Conference on Recent Advances in Information Technology. [S.l.]: IEEE, 2012.
[12] BAKER S, MATTHEWS I. Lucas-Kanade 20 years on: a unifying framework[J]. International Journal of Computer Vision, 2004, 56(3):221-255.
[13] ZHANG Q, WANG Y, ZHU L, et al. Research on real-time reasoning based on jetson TX2 heterogeneous acceleration YOLOv4[C]//Proceedings of 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA). [S.l.]: IEEE, 2021.
[14] KAMNITSAS K, LEDIG C, NEWCOMBE V, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation[J]. Medical Image Analysis, 2016, 36:61.
[15] MUR-ARTAL R, MONTIEL J M M, TARDOS J D. ORB-SLAM: A versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2015, 31(5):1147-1163.
[16] 吴刚, 武春风, 侯晴宇,等. 基于不变矩特征匹配的目标定位方法[J]. 光学精密工程, 2009, 17(2):460-468. WU Gang, WU Chunfeng, HOU Qingyu, et al. Target location method based on invariable moment feature matching[J]. Guangxue Jingmi Gongcheng/Optics and Precision Engineering, 2009, 17(2):460-468.
[17] 胡树楷, 杨洋, 王新梅. 几种LDPC码的最小汉明距离的计算[J]. 北京邮电大学学报, 2012, 35(2):1-4. HU Shukai, YANG Yang, WANG Xinmei. Calculation of minimum hamming distance of several LDPC codes[J]. Journal of Beijing University of Posts and Telecommunications, 2012, 35(2):1-4.
[18] DEHURI S, ROY R, CHO S B, et al. An improved swarm optimized functional link artificial neural network (ISO-FLANN) for classification[J]. Journal of Systems & Software, 2012, 85(6):1333-1345.
[19] CARON F, DUFLOS E, POMORSKI D, et al. GPS/IMU data fusion using multisensor kalman filtering: introduction of contextual aspects[J]. Information Fusion, 2017, 7(2):221-230.
[20] 赵玮, 王强, 何晓晖,等. 基于BA优化核参数的非线性Volterra滤波方法研究[J]. 仪表技术与传感器, 2019(6):77-81. ZHAO Wei, WANG Qiang, HE Xiaohui, et al. Research on nonlinear volterra filtering method based on BA Optimization of kernel parameters[J]. Instrument Technique and Sensor, 2019(6):77-81.
[21] 熊磊, 刘国栋, 刘小勇,等. 数字图像相关中基于非线性灰度改变模型的灰度梯度迭代算法[J]. 机床与液压, 2012, 40(13):70-70. XIONG Lei, LIU Guodong, LIU Xiaoyong, et al. Grayscale gradient iteration algorithm based on nonlinear grayscale change model in digital image correlation[J]. Machine Tool & Hydraulics, 2012, 40(13):70-70.
[22] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of 2014 European Conference on Computer Vision. [S.l.]: Springer International Publishing,2014.
[23] LONG SUN,TAO WU,GUANGCAI SUN,et al.object detection research of SAR image using improved faster regionbased convolutional neural network[J].Journal of Geodesy and Geoinformation Science,2020,3(3):18-28.
[24] HE K, ZHANG X, REN S, et al. Identity mappings in deep residual networks[C]//Proceedings of 2016 European Conference on Computer Vision. [S.l.]: Springer International Publishing, 2016.
[25] JIANG J, FU X, QIN R, et al. High-speed lightweight ship detection algorithm based on YOLO-V4 for three-channels RGB SAR image[J]. Remote Sensing, 2021, 13(10):1909.
[26] JIANYA GONG,SHUNPING JI.Photogrammetry and deep learning[J].Journal of Geodesy and Geoinformation Science,2018(01):1-15.
[27] VINYALS O,TOSHEV A,BENGIO S, et al. Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016, 39(4):652-663.