Visual odometry optimizing bounded with semantic elements association in dynamic scenes

doi:10.11947/j.AGCS.2021.20210308

Abstract

Abstract: Cameras can help to make low-cost positioning and environment perceiving in autopilot but dynamic objects give negative effects to visual odometry. This paper gives a model to optimize it based on semantic elements association. It uses such techniques as objects detection and semantic segmentation to identify semantic objects and distinguish dynamic semantic elements(DSE) from static ones. Then it filters bad keypoints by a dynamic feature mask in visual positioning. In practice, this proposed method detects DSE even when there are objects with duality of moving and static. In an experiment on campus roads, negative influences were found especially when a robot turning around or a moving object crossing its camera view. It showed that average accuracy in detecting DSE was 87% and the largest difference between trajectories before and after semantic association optimizing in sub-sequences was 2.463 m. Compared to ground truth, the RMSE of proposed method dropped by 38%.

Key words: visual odometry, SLAM, dynamic scenes, semantic association, semantic segmentation

CLC Number:

P237

SHAO Xiaohang, WU Hangbin, LIU Chun, CHEN Chen, CAI Tianchi, CHENG Fanjin. Visual odometry optimizing bounded with semantic elements association in dynamic scenes[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1478-1486.

References

[1] PADEN B, ČÁP M, YONG S Z, et al. A survey of motion planning and control techniques for self-driving urban vehicles[J]. IEEE Transactions on Intelligent Vehicles, 2016, 1(1): 33-55.
[2] 张一, 姜挺, 江刚武, 等. 特征法视觉SLAM逆深度滤波的三维重建[J]. 测绘学报, 2019, 48(6): 708-717. DOI: 10.11947/j.AGCS.2019.20180421. ZHANG Yi, JIANG Ting, JIANG Gangwu, et al. 3D reconstruction with inverse depth filter of feature-based visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 708-717. DOI: 10.11947/j.AGCS.2019.20180421.
[3] 王晨捷, 罗斌, 李成源, 等. 无人机视觉SLAM协同建图与导航[J]. 测绘学报, 2020, 49(6): 767-776. DOI: 10.11947/j.AGCS.2020.20190145. WANG Chenjie, LUO Bin, LI Chengyuan, et al. The collaborative mapping and navigation based on visual SLAM in UAV platform[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(6): 767-776. DOI: 10.11947/j.AGCS.2020.20190145.
[4] KLEIN G, MURRAY D. Parallel tracking and mapping for small AR workspaces[C]//Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara, Japan: IEEE, 2007: 225-234.
[5] MUR-ARTAL R, MONTIEL J M M, TARDÓS J D. ORB-SLAM: a versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2015, 31(5): 1147-1163.
[6] MUR-ARTALR, TARDÓS J D. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
[7] CAMPOS C,ELVIRA R, RODRÍGUEZ J J G, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(1): 1-17.
[8] NEWCOMBE R A, LOVEGROVE S J, DAVISON A J. DTAM: Dense tracking and mapping in real-time[C]//Proceedings of 2011 International Conference on Computer Vision. Barcelona, Spain: IEEE, 2011: 2320-2327.
[9] FORSTER C, PIZZOLI M, SCARAMUZZA D. SVO: Fast semi-direct monocular visual odometry[C]//Proceedings of 2014 IEEE International Conference on Robotics and Automation (ICRA). Hong Kong, China: IEEE, 2014: 15-22.
[10] BAHRAINI M S, BOZORG M, RAD A B. SLAM in dynamic environments via ML-RANSAC[J]. Mechatronics, 2018, 49: 105-118.
[11] STRECKE M, STUCKLER J. EM-fusion: Dynamic object-level SLAM with probabilistic data association[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 5864-5873.
[12] 弋英民, 刘丁. 动态环境下基于路径规划的机器人同步定位与地图构建[J]. 机器人, 2010, 32(1): 83-90. YI Yingmin, LIU Ding. Robot simultaneous localization and mapping based on path planning in dynamic environments[J]. Robot, 2010, 32(1): 83-90.
[13] ÖZKAN E, WAHLSTRÖM N, GODSILL S J. Rao-Blackwellised particle filter for star-convex extended target tracking models[C]//Proceedings of the 19th International Conference on Information Fusion (FUSION). Heidelberg, Germany: IEEE, 2016: 1193-1199.
[14] 龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6): 693-704. DOI: 10.11947/j.AGCS.2018.20170640. GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 693-704. DOI: 10.11947/j.AGCS.2018.20170640.
[15] 邸凯昌, 万文辉, 赵红颖, 等. 视觉SLAM技术的进展与应用[J]. 测绘学报, 2018, 47(6): 770-779. DOI: 10.11947/j.AGCS.2018.20170652. DI Kaichang, WAN Wenhui, ZHAO Hongying, et al. Progress and applications of visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 770-779. DOI: 10.11947/j.AGCS.2018.20170652.
[16] YU C, LIU Z X, LIU X J, et al. DS-SLAM: a semantic visual SLAM towards dynamic environments[C]//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain: IEEE, 2018: 1168-1174.
[17] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[18] BESCOS B, FÁCIL J M, CIVERA J, et al. DynaSLAM: tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.
[19] HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 2980-2988.
[20] RUNZ M, BUFFIER M, AGAPITO L. MaskFusion: real-time recognition, tracking and reconstruction of multiple moving objects[C]//Proceedings of 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany: IEEE, 2018: 10-20.
[21] 李琳辉, 张溪桐, 连静, 等. 结合道路结构化特征的语义SLAM算法[J]. 哈尔滨工业大学学报, 2021, 53(2): 175-183. LI Linhui, ZHANG Xitong, LIAN Jing, et al. Semantic SLAM algorithm combined with road structured features[J]. Journal of Harbin Institute of Technology, 2021, 53(2): 175-183.
[22] ZHONG F W, WANG S, ZHANG Z Q, et al. Detect-SLAM: Making object detection and SLAM mutually beneficial[C]//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV, USA: IEEE, 2018: 1001-1010.
[23] KONOLIGE K, AGRAWAL M. FrameSLAM: From bundle adjustment to real-time visual mapping[J]. IEEE Transactions on Robotics, 2008, 24(5): 1066-1077.
[24] WESTOBY M J, BRASINGTON J, GLASSER N F, et al. “Structure-from-Motion” photogrammetry: a low-cost, effective tool for geoscience applications[J]. Geomorphology, 2012, 179: 300-314.
[25] 黄瑞, 张轶. 高适应性激光雷达SLAM[J]. 电子科技大学学报, 2021, 50(1): 52-58. HUANG Rui, ZHANG Yi. High adaptive LiDAR simultaneous localization and mapping[J]. Journal of University of Electronic Science and Technology of China, 2021, 50(1): 52-58.
[26] HULETSKI A, KARTASHOV D, KRINKIN K. Evaluation of the modern visual SLAM methods[C]//Proceedings of 2015 Artificial Intelligence and Natural Language and Information Extraction, Social Media and Web Search FRUCT Conference (AINL-ISMW FRUCT). St. Petersburg, Russia: IEEE, 2015: 19-25.