多镜头组合式相机的全景SLAM

doi:10.11947/j.AGCS.2019.20180443

摘要/Abstract

摘要： 同步定位与地图构建（simultaneous localization and mapping，SLAM）是摄影测量、计算机视觉和机器人学的研究热点，并广泛应用于移动测图系统、机器人、无人驾驶车等。本文提出一种针对多镜头组合式全景相机的基于特征的SLAM解决方案。首先，本文建立了鱼眼相机的高精度检校模型，以保证鱼眼相机与全景相机之间的高精度坐标转换；然后，将多镜头组合式全景成像模型嵌入SLAM的初始化、局部地图生成、关键帧选取、图优化等各个流程中。此外，考虑全景相机变形大、基线长的不利因素，本文在特征匹配、平差、特征点跟踪等SLAM的各个步骤都进行了针对性改进。本文在两套车载全景数据集共8000余张全景影像上进行试验。结果表明，本文所提出的全景SLAM很好地实现了全景相机的自动定位与地图构建功能，并达到了接近GPS参考的极高定位精度而无须借助GPS/IMU组合导航系统。相对于主流的基于平面相机的各类SLAM系统，如Mono-SLAM、Stereo-SLAM以及RGB-D SLAM，本文提出的全景SLAM可作为良好的补充，并为GPS信号失锁时的传感器定位提供廉价的自动解决方案。

关键词: SLAM, 全景相机, 鱼眼相机, 相机检校

Abstract: Simultaneous localization and mapping (SLAM) is a research hotspot in fields of photogrammetry, computer vision and robotics, and has been widely applied in mobile mapping system, robots, driverless car, etc. This paper presents a fully automated feature based SLAM solution for a panoramic imaging system consisted of multi-camera rig. First, we developed a fisheye camera calibration model for guaranteeing high accurate coordinate transformation between the fisheye camera and the panoramic camera. Second, we imbedded the panoramic camera model into the SLAM process including initialization, local map building, key frame selection, graph optimization and bundle adjustment. In addition, we developed the algorithm in the processes of feature matching, bundle adjustment, frame tracking considering the disadvantages from the large image distortion and long baseline of the panoramic camera system. Experiments are executed on two data sets with more than 8000 panoramic images. Results show that the proposed panoramic SLAM solution achieves automatic camera localization and map construction, and the localization accuracy approaches the GPS reference. With respect to the mainstream SLAM systems based on conventional cameras, such as Mono-SLAM, Stereo-SLAM and RGB-D SLAM, our proposed panoramic SLAM system could serve as a beneficial supplement and supplies a cheap solution for GPS denied localization problem.

Key words: SLAM, panoramic camera, fisheye camera, camera calibration

中图分类号:

P237

季顺平, 秦梓杰. 多镜头组合式相机的全景SLAM[J]. 测绘学报, 2019, 48(10): 1254-1265.

JI Shunping, QIN Zijie. Panoramic SLAM for multi-camera rig[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10): 1254-1265.

参考文献 34

[1]	龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640. GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640.
[2]	邸凯昌, 万文辉, 赵红颖, 等. 视觉SLAM技术的进展与应用[J]. 测绘学报, 2018, 47(6):770-779. DOI:10.11947/j.AGCS.2018.20170652. DI Kaichang, WANG Wenhui, ZHAO hongying, et al. Progress and application of visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):770-779.DOI:10.11947/j.AGCS.2018. 20170652.
[3]	LIU H M, ZHANG G F, BAO H J. A survey of monocular simultaneous localization and mapping[J]. Journal of Computer-Aided Design & Computer Graphics, 2016, 28(6):855-868.
[4]	DAVISON A J, REID I D, MOLTON N D, et al. MonoSLAM:real-time single camera SLAM[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):1052-1067.
[5]	MUR-ARTAL R, MONTIEL J M M, TARDÓS J D. ORB-SLAM:A versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2017, 31(5):1147-1163.
[6]	ENGEL J, SCHÖPS T, CREMERS D. LSD-SLAM:large-scale direct monocular SLAM[M]//FLEET D, PAJDLA T, SCHIELE B, et al. Computer Vision-ECCV 2014. Zurich:Cham- Springer, 2014:834-849.
[7]	FORSTER C, PIZZOLI M, SCARAMUZZA D. SVO:Fast semi-direct monocular visual odometry[C]//Proceedings of IEEE International Conference on Robotics and Automation. Hong Kong:IEEE, 2014:15-22.
[8]	FORSTER C, ZHANG Z C, GASSNER M, et al. SVO:semidirect visual odometry for monocular and multicamera systems[J]. IEEE Transactions on Robotics, 2017, 33(2):249-265.
[9]	GAO X, ZHANG T. Robust RGB-D simultaneous localization and mapping using planar point features[J]. Robotics and Autonomous Systems, 2015, 72(10):1-14.
[10]	NEWMAN P, HO K. SLAM-loop closing with visually salient features[C]//Proceedings of 2005 IEEE International Conference on Robotics and Automation. Barcelona:IEEE, 2005:635-642.
[11]	RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB:An efficient alternative to SIFT or SURF[C]//Proceedings of 2011 International Conference on Computer Vision. Barcelona:IEEE, 2011:2564-2571.
[12]	季顺平, 史云. 多镜头组合型全景相机两种成像模型的定位精度比较[J]. 测绘学报, 2014, 43(12):1252-1258. DOI:10.13485/j.cnki.11-2089.2014.0169. JI Shunping, SHI Yun. Comparison of two sensor models for multi-camera rig system in measurements[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12):1252-1258. DOI:10.13485/j.cnki.11-2089.2014.0169.
[13]	季顺平, 史云. 高噪声环境下基于参考影像的车载序列影像定位方法[J]. 测绘学报, 2014, 43(11):1174-1181. DOI:10.13485/j.cnki.11-2089.2014.0181. JI Shunping, SHI Yun. Georegistration of ground sequential imagery with geo-referenced aerial images in high noise environments[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(11):1174-1181. DOI:10.13485/j.cnki.11-2089.2014.0181.
[14]	RITUERTO A, PUIG L, GUERRERO J J. Visual SLAM with an omnidirectional camera[C]//Proceedings of the 20th International Conference on Pattern Recognition. Istanbul, Turkey:IEEE, 2010:348-351.
[15]	LEMAIRE T, LACROIX S. SLAM with panoramic vision[J]. Journal of Field Robotics, 2010, 24(1-2):91-111.
[16]	JI Shunping, SHI Yun, SHAN Jie, et al. Particle filtering methods for georeferencing panoramic image sequence in complex urban scenes[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 105(7):1-12.
[17]	CARUSO D, ENGEL J, CREMERS D. Large-scale direct SLAM for omnidirectional cameras[C]//Proceedings of 2015 IEEE/RSJ IEEE International Conference on Intelligent Robots and Systems. Hamburg:IEEE, 2015:141-148.
[18]	谢东海, 钟若飞, 吴俣, 等. 球面全景影像相对定向与精度验证[J]. 测绘学报, 2017, 46(11):1822-1829. DOI:10.11947/j.AGCS.2017.20160645. XIE Donghai, ZHONG Ruofei, WU Yu, et al. Relative pose estimation and accuracy verification of spherical panoramic image[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11):1822-1829. DOI:10.11947/j.AGCS.2017.20160645.
[19]	闫利, 王奕丹. 扩展共线方程结合车载组合全景相机影像区域网平差[J]. 测绘学报, 2017, 46(4):460-467. DOI:10.11947/j.AGCS.2017.20160464. YAN Li, WANG Yidan. Block adjustment of vehicle-borne multi-camera rig images using extended collinearity equations[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(4):460-467. DOI:10.11947/j.AGCS.2017.20160464.
[20]	石丽梅, 赵红蕊, 李明海, 等. 车载移动测图系统外方位元素标定方法[J]. 测绘学报, 2015, 44(1):52-58. DOI:10.11947/j.AGCS.2015.20130203. SHI Limei, ZHAO Hongrui, LI Minghai, et al. Extrinsic calibration for vehicle-based mobile mapping system[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(1):52-58. DOI:10.11947/j.AGCS.2015.20130203.
[21]	GEYER C, DANIILIDIS K. Catadioptric camera calibration[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra, Greece, Greece:IEEE, 1999:398-404.
[22]	SATO T, IKEDA S, YOKOYA N. Extrinsic camera parameter recovery from multiple image sequences captured by an omni-directional multi-camera system[C]//Proceedings of European Conference on Computer Vision. Prague, Czech Republic:Springer, 2004:326-340.
[23]	MUR-ARTAL R, TARDÓS J D. ORB-SLAM2:An open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5):1255-1262.
[24]	IKEDA S, SATO T, YOKOYA N. High-resolution panoramic movie generation from video streams acquired by an omnidirectional multi-camera system[C]//Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI2003. Tokyo, Japan:IEEE, 2003.
[25]	GUTIERREZ D, RITUERTO A, MONTIEL J M M, et al. Adapting a real-time monocular visual SLAM from conventional to omnidirectional cameras[C]//Proceedings of 2011 IEEE International Conference on Computer Vision Workshops. Barcelona, Spain:IEEE, 2011:343-350.
[26]	PAYÁ L, FERNÁNDEZ L, GIL A, et al. Map building and monte carlo localization using global appearance of omnidirectional images[J]. Sensors, 2010, 10(12):11468-11497.
[27]	CUMMINS M, NEWMAN P. FAB-MAP:appearance-based place recognition and mapping using a learned visual vocabulary model[C]//Proceedings of the 27th International Conference on Machine Learning.Haifa, Israel:[s.n.], 2010.
[28]	SATO T, YOKOYA N. Efficient hundreds-baseline stereo by counting interest points for moving omni-directional multi-camera system[J]. Journal of Visual Communication and Image Representation, 2010, 21(5-6):416-426.
[29]	LEPETIT V, MORENO-NOGUER F, FUA P. EPnP:an accurate O(n) solution to the PnP problem[J]. International Journal of Computer Vision, 2009, 81(2):155-166.
[30]	KÜMMERLE R, GRISETTI G, STRASDAT H, et al. G2o:a general framework for graph optimization[C]//Proceedings of 2011 IEEE International Conference on Robotics and Automation. Shanghai, China:IEEE, 2011:3607-3613.
[31]	STRASDAT H, MONTIEL J M M, DAVISON A J. Scale drift-aware large scale monocular SLAM[M]//Robotics:Science and Systems. Zurich:The MIT Press, 2010.
[32]	HORN B K P, HILDENH M, NEGAHDARIPOUR S. Closed-form solution of absolute orientation using orthonormal matrices[J]. Journal of the Optical Society of America A, 1998, 5(7):1127-1135.
[33]	KANNALA J, BRANDT S S. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(8):1335-1340.
[34]	ZHANG Guofeng, LIU Haomin, DONG Zilong, et al. Efficient non-consecutive feature tracking for robust structure-from-motion[J]. IEEE Transactions on Image Processing, 2016, 25(12):5957-5970.