A thermal-inertial odometry with point and line fusion for the weak textured dark scenes

doi:10.11947/j.AGCS.2025.20240251

Abstract

Abstract:

Traditional visual SLAM performs poorly or even fails to work in challenging environments such as significant changes in light conditions, darkness, smoke, and fog. In contrast, infrared cameras possess greater anti-interference capabilities. Nevertheless, the performance of infrared SLAM is severely affected by the poor imaging quality and noise of infrared cameras. In this paper, a point and line fusion infrared inertial odometry method is proposed, which is based on the thermal radiation imaging characteristics of infrared cameras and considers the weak texture characteristics in structured scenes. In the front-end, a point tracking algorithm based on the optical flow method is employed with a filtering process to eliminate unstable point features. The LSD algorithm is enhanced to extract stable line features, and line feature tracking is conducted using the LBD descriptor. A sliding window in the back-end is used to construct a tightly coupled graph optimization model that includes point, line, and IMU information. Finally, validation is conducted using open-source datasets and measured data from underground garages, respectively. The results demonstrate that the accuracy and robustness of the point-line combined thermal inertial odometer are significantly improved compared to traditional visual SLAM algorithms, which helps unmanned systems achieve robust localization in dark and weakly textured scenes.

Key words: SLAM, weak texture scenes, long-wave infrared, point and line fusion, monocular thermal-inertial odometry

CLC Number:

P232

Luguang LAI, Dongqing ZHAO, Linyang LI, Wenzhe FAN, Xiongqing LI, Pengfei LI. A thermal-inertial odometry with point and line fusion for the weak textured dark scenes[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(3): 448-460.

Figures/Tables 18

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Tab. 1

Fig. 6

Fig. 7

Tab. 2

Tab. 3

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Tab. 4

Fig. 14

References 44

[1]	张继贤, 刘飞. 视觉SLAM环境感知技术现状与智能化测绘应用展望[J]. 测绘学报, 2023, 52(10): 1617-1630. DOI:. doi: 10.11947/j.AGCS.2023.20220240
	ZHANG Jixian, LIU Fei. Review of visual SLAM environment perception technology and intelligent surveying and mapping application[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(10): 1617-1630. DOI:. doi: 10.11947/j.AGCS.2023.20220240
[2]	LI Xingxing, ZHANG Xiaohong, NIU Xiaoji, et al. Progress and achievements of multi-sensor fusion navigation in China during 2019—2023[J]. Journal of Geodesy and Geoinformation Science, 2023, 6(3): 102-114.
[3]	CADENA C, CARLONE L, CARRILLO H, et al. Past, present, and future of simultaneous localization and mapping: toward the robust-perception age[J]. IEEE Transactions on Robotics, 2016, 32(6): 1309-1332.
[4]	杨元喜. 综合PNT体系及其关键技术[J]. 测绘学报, 2016, 45(5): 505-510. DOI:. doi: 10.11947/j.AGCS.2016.20160127
	YANG Yuanxi. Concepts of comprehensive PNT and related key technologies[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(5): 505-510. DOI:. doi: 10.11947/j.AGCS.2016.20160127
[5]	WU Yuzhen, WANG Lingxue, ZHANG Lian, et al. Catadioptric omnidirectional thermal odometry in dynamic environment[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2024, 216: 45-65.
[6]	YANG Yuanxi. Resilient PNT concept frame[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(3): 1-7.
[7]	DI Kaichang, WAN Wenhui, ZHAO Hongying, et al. Progress and applications of visual SLAM[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 38-49.
[8]	KONG Da, ZHANG Yu, DAI Weichen. Direct near-infrared-depth visual SLAM with active lighting[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7057-7064.
[9]	王铉彬, 李星星, 廖健驰, 等. 基于图优化的紧耦合双目视觉/惯性/激光雷达SLAM方法[J]. 测绘学报, 2022, 51(8): 1744-1756. DOI:. doi: 10.11947/j.AGCS.2022.20210503
	WANG Xuanbin, LI Xingxing, LIAO Jianchi, et al. Tightly-coupled stereo visual-inertial-LiDAR SLAM based on graph optimization[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(8): 1744-1756. DOI:. doi: 10.11947/j.AGCS.2022.20210503
[10]	蔡毅, 王岭雪. 红外成像技术中的9个问题[J]. 红外技术, 2013, 35(11): 671-682.
	CAI Yi, WANG Lingxue. Nine issues associated with infrared imaging technology[J]. Infrared Technology, 2013, 35(11): 671-682.
[11]	EBADI K, BERNREITER L, BIGGIE H, et al. Present and future of SLAM in extreme environments: the DARPA SubT challenge[J]. IEEE Transactions on Robotics, 2023, 40: 936-959.
[12]	VAN MANEN B R, SLUITER V, MERSHA A Y. FirebotSLAM: thermal SLAM to increase situational awareness in smoke-filled environments[J]. Sensors, 2023, 23(17): 7611.
[13]	VIDAS S, SRIDHARAN S. Hand-held monocular SLAM in thermal-infrared[C]//Proceedings of 2012 International Conference on Control Automation Robotics & Vision. Guangzhou: IEEE, 2012: 859-864.
[14]	PAPACHRISTOS C, MASCARICH F, ALEXIS K. Thermal-inertial localization for autonomous navigation of aerial robots through obscurants[C]//Proceedings of 2018 International Conference on Unmanned Aircraft Systems. Dallas: IEEE, 2018: 394-399.
[15]	KHATTAK S, PAPACHRISTOS C, ALEXIS K. Key frame-based direct thermal-inertial odometry[EB/OL]. [2024-06-05]. https://arxiv.org/abs/1903.00798v1.
[16]	CHEN Wenqiang, WANG Yu, CHEN Haoyao, et al. EIL-SLAM: depth-enhanced edge-based infrared-LiDAR SLAM[J]. Journal of Field Robotics, 2022, 39(2): 117-130.
[17]	SHIN Y S, KIM A. Sparse depth enhanced direct thermal-infrared SLAM beyond the visible spectrum[J]. IEEE Robotics and Automation Letters, 2019, 4(3): 2918-2925.
[18]	赖路广, 赵冬青, 李林阳, 等. 长波红外成像技术及其在SLAM中的应用[J]. 测绘科学, 2024, 49(3): 77-86.
	LAI Luguang, ZHAO Dongqing, LI Linyang, et al. Long wave infrared imaging technology and its application in SLAM[J]. Science of Surveying and Mapping, 2024, 49(3): 77-86.
[19]	SAPUTRA M R U, DE GUSMAO P P B, LU C X, et al. DeepTIO: a deep thermal-inertial odometry with visual hallucination[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 1672-1679.
[20]	ZHAO Shibo, WANG Peng, ZHANG Hengrui, et al. TP-TIO: a robust thermal-inertial odometry with deep ThermalPoint[C]//Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Las Vegas: IEEE, 2020: 4505-4512.
[21]	JIANG Jiajun, CHEN Xingxin, DAI Weichen, et al. Thermal-inertial SLAM for the environments with challenging illumination[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 8767-8774.
[22]	WANG Yu, CHEN Haoyao, LIU Yufeng, et al. Edge-based monocular thermal-inertial odometry in visually degraded environments[J]. IEEE Robotics and Automation Letters, 2023, 8(4): 2078-2085.
[23]	HE Ziqi, LI Gang. A visual-inertial navigation coupled localization method based on adaptive point-line feature extraction[J]. IEEE Sensors Journal, 2023, 23(20): 25096-25104.
[24]	LIU Xin, WEN Shuhuan, ZHANG Hong. A real-time stereo visual-inertial SLAM system based on point-and-line features[J]. IEEE Transactions on Vehicular Technology, 2023, 72(5): 5747-5758.
[25]	ZHAO Weiqiang, SUN Hang, ZHANG Xinyu, et al. Visual SLAM combining lines and structural regularities: towards robust localization[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(6): 5047-5064.
[26]	SHEN Shaojie, MICHAEL N, KUMAR V. Tightly-coupled monocular visual-inertial fusion for autonomous flight of rotorcraft MAVs[C]//Proceedings of 2015 IEEE International Conference on Robotics and Automation. Seattle: IEEE, 2015: 5303-5310.
[27]	QIN Tong, LI Peiliang, SHEN Shaojie. VINS-Mono: a robust and versatile monocular visual-inertial state estimator[J]. IEEE Transactions on Robotics, 2018, 34(4): 1004-1020.
[28]	ZHANG Lilian, KOCH R. An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency[J]. Journal of Visual Communication and Image Representation, 2013, 24(7): 794-805.
[29]	GIOI R G, JAKUBOWICZ J, MOREL J M, et al. LSD: a fast line segment detector with a false detection control[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(4): 722-732.
[30]	QIN Tong, CAO Shaozu, PAN Jie, et al. A general optimization-based framework for global pose estimation with multiple sensors[EB/OL]. [2024-01-11]. http://arxiv.org/abs/1901.03642v1.
[31]	BARTOLI A, STURM P. Structure-from-motion using lines: representation, triangulation, and bundle adjustment[J]. Computer Vision and Image Understanding, 2005, 100(3): 416-441.
[32]	FU Qiang, WANG Jialong, YU Hongshan, et al. PL-VINS: real-time monocular visual-inertial SLAM with point and line features[EB/OL]. [2024-04-16]. https://arxiv.org/abs/2009.07462v1.
[33]	LEE J, PARK S Y. PLF-VINS: real-time monocular visual-inertial SLAM with point-line fusion and parallel-line fusion[J]. IEEE Robotics and Automation Letters, 6(4): 7033-7040.
[34]	HE Yijia, ZHAO Ji, GUO Yue, et al. PL-VIO: tightly-coupled monocular visual-inertial odometry using point and line features[J]. Sensors, 2018, 18(4): 1159.
[35]	HUBER P J. Robust estimation of a location parameter[J]. The Annals of Mathematical Statistics, 1964, 35(1): 73-101.
[36]	SIBLEY G, MATTHIES L, SUKHATME G. Sliding window filter with application to planetary landing[J]. Journal of Field Robotics, 2010, 27(5): 587-608.
[37]	FORSTER C, CARLONE L, DELLAERT F, et al. On-manifold preintegration for real-time visual-inertial odometry[J]. IEEE Transactions on Robotics, 2017, 33(1): 1-21.
[38]	WEI Hongyu, ZHANG Tao, ZHANG Liang, et al. Multifeatures association bundle adjustment constraints for visual-oriented localization[J]. IEEE Sensors Journal, 2023, 23(22): 27909-27920.
[39]	YIN Jie, LI Ang, LI Tao, et al. M2DGR: a multi-sensor and multi-scenario SLAM dataset for ground robots[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 2266-2273.
[40]	ZHOU Yuxuan, LI Xingxing, LI Shengyu, et al. Ground-VIO: monocular visual-inertial odometry with online calibration of camera-ground geometric parameters[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(10): 14328-14343.
[41]	ZHANG Zhengyou. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330-1334.
[42]	FURGALE P, REHDER J, SIEGWART R. Unified temporal and spatial calibration for multi-sensor systems[C]//Proceeding of 2013 IEEE/RSJ International Conference on Intelligent Robots & Systems. Tokyo: IEEE, 2013: 1280-1286.
[43]	SHAN Tixiao, ENGLOT B, MEYERS D, et al. LIO-SAM: tightly-coupled LiDAR inertial odometry via smoothing and mapping[C]//Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Las Vegas: IEEE, 2020: 5135-5142.
[44]	LIM H, KIM Y, JUNG K, et al. Avoiding degeneracy for monocular visual SLAM with point and line features[C]//Proceedings of 2021 IEEE International Conference on Robotics and Automation. Xi'an: IEEE, 2021: 11675-11681.

数据集	Hall02	Room_dark04	Street02	Street03	Street06	Street08
光照条件	室内黑暗	室内黑暗	白天	夜晚	夜晚	夜晚
轨迹长度/m	83.0	59.8	330.2	423.9	347.6	340.6

RMSE	Hall02	Room_dark04	Street02	Street03	Street06	Street08
VINS-Fusion1	2.70	0.84	—	8.61	13.28	17.92
PL-VINS1	—	0.82	—	5.64	—	12.44
TIO1	2.25	0.75	2.63	2.71	3.49	5.83
VINS-Fusion2	3.35	1.44	—	9.46	55.74	22.57
PL-VINS2	—	1.31	—	5.84	—	21.00
TIO2	2.97	1.02	8.91	2.86	32.79	11.10

设备	设备型号	参数信息
红外相机	Gaode PLUG 617	分辨率：640×512 水平视场角：92.0° 垂直视场角：70.6° 帧率：25 Hz
IMU	I300BA50	陀螺仪零偏：3°/h 加速度计零偏：100 m Gal 角度随机游走：0.15°/h 速度随机游走：0.02 m/s/h 采样频率：200 Hz

数据集	轨迹长度	VINS-Fusion1	PL-VINS1	TIO1	VINS-Fusion2	PL-VINS2	TIO2
Parking01	330.08	4.89	4.79	4.36	13.36	12.83	9.68
Parking02	229.99	—	—	8.00	—	—	11.98
Parking03	141.93	5.67	5.33	3.72	6.38	8.00	4.95
Parking04	115.69	4.32	3.13	2.64	6.20	5.34	3.74
Parking05	166.30	2.57	2.35	2.01	5.19	4.12	3.56