Cross-modal sensor self-calibration method for highway point-line feature integrated mobile mapping system

doi:10.11947/j.AGCS.2025.20240165

Abstract

Abstract:

The onboard mobile mapping system is affected by pre-calibration errors, installation errors, and camera projection errors, leading to mismatches between the acquired point clouds and panoramic image sequence data. In highway scenarios, existing calibration methods face challenges due to the linear distribution of highways and the significant depth-of-field differences in panoramic images. These factors cause uneven distribution of control points with high correlation, making it difficult to adequately calibrate the camera's exterior orientation parameters, especially the translation components related to the vehicle's roll angle and travel direction. To address this issue, this paper proposes a self-calibration method for panoramic cameras in highway scenarios, integrating point-line features with depth information. The method improves the sensitivity of the adjustment model to translation components through inverse distance weighting, extracts point-line features from highway infrastructure, and constructs a joint adjustment model to reduce the impact of depth differences on uneven feature distribution, ensuring high sensitivity to errors in all directions. Experimental results show that the proposed method can accurately calibrate the exterior orientation parameters of the panoramic camera, with calibration accuracy better than 3 pixels, and the calibration model demonstrates high responsiveness to errors in all directions.

Key words: panoramic camera self-calibration, point-line feature integration, panoramic camera internal orientation elements self-check, mobile mapping system

CLC Number:

P228

Daiwei ZHANG, Xuming GE, Han HU, Qing ZHU, Bo XU, Qiang WANG. Cross-modal sensor self-calibration method for highway point-line feature integrated mobile mapping system[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(4): 760-772.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Tab. 1

Fig. 10

Tab. 2

Fig. 11

Fig. 12

Fig. 13

Fig. 14

References 42

[1]	PUENTE I, GONZÁLEZ-JORGE H, MARTÍNEZ-SÁNCHEZ J, et al. Review of mobile mapping and surveying technologies[J]. Measurement, 2013, 46(7): 2127-2145.
[2]	CUI T, JI S, SHAN J, et al. Line-based registration of panoramic images and LiDAR point clouds for mobile mapping[J]. Sensors (Basel), 2016, 17(1): E70.
[3]	HUSSNAIN Z, ELBERINK S O, VOSSELMAN G. Automatic extraction of accurate 3D tie points for trajectory adjustment of mobile laser scanners using aerial imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 154: 41-58.
[4]	BAO Sheng, SHI Wenzhong, CHEN Pengxin, et al. A systematic mapping framework for backpack mobile mapping system in common monotonous environments[J]. Measurement, 2022, 197: 111243.
[5]	ARMENAKIS C, GAO Yu, GEOMATICS G S. Co-registration of LiDAR and photogrammetric data for updating building databases[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2010, 38(2): 96-100.
[6]	ZHANG Yi, CUI Zhifang. Registration of terrestrial LiDAR and panoramic imagery using the spherical epipolar line and spherical absolute orientation model[J]. IEEE Sensors Journal, 2022, 22(13): 13088-13098.
[7]	PANDEY G, MCBRIDE J R, SAVARESE S, et al. Automatic extrinsic calibration of vision and LiDAR by maximizing mutual information[J]. Journal of Field Robotics, 2015, 32(5): 696-722.
[8]	MILED M, SOHEILIAN B, HABETS E, et al. Hybrid online mobile laser scanner calibration through image alignment by mutual information[J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, 3(1): 25-31.
[9]	WANG Ruisheng, FERRIE F P, MACFARLANE J. Automatic registration of mobile LiDAR and spherical panoramas[C]//Proceedings of 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Providence: IEEE, 2012: 33-40.
[10]	PARMEHR E G, FRASER C S, ZHANG Chunsun, et al. Automatic registration of optical imagery with 3D LiDAR data using statistical similarity[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 88: 28-40.
[11]	GUISLAIN M, DIGNE J, CHAINE R, et al. Fine scale image registration in large-scale urban LiDAR point sets[J]. Computer Vision and Image Understanding, 2017, 157: 90-102.
[12]	ZHAO Yipu, WANG Yuanfang, TSAI Y. 2D-image to 3D-range registration in urban environments via scene categorization and combination of similarity measurements[C]//Proceedings of 2016 IEEE International Conference on Robotics and Automation. Stockholm: IEEE, 2016: 1866-1872.
[13]	CORSINI M, DELLEPIANE M, GANOVELLI F, et al. Fully automatic registration of image sets on approximate geometry[J]. International Journal of Computer Vision, 2013, 102(1): 91-111.
[14]	LEVINSON J, THRUN S. Automatic online calibration of cameras and lasers[C]//Proceedings of 2013 Robotics: Science and Systems. Robotics: Science and Systems Foundation, 2013.
[15]	WANG Yuan, LI Yuhao, CHEN Yiping, et al. Automatic registration of point cloud and panoramic images in urban scenes based on pole matching[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 115: 103083.
[16]	ZHENG Shunyi, HUANG Rongyong, ZHOU Yang. Registration of optical images with LiDAR data and its accuracy assessment[J]. Photogrammetric Engineering & Remote Sensing, 2013, 79(8): 731-741.
[17]	YANG Bisheng, CHEN Chi. Automatic registration of UAV-borne sequent images and LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 101: 262-274.
[18]	ISHIKAWA R, OISHI T, IKEUCHI K. LiDAR and camera calibration using motions estimated by sensor fusion odometry[C]//Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid: IEEE, 2018: 7342-7349.
[19]	LI Na, HUANG Xianfeng, ZHANG Fan, et al. Registration of aerial imagery and LiDAR data in desert areas using sand ridges[J]. The Photogrammetric Record, 2015, 30(151): 263-278.
[20]	LI Jianping, YANG Bisheng, CHEN Chi, et al. Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 136: 41-57.
[21]	HOFMANN S, EGGERT D, BRENNER C. Skyline matching based camera orientation from images and mobile mapping point clouds[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014, II-5: 181-188.
[22]	ALBA M, BARAZZETTI L, SCAIONI M, et al. Automatic registration of multiple laser scans using panoramicrgb and intensity images[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2012, ⅩⅩⅩⅧ-5/W12: 49-54.
[23]	LOWE D G. Distinctive image features from scale-invariant key points[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[24]	MOREL J M, YU Guoshen. ASIFT: a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2): 438-469.
[25]	SCHENK T. From point-based to feature-based aerial triangulation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2004, 58(5/6): 315-329.
[26]	SHENG Qinghong, WANG Qi, ZHANG Xinyue, et al. Registration of urban aerial image and LiDAR based on line vectors[J]. Applied Sciences, 2017, 7(10): 965.
[27]	VON GIOI R G, JAKUBOWICZ J, MOREL J M, et al. LSD: a fast line segment detector with a false detection control[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(4): 722-732.
[28]	ARMENAKIS C, GAO Yu, GEOMATICS G S. Co-registration of LiDAR and photogrammetric data for updating building databases[C]//Proceedings of 2010 Core Spatial Databases-Updating, Maintenance and Services-from Theory to Practice. [S.l.]: IEEE, 2010.
[29]	WEN Chenglu, SUN Xiaotian, HOU Shiwei, et al. Line structure-based indoor and outdoor integration using backpacked and TLS point cloud data[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(11): 1790-1794.
[30]	LI Kai, ZHANG Xudong, LI Kun, et al. Vision global localization with semantic segmentation and interest feature points[C]//Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Las Vegas: IEEE, 2020: 4581-4587.
[31]	TAN Jun, LI Jian, AN Xiangjing, et al. Robust curb detection with fusion of 3D-LiDAR and camera data[J]. Sensors, 2014, 14(5): 9046-9073.
[32]	CHOI S, KIM T, YU W. Performance evaluation of RANSAC family[C]//Proceedings of 2009 British Machine Vision Conference. London: British Machine Vision Association, 2009: 1-12.
[33]	WEI Dong, ZHANG Yongjun, LIU Xinyi, et al. Robust line segment matching across views via ranking the line-point graph[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171: 49-62.
[34]	朱宁宁, 杨必胜, 陈驰, 等. 点-线特征联合的全景图像位姿解算方法[J]. 测绘学报, 2023, 52(2): 218-229. DOI:. doi: 10.11947/j.AGCS.2023.20210565
	ZHU Ningning, YANG Bisheng, CHEN Chi, et al. Position-attitude calculation of panoramic image based on point-line feature combination[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2): 218-229. DOI:. doi: 10.11947/j.AGCS.2023.20210565
[35]	HU Qingyong, YANG Bo, XIE Linhai, et al. RandLA-net: efficient semantic segmentation of large-scale point clouds[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11105-11114.
[36]	ZHANG Daiwei, XU Bo, HU Han, et al. Spherical Hough transform for robust line detection toward a 2D-3D integrated mobile mapping system[J]. Photogrammetric Engineering & Remote Sensing, 2023, 89(5): 311-320.
[37]	JIA Xin, ZHU Qing, GE Xuming, et al. Robust guardrail instantiation and trajectory optimization of complex highways based on mobile laser scanning point clouds[J]. Photogrammetric Engineering & Remote Sensing, 2023, 89(3): 151-161.
[38]	MA Ruifeng, GE Xuming, ZHU Qing, et al. Model-driven precise degradation analysis method of highway marking using mobile laser scanning point clouds[J]. Photogrammetric Engineering & Remote Sensing, 2023, 89(4): 245-258.
[39]	朱宁宁. 柱面全景图像拼接方法的仿真分析[J]. 测绘学报, 2017, 46(4): 487-497. DOI:. doi: 10.11947/j.AGCS.2017.20160456
	ZHU Ningning. Simulation analysis of cylindrical panoramic image mosaic[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(4): 487-497. DOI:. doi: 10.11947/j.AGCS.2017.20160456
[40]	ZHU Ningning. Simulation analysis of spherical panoramic mosaic[J]. Signal Processing, 2019, 158: 190-200.
[41]	季顺平, 秦梓杰. 多镜头组合式相机的全景SLAM[J]. 测绘学报, 2019, 48(10): 1254-1265. DOI:. doi: 10.11947/j.AGCS.2019.20180443
	JI Shunping, QIN Zijie. Panoramic SLAM for multi-camera rig[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10): 1254-1265. DOI:. doi: 10.11947/j.AGCS.2019.20180443
[42]	ZHU Ningning, JIA Yonghong, HUANG Xia. Semiautomatically register MMS LiDAR points and panoramic image sequence using road Lamp and lane[J]. Photogrammetric Engineering & Remote Sensing, 2019, 85(11): 829-40.

采集参数	都汶	丛黑
测量路段长度/km	14	11
车辆行驶速度/（km/h）	40	20
扫描仪扫描频率/w Hz	50	50
扫描仪转速/（rotation/s）	100	150
影像采集频率/（image/s）	1	0.5
全景影像分辨率/像素	4096×2048	4096×2048

		都汶			丛黑
		检校前	SIFT	本文方法	检校前	SIFT	本文方法
改正值	Δx/m	0	—	0.102	0	－0.039	－0.041
	Δy/m	0	—	0.079	0	0.035	0.040
	Δz/m	0	—	0.011	0	－0.059	0.010
	Δyaw/（°）	0	—	0.665	0	－0.475	－0.952
	Δpitch/（°）	0	—	0.332	0	－1.002	0.412
	Δroll/（°）	0	—	－0.097	0	0.002	－0.005
检校结果	Average	8.2	—	2.3	9.1	5.4	2.5
	Max	15	—	4	16	7	4
	RMSE	2.458	—	0.891	2.878	1.237	0.680