Position-attitude calculation of panoramic image based on point-line feature combination

doi:10.11947/j.AGCS.2023.20210565

Abstract

Abstract: At present, the position and attitude parameters of panoramic images are mostly solved by point features, while the line features commonly existing in the scene have not been fully utilized. In this paper, a solution method based on point-line feature combination is proposed, which can not only be used to solve the position and attitude of panoramic image in the scene with missing point features, but also improve the accuracy and robustness in the scene with sufficient point features. The line feature in this method is represented by any two points on the line, which does not require the corresponding relationship between the panoramic image and the 3D scene, so it is easy to select and has great practicability. Firstly, the direct linear transform (DLT) is used to construct the point-line feature combination model of panoramic image, and the simplified model is obtained for horizontal and vertical lines; Then, using the simulated road scene, the applicability of the model is analyzed from the two aspects:different combinations of feature points and lines, large attitude angle, and the tolerance is analyzed by manually introducing different types and magnitude of point-line errors; Finally, this method is applied to the fusion of panoramic image and LiDAR points. It is proved that this method of point-line feature combination is better than the simple point method in accuracy, robustness and tolerance.

Key words: panoramic image, point-line feature combination, position and attitude calculation, simulation analysis, panoramic image and LiDAR points fusion

CLC Number:

P227

ZHU Ningning, YANG Bisheng, CHEN Chi, DONG Zhen. Position-attitude calculation of panoramic image based on point-line feature combination[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(2): 218-229.

References

[1] LAN Jinhui, LI Jian, HU Guangda, et al. Distance estimation using a panoramic sensor based on a novel spherical optical flow algorithm[J]. Optics & Laser Technology, 2013, 45:168-176.
[2] TEHRANI M H, GARRATT M A, ANAVATTI S. Horizon-based attitude estimation from a panoramic vision sensor[J]. IFAC Proceedings Volumes, 2012, 45(1):185-188.
[3] 李德仁. 论地球空间信息的三维可视化:基于图形还是基于影像[J]. 测绘学报, 2010, 39(2):111-114. LI Deren. 3D visualization of geospatial information:graphics based or imagery based[J]. Acta Geodaetica et Cartographica Sinica, 2010, 39(2):111-114.
[4] ALMARAZ-CABRAL C C, GONZALEZ-BARBOSA J J, VILLA J, et al. Fringe projection profilometry for panoramic 3D reconstruction[J]. Optics and Lasers in Engineering, 2016, 78:106-112.
[5] BRENNAN P F, PONTO K, CASPER G, et al. Virtualizing living and working spaces:proof of concept for a biomedical space-replication methodology[J]. Journal of Biomedical Informatics, 2015, 57:53-61.
[6] 杨化超, 张磊, 姚国标, 等. 局部单应约束的高精度图像自动配准方法[J]. 测绘学报, 2012, 41(3):401-408. YANG Huachao, ZHANG Lei, YAO Guobiao, et al. An automated image registration method with high accuracy based on local homography constraint[J]. Acta Geodaetica et Cartographica Sinica, 2012, 41(3):401-408.
[7] 吴凡路, 刘建军, 任鑫, 等. 嫦娥三号全景相机图像全景镶嵌方法的研究[J]. 光学学报, 2014, 34(9):182-190. WU Fanlu, LIU Jianjun, REN Xin, et al. Research on panoramic mosaics method of Chang'E-3 Pancam images[J]. Acta Optica Sinica, 2014, 34(9):182-190.
[8] 霍春宝, 童帅, 赵立辉, 等. SIFT特征匹配的显微全景图拼接[J]. 辽宁工程技术大学学报(自然科学版), 2015, 34(1):93-96. HUO Chunbao, TONG Shuai, ZHAO Lihui, et al. Microscopic panorama splicing based on sift features matching[J]. Journal of Liaoning Technical University (Natural Science), 2015, 34(1):93-96.
[9] 黄立勤, 陈财淦. 全景图拼接中图像融合算法的研究[J]. 电子与信息学报, 2014, 36(6):1292-1298. HUANG Liqin, CHEN Caigan. Study on image fusion algorithm of panoramic image stitching[J]. Journal of Electronics & Information Technology, 2014, 36(6):1292-1298.
[10] KIM D H, YOON Y I, CHOI J S. An efficient method to build panoramic image mosaics[J]. Pattern Recognition Letters, 2003, 24(14):2421-2429.
[11] CHEN Haipeng, SHEN Xuanjing, LI Xiaofei, et al. Bionic mosaic method of panoramic image based on compound eye of fly[J]. Journal of Bionic Engineering, 2011, 8(4):440-448.
[12] GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Journal of Geodesy and Geoinformation Science, 2018(1):1-15.
[13] XU Yusheng, HUANG Huang, TONG Tong, et al. Exploiting robust estimators in phase correlation of 3D point clouds for 6 DoF pose estimation[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3):72-90.
[14] 季顺平, 史云. 车载全景相机的影像匹配和光束法平差[J]. 测绘学报, 2013, 42(1):94-100, 107. JI Shunping, SHI Yun. Image matching and bundle adjustment using vehicle-based panoramic camera[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(1):94-100, 107.
[15] 谢东海, 钟若飞, 吴俣, 等. 球面全景影像相对定向与精度验证[J]. 测绘学报, 2017, 46(11):1822-1829.DOI:10.11947/j.AGCS.2017.20160645. XIE Donghai, ZHONG Ruofei, WU Yu, et al. Relative pose estimation and accuracy verification of spherical panoramic image[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11):1822-1829.DOI:10.11947/j.AGCS.2017.20160645.
[16] NISTER D. An efficient solution to the five-point relative pose problem[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(6):756-770.
[17] 季顺平, 史云. 多镜头组合型全景相机两种成像模型的定位精度比较[J]. 测绘学报, 2014, 43(12):1252-1258. JI Shunping, SHI Yun. Comparison of two sensor models for multi-camera rig system in measurements[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12):1252-1258.
[18] 朱宁宁. 柱面全景图像拼接方法的仿真分析[J]. 测绘学报, 2017, 46(4):487-497.DOI:10.11947/j.AGCS.2017.20160456. ZHU Ningning. Simulation analysis of cylindrical panoramic image mosaic[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(4):487-497.DOI:10.11947/j.AGCS.2017.20160456.
[19] 马洪超, 姚春静, 邬建伟. 利用线特征进行高分辨率影像与LiDAR点云的配准[J]. 武汉大学学报(信息科学版), 2012, 37(2):136-140, 159. MA Hongchao, YAO Chunjing, WU Jianwei. Registration of LiDAR point clouds and high resolution images based on linear features[J]. Geomatics and Information Science of Wuhan University, 2012, 37(2):136-140, 159.
[20] 何培培, 万幼川, 杨威, 等. 基于线特征的城区激光点云与影像自动配准[J].光学学报, 2015, 35(5):360-368. HE Peipei, WAN Youchuan, YANG Wei, et al. Automatic registration of urban laser point cloud with aerial image data based on straight-lines[J]. Acta Optica Sinica, 2015, 35(5):360-368.
[21] 陈驰, 杨必胜, 彭向阳. 低空UAV激光点云和序列影像的自动配准方法[J]. 测绘学报, 2015, 44(5):518-525.DOI:10.11947/j.AGCS.2015.20130558. CHEN Chi, YANG Bisheng, PENG Xiangyang. Automatic registration of low altitude UAV sequent images and laser point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(5):518-525.DOI:10.11947/j.AGCS.2015.20130558.
[22] ZHU Ningning, JIA Yonghong, JI Shunping. Registration of panoramic/fish-eye image sequence and LiDAR points using skyline features[J]. Sensors (Basel, Switzerland), 2018, 18(5):1651.
[23] 闫利, 索一凡, 曹亮. 车载激光雷达点云与全景影像的自动配准[J]. 测绘科学, 2016, 41(4):113-117, 123. YAN Li, SUO Yifan, CAO Liang. Registration of vehicle-borne LiDAR point cloud and panoramic image[J]. Science of Surveying and Mapping, 2016, 41(4):113-117, 123.
[24] YANG Bisheng, CHEN Chi. Automatic registration of UAV-borne sequent images and LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 101(3):262-274.
[25] ZHU Ningning. Simulation analysis of spherical panoramic mosaic[J]. Signal Processing, 2019, 158(5):190-200.
[26] 杨必胜, 董震. 点云智能处理[M]. 北京:科学出版社, 2020. YANG Bisheng, DONG Zhen. Point cloud intelligent processing[M]. Beijing:Science Press, 2020.
[27] 杨必胜, 梁福逊, 黄荣刚. 三维激光扫描点云数据处理研究进展、挑战与趋势[J]. 测绘学报, 2017, 46(10):1509-1516.DOI:10.11947/j.AGCS.2017.20170351. YANG Bisheng, LIANG Fuxun, HUANG Ronggang. Progress, challenges and perspectives of 3D LiDAR point cloud processing[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10):1509-1516.DOI:10.11947/j.AGCS.2017.20170351.
[28] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019, 48(12):1575-1585.DOI:10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1575-1585.DOI:10.11947/j.AGCS.2019.20190465.
[29] WANG Wang, LI Shaochun, WANG Wang, et al.. A simple deep learning network for classification of 3D mobile LiDAR point clouds[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3):49-59.
[30] ZHU Ningning, JIA Yonghong, HUANG Xia. Semiautomatically register MMS LiDAR points and panoramic image sequence using road lamp and lane[J]. Photogrammetric Engineering & Remote Sensing, 2019, 85(11):829-840.
[31] ZHU Ningning, YANG Bisheng, DONG Zhen, et al. Automatic registration of mobile mapping system lidar points and panoramic-image sequences by relative orientation model[J].Photogrammetric Engineering & Remote Sensing, 2021,87(12):913-922.