The technology and intelligent development of 3D line cloud reconstruction from multiple images

doi:10.11947/j.AGCS.2024.20230447

Abstract

Abstract:

As a collection of line segments, 3D line clouds shave distinct geometric structures and semantic information in each individual feature. They can serve as efficient guiding, controlling, and abstract representation elements in structured 3D reconstruction, compensating for the deficiencies in edge description and lack of initial structure in point clouds. These line clouds represent important structured features that can change the traditional “one-layer skin” 3D model (where different objects are mutually adherent, making spatial analysis and decision-making difficult). However, how to reconstruct useful line clouds from multi-view images and make effective use of them has always been a challenging problem in this field. This article reviews the development of 3D line clouds, introduces related reconstruction methods, and analyzes existing difficulties and shortcomings. Combining the background of transformation from digitization to intelligent surveying and mapping technology, it discusses the three questions of what to build, how to build, and how to use line clouds in real-world 3D scenarios. The article also introduces and prospects the intelligent development of line cloud reconstruction and application, hoping to provide a reference for researchers working on real 3D reconstruction and line clouds.

Key words: 3D line clouds, 3D realistic scene, 3D reconstruction, intelligent surveying and mapping, multiple images

CLC Number:

Dong WEI, Xinyi LIU, Yongjun ZHANG. The technology and intelligent development of 3D line cloud reconstruction from multiple images[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(6): 1025-1036.

Figures/Tables 8

Fig.1

Fig.2

Tab.1

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

References 75

[1]	SUGIURA T, TORII A, OKUTOMI M. 3D surface reconstruction from point-and-line cloud[C]//Proceedings of 2015 International Conference on 3D Vision. [S.l.]: ACM Press, 2015: 264-272.
[2]	ZHANG Zhengyou. Token tracking in a cluttered scene[J]. Image and Vision Computing, 1994, 12(2):110-120.
[3]	MEDIONI G, NEVATIA R. Segment-based stereo matching[J]. Computer Vision, Graphics, and Image Processing, 1985, 31(1):2-18.
[4]	SCHMID C, ZISSERMAN A. Automatic line matching across views[C]//Proceedings of 1997 Conference on Computer Vision and Pattern Recognition. San Juan: IEEE, 1997.
[5]	HARRIS C, STEPHENS M. A combined corner and edge detector[C]//Proceedings of 1988 Alvey Vision Conference. Manchester: Alvey Vision Club, 1988.
[6]	LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[7]	SUN Jianming, SHEN Zehong, WANG Yuang, et al. LoFTR: Detector-free local feature matching with transformers[C]//Proceedings of 2021 Conference on Computer Vision and Pattern Recognition. Virtual: CVPR, 2021.
[8]	陈军, 刘万增, 武昊, 等. 智能化测绘的基本问题与发展方向[J]. 测绘学报, 2021, 50(8):995-1005. DOI:10.11947/j.AGCS.2021.20210235.
	CHEN Jun, LIU Wanzeng, WU Hao, et al. Smart surveying and mapping: fundamental issues and research agenda[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8):995-1005. DOI:10.11947/j.AGCS.2021.20210235.
[9]	陈军, 田海波, 高崟, 等. 实景三维中国的总体架构与主体技术[J/OL]. 测绘学报: 1-20.[2024-06-12].http://kns.cnki.net/kcms/detail/11.2089.P.20240417.0946.002.html.
	CHEN Jun, TIAN Haibo, GAO Yin, et al. China's national 3D mapping program (3dRGLM): overall architecture and key technological issues[J/OL]. Acta Geodaetica et Cartographica Sinica: 1-20.[2024-06-12].http://kns.cnki.net/kcms/detail/11.2089.P.20240417.0946.002.html.
[10]	王广帅, 万一, 张永军. 交叉点结构特征约束的机载LiDAR点云与多视角航空影像配准[J]. 地球信息科学学报, 2020, 22(9):1868-1877.
	WANG Guangshuai, WAN Yi, ZHANG Yongjun. Registration of airborne LiDAR data and multi-view aerial images constrained by junction structure features[J]. Journal of Geo-Information Science, 2020, 22(9):1868-1877.
[11]	唐敏, 张祖勋, 张剑清. 基于广义点理论的多基线影像钣金件3D重建与尺寸检测[J]. 武汉大学学报(信息科学版), 2007, 32(12):1095-1098, 1134.
	TANG Min, ZHANG Zuxun, ZHANG Jianqing. 3D reconstruction and inspection of industrial sheet metal parts with multiple baseline images based on generalized point photogrammetry theory[J]. Geomatics and Information Science of Wuhan University, 2007, 32(12):1095-1098, 1134.
[12]	张永军. 基于广义点摄影测量的圆和圆角矩形三维重建[J]. 哈尔滨工业大学学报, 2008, 40(1):136-140.
	ZHANG Yongjun. Reconstruction of circles and round rectangles by generalized point photogrammetry[J]. Journal of Harbin Institute of Technology, 2008, 40(1):136-140.
[13]	张祖勋, 张剑清. 广义点摄影测量及其应用[J]. 武汉大学学报(信息科学版), 2005, 30(1):1-5.
	ZHANG Zuxun, ZHANG Jianqing. Generalized point photogrammetry and its application[J]. Geomatics and Information Science of Wuhan University, 2005, 30(1):1-5.
[14]	ZHANG Zuxun, ZHANG Yongjun, ZHANG Jianqing, et al. Photogrammetric modeling of linear features with generalized point photogrammetry[J]. Photogrammetric Engineering & Remote Sensing, 2008, 74(9):1119-1127.
[15]	HOFER M, MAURER M, BISCHOF H. Efficient 3D scene abstraction using line segments[J]. Computer Vision and Image Understanding, 2017, 157(3):167-178.
[16]	WEI Dong, WAN Yi, ZHANG Yongjun, et al. ELSR: efficient line segment reconstruction with planes and points guidance[C]//Proceedings of 2022 Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022.
[17]	CONNETTE C P, MEISTER O, HAGELE M, et al. Decomposition of line segments into corner and statistical grown line features in an EKF-SLAM framework[C]//Proceedings of 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. San Diego: IEEE, 2007.
[18]	LEMAIRE T, LACROIX S. Monocular-vision based SLAM using line segments[C]//Proceedings of 2007 IEEE International Conference on Robotics and Automation. Rome: IEEE, 2007.
[19]	QIAO Chengyu, BAI Tingming, XIANG Zhiyu, et al. Superline: a robust line segment feature for visual SLAM[C]//Proceedings of 2021 International Conference on Intelligent Robots and Systems. Prague: IEEE, 2021.
[20]	ZHANG Yongjun, WEI Dong, LI Yansheng. AG3line: active grouping and geometry-gradient combined validation for fast line segment extraction[J]. Pattern Recognition, 2021, 113:107834.
[21]	PAUTRAT R, BARÁTH D, LARSSON V, et al. DeepLSD: line segment detection and refinement with deep image gradients[C]//Proceedings of 2022 Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022.
[22]	AKINLAR C, TOPAL C. Edlines: real-time line segment detection by edge drawing (ed) [C]//Proceedings of the 18th IEEE International Conference on Image Processing. Brussels: IEEE, 2011.
[23]	VON GIOI R G, JAKUBOWICZ J, MOREL J M, et al. LSD: a fast line segment detector with a false detection control[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(4):722-732.
[24]	ALMAZAN E J, TAL R, QIAN Yiming, et al. MCMLSD: a dynamic programming approach to line segment detection[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017.
[25]	OK A Ö, WEGNER J D, HEIPKE C, et al. Accurate reconstruction of near-epipolar line segments from stereo aerial images[J]. Photogrammetrie Fernerkundung Geoinformation, 2012, 2012(4):345-358.
[26]	陈军, 刘建军, 田海波. 实景三维中国建设的基本定位与技术路径[J]. 武汉大学学报(信息科学版), 2022, 47(10):1568-1575.
	CHEN Jun, LIU Jianjun, TIAN Haibo. Basic directions and technological path for building 3D realistic geospatial scene in China[J]. Geomatics and Information Science of Wuhan University, 2022, 47(10):1568-1575.
[27]	JAIN A, KURZ C, THORMÄHLEN T, et al. Exploiting global connectivity constraints for reconstruction of 3D line segments from images[C]//Proceedings of 2010 Conference on Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010.
[28]	RAMALINGAM S, ANTUNES M, SNOW D, et al. Line-sweep: cross-ratio for wide-baseline matching and 3D reconstruction[C]//Proceedings of 2015 Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015.
[29]	LI Kai, YAO Jian. Line segment matching and reconstruction via exploiting coplanar cues[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 125:33-49.
[30]	WANG Wei, GAO Wei, CUI Hainan, et al. Reconstruction of lines and planes of urban buildings with angle regularization[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 165:54-66.
[31]	GUO Jianwei, LIU Yanchao, SONG Xin, et al. Line-based 3D building abstraction and polygonal surface reconstruction from images[J]. IEEE Transactions on Visualization and Computer Graphics, 2024, 3(18):1-15.
[32]	LIU Shaohui, YU Yifan, PAUTRAT Remi, et al. 3D Line mapping revisited[C]//Proceedings of 2023 Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE2023.
[33]	FANG Tong, CHEN Min, HU Han, et al. 3D line segment reconstruction with depth maps for photogrammetric mesh refinement in man-made environments[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:3332211.
[34]	张永军, 张祖勋, 龚健雅. 天空地多源遥感数据的广义摄影测量学[J]. 测绘学报, 2021, 50(1):1-11. DOI:10.11947/j.AGCS.2021.20200245.
	ZHANG Yongjun, ZHANG Zuxun, GONG Jianya. Generalized photogrammetry of spaceborne, airborne and terrestrial multi-source remote sensing datasets[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(1):1-11. DOI:10.11947/j.AGCS.2021.20200245.
[35]	BAY H, FERRARI V, GOOL L. Wide-baseline stereo matching with line segments[C]//Proceedings of 2005 Conference on Computer Vision and Pattern Recognition. Colorado: IEEE, 2005.
[36]	王志衡, 吴福朝. 均值-标准差描述子与直线匹配[J]. 模式识别与人工智能, 2009, 22(1):32-39.
	WANG Zhiheng, WU Fuchao. Mean-standard deviation descriptor and line matching[J]. Pattern Recognition and Artificial Intelligence, 2009, 22(1):32-39.
[37]	WANG Zhiheng, WU Fuchao, HU Zhanyi. MSLD: a robust descriptor for line matching[J]. Pattern Recognition, 2009, 42(5):941-953.
[38]	KIM H, LEE S, LEE Y. Wide-baseline stereo matching based on the line intersection context for real-time workspace modeling[J]. Journal of the Optical Society of America A, 2014, 31(2):421-435.
[39]	LÓPEZ J, SANTOS R, FDEZ-VIDAL X R, et al. Two-view line matching algorithm based on context and appearance in low-textured images[J]. Pattern Recognition, 2015, 48(7):2164-2184.
[40]	ZHANG Lilian, KOCH R. An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency[J]. Journal of Visual Communication and Image Representation, 2013, 24(7):794-805.
[41]	AL-SHAHRI M, YILMAZ A. Line matching in wide-baseline stereo: a top-down approach[J]. IEEE Transactions Image Process, 2014, 23(9):4199-4210.
[42]	WEI Dong, ZHANG Yongjun, LIU Xinyi, et al. Robust line segment matching across views via ranking the line-point graph[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:49-62.
[43]	WEI Dong, ZHANG Yongjun, LI Chang. Robust line segment matching via reweighted random walks on the homography graph[J]. Pattern Recognition, 2021, 111:107693.
[44]	LOURAKIS M I A, HALKIDIS S T, ORPHANOUDAKIS S C. Matching disparate views of planar surfaces using projective invariants[J]. Image and Vision Computing, 2000, 18(9):673-683.
[45]	CHO M, LEE J, LEE K M. Reweighted random walks for graph matching[C]//Proceedings of 2010 European Conference on Computer Vision. Hersonissos: IEEE, 2010.
[46]	BARATH D, NOSKOVA J, IVASHECHKIN M, et al. MAGSAC++, a fast, reliable and accurate robust estimator[C]//Proceedings of 2020 Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020.
[47]	LI Kai, YAO Jian, LI Li, et al. 3D line segment reconstruction in structured scenes via coplanar line segment clustering[C]//Proceedings of 2017 Asian Conference on Computer Vision. Cham: Springer, 2017: 46-61.
[48]	宋伟东, 朱红, 王竞雪, 等. 多重约束下的近景影像线特征匹配方法[J]. 中国图象图形学报, 2016, 21(6):764-770.
	SONG Weidong, ZHU Hong, WANG Jingxue, et al. Line feature matching method based on multiple constraints for close-range images[J]. Journal of Image and Graphics, 2016, 21(6):764-770.
[49]	WANG Qiang, ZHANG Wei, LIU Xiaolong, et al. Line matching of wide baseline images in an affine projection space[J]. International Journal of Remote Sensing, 2020, 41(2):632-654.
[50]	严少华, 陈敏, 赵怡涛, 等. 局部结构约束下的宽基线影像直线特征匹配[J]. 测绘通报, 2020 (5):51-54.
	YAN Shaohua, CHEN Min, ZHAO Yitao, et al. Local structure based line segment matching for wide-baseline images[J]. Bulletin of Surveying and Mapping, 2020 (5):51-54.
[51]	CHEN Min, YAN Shaohua, QIN Rongjun, et al. Hierarchical line segment matching for wide-baseline images via exploiting viewpoint robust local structure and geometric constraints[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 181:48-66.
[52]	CHEN Min, LI Wen, FANG Tong, et al. An adaptive feature region-based line segment matching method for viewpoint-changed images with discontinuous parallax and poor textures[J]. International Journal of Applied Earth Observation and Geoinformation, 2023, 117:103209.
[53]	LI Kai, YAO Jian, LU Xiaohu, et al. Hierarchical line matching based on line-junction-line structure descriptor and local homography estimation[J]. Neurocomputing, 2016, 184:207-220.
[54]	OK A O, WEGNER J D, HEIPKE C, et al. Matching of straight line segments from aerial stereo images of urban areas[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2012, 74:133-152.
[55]	FAN Bin, WU Fuchao, HU Zhanyi. Line matching leveraged by point correspondences[C]//Proceedings of 2010 Conference on Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010.
[56]	FAN Bin, WU Fuchao, HU Zhanyi. Robust line matching through line-point invariants[J]. Pattern Recognition, 2012, 45(2):794-805.
[57]	SUN Yanbiao, ZHAO Liang, HUANG Shoudong, et al. Line matching based on planar homography for stereo aerial images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 104:1-17.
[58]	HARTLEY R, ZISSERMAN A. Multiple view geometry in computer vision [M]. New York: Cambridge University Press, 2004.
[59]	LANGLOIS P A, BOULCH A, MARLET R. Surface reconstruction from 3D line segments[C]//Proceedings of 2019 International Conference on 3D Vision. Québec City: IEEE, 2019.
[60]	WITT J, MENTGES G. Maximally informative surface reconstruction from lines[C]//Proceedings of 2014 International Conference on Robotics and Automation. Hong Kong: IEEE, 2014.
[61]	NAN Liangliang, WONKA P. PolyFit: polygonal surface reconstruction from point clouds[C]//Proceedings of 2017 International Conference on Computer Vision (ICCV). Venice: IEEE, 2017.
[62]	王伟, 于磊, 任国恒, 等. 城市建筑立面三维“线-面” 结构快速重建[J]. 电子学报, 2021, 49(8):1551-1560.
	WANG Wei, YU Lei, REN Guoheng, et al. Rapidly reconstructing 3D line-plane structures of urban building facades[J]. Acta Electronica Sinica, 2021, 49(8):1551-1560.
[63]	李畅. 城市街道立面自动重建关键技术研究[J]. 测绘学报, 2011, 40(2):268.
	LI Chang. Researching on key technique for 3D auto-reconstruction of city street elevation[J]. Acta Geodaetica et Cartographica Sinica, 2011, 40(2):268.
[64]	ZHOU Ying, WANG Lingling, LOVE P, et al. Three-dimensional (3D) reconstruction of structures and landscapes: a new point-and-line fusion method[J]. Advanced Engineering Informatics, 2019, 42:100961.
[65]	LI Shiwei, YAO Yao, FANG Tian, et al. Reconstructing thin structures of manifold surfaces by integrating spatial curves[C]//Proceedings of 2018 Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018.
[66]	QIN Rongjun, CHEN Min, HUANG Xu, et al. Disparity refinement in depth discontinuity using robustly matched straight lines for digital surface model generation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(1):174-185.
[67]	HOFER M, MAURER M, BISCHOF H. Improving sparse 3D models for man-made environments using line-based 3D reconstruction[C]//Proceedings of the 2nd International Conference on 3D Vision. Tokyo: IEEE, 2014.
[68]	LIU Xinyi, ZHANG Yongjun, LING Xiao, et al. TopoLAP: topology recovery for building reconstruction by deducing the relationships between linear and planar primitives[J]. Remote Sensing, 2019, 11(11):1372.
[69]	KAISER T, CLEMEN C, MAAS H G. Automatic co-registration of photogrammetric point clouds with digital building models[J]. Automation in Construction, 2022, 134:104098.
[70]	LING Xiao, QIN Rongjun. A graph-matching approach for cross-view registration of over-view and street-view based point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 185:2-15.
[71]	王广帅. 无人机倾斜摄影铁路轨道线高精度自动重建[J]. 测绘通报, 2022 (5):133-139, 156.
	WANG Guangshuai. High-precision automatic reconstruction of railway line with UAV oblique photography[J]. Bulletin of Surveying and Mapping, 2022 (5):133-139, 156.
[72]	XU Chang, LI Qingwu, ZHOU Qingkai, et al. Power line-guided automatic electric transmission line inspection system[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71:1-18.
[73]	PASTUCHA E, PUNIACH E, SCISLOWICZ A, et al. 3D reconstruction of power lines using UAV images to monitor corridor clearance[J]. Remote Sensing, 2020, 12(22):3698.
[74]	GANOVELLI F, MALOMO L, SCOPIGNO R. Reconstructing power lines from images[C]//Proceedings of 2018 International Conference Image and Vision Computing New Zealand. New Zealand: IEEE, 2018.
[75]	陈军, 艾廷华, 闫利, 等. 智能化测绘的混合计算范式与方法研究[J/OL]. 测绘学报: 1-19[2024-06-12].http://kns.cnki.net/kcms/detail/11.2089.P.20240415.1049.002.html.
	CHEN Jun, AI Tinghua, YAN Li, et al. Hybrid computational paradigm and methods for intelligentized surveying and mapping[J/OL]. Acta Geodaetica et Cartographica Sinica: 1-19[2024-06-12].http://kns.cnki.net/kcms/detail/11.2089.P.20240415.1049.002.html.

算法	特点
文献[27]	三维线段的反投影检验中使用纹理强度评价，不依赖直线提取结果；可获取较多数量的三维线段，但运行速度慢
Line-sweep^[28]	采用匹配点引导的扫描匹配策略，效果稳健且计算复杂度低
文献[29]	基于线段编组的平面约束纠正错误前方交会；匹配过程不需要影像的初始位姿；可以重建出较多且规整的三维结构；运行速度未知
Line3D++^[15]	并行匹配，运行速度较快；使用点线联合的光束法平差优化了重建精度；但严格的单一阈值融合策略使得线云数量较少
文献[30]	利用深度学习改善两视前方交会沿核面的误差问题；可以重建较多的线段；但深度学习策略的泛化性有待考察
ELSR^[16]	利用CPU多线程加速，大场景重建速度快；线云数量较多；但单一融合阈值与代表线段获取的线云仍需要进行进一步的精度优化
文献[31]	利用多尺度策略改进了线段提取算法；可重建出更多的三维线；并在此基础上进行了建筑物重建
Linemap^[32]	支持不同的线段提取与匹配算法组合；引入灭点与特征点优化精度；数量与精度表现较好；速度有待优化
文献[33]	使用深度图提升速度与精度；数量与质量上都有较好的表现；对三维网格优化进行了应用
SLAM^[17-19]	增量式重建；需要窄基线像对