融合CNN和MRF的激光点云层次化语义分割方法

doi:10.11947/j.AGCS.2021.20200095

测绘学报 ›› 2021, Vol. 50 ›› Issue (2): 215-225.doi: 10.11947/j.AGCS.2021.20200095

融合CNN和MRF的激光点云层次化语义分割方法

蒋腾平^1,2,3, 王永君^2,4,5, 张林淇^2,4,5, 梁冲^2,4,5, 孙剑^2,4,5

1. 自然资源部城市土地资源监测与仿真重点实验室, 广东深圳 518034;
2. 南京师范大学虚拟地理环境教育部重点实验室, 江苏南京 210093;
3. 武汉大学测绘遥感信息工程国家重点实验室, 湖北武汉 430079;
4. 江苏省地理信息资源开发与利用协同创新中心, 江苏南京 210023;
5. 地理环境演化模拟国家重点实验室建设培育点, 江苏南京 210093

收稿日期:2020-03-17 修回日期:2020-07-04 发布日期:2021-03-03
通讯作者: 王永君 E-mail:wangyongjun@njnu.edu.cn
作者简介:蒋腾平(1993-),男,博士生,研究方向为三维点云语义识别和建模。E-mail:jiangtp_3d@whu.edu.cn
基金资助:
自然资源部城市土地资源监测与仿真重点实验室开放基金资助课题（KF-2018-03-070）；国家自然科学基金（41771439）；国家重点研发计划（2016YFB0502304）；江苏省研究生科研与实践创新计划项目（KYCX18_1206）

A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF

JIANG Tengping^1,2,3, WANG Yongjun^2,4,5, ZHANG Linqi^2,4,5, LIANG Chong^2,4,5, SUN Jian^2,4,5

1. Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen 518034, China;
2. Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing 210093, China;
3. State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China;
4. Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing 210023, China;
5. State Key Laboratory Cultivation Base of Geographical Environment Evolution, Nanjing Normal University, Nanjing 210093, China

Received:2020-03-17 Revised:2020-07-04 Published:2021-03-03
Supported by:
The Open Fund of Key Laboratory of Urban Land Resources Monitoring and Simulation,Ministry of Natural Resources (No. KF-2018-03-070);The National Natural Science Foundation of China (No. 41771439);The National Key Research and Development Program of China (No. 2016YFB0502304);The Postgraduate Research and Practice Innovation Program of Jiangsu Province (No. KYCX18_1206)

摘要/Abstract

摘要： 三维点云语义分割的结果包含着对场景中多个目标的识别，是三维场景信息提取的重要环节，在智慧城市等多个领域扮演关键角色。由于三维激光点云数据量庞大、场景复杂性高等问题，大多数现有方法只能以相对较低的识别率提取有限类型的对象。本文提出了一种在三维激光点云场景中结合残差学习和马尔可夫随机场（MRF）优化的层次化多类型目标自动提取框架。该框架首先将点云滤波为地面点和非地面点；然后从非地面点中提取建筑物以降低场景复杂度；接着基于现有深度模型引入残差学习模块对剩下点云进行逐点分类；最后采用马尔可夫随机场进行后处理和分类结果优化，以提高激光点云语义分割的准确率。对3个室外大规模点云场景分别进行的试验结果表明，本文方法可以对多种类型的点云场景进行有效语义分割，每个数据集的3项指标（召回率、精确度和F1值）分别为（94.6%、96.8%、95.7%）、（88.5%、90.5%、89.2%）和（95.3%、95.2%、95.3%）。此外，与现有较前沿方法相比，本文方法显著提高了语义分割性能。

关键词: 激光点云, 语义分割, 层次化提取, 残差学习, 马尔可夫随机场(MRF)

Abstract: The result of point cloud semantic segmentation includes the recognition of multiple objects in the scene, which is an important part of 3D scene information extraction. It also plays a key role in many fields such as smart cities. Since the large amount of data and high scene complexity, however, most existing methods can only extract a limited type of objects with a relatively low recognition rate from laser scanning data. This paper proposes a hierarchical multiple object automatic extraction framework combining residual learning and Markov random field (MRF) optimization in a 3D LiDAR point cloud scene. The framework first filters raw data into ground points and non-ground points; buildings are extracted from non-ground points to reduce scene complexity; then the residual learning network is adopted to pointwise classify the remaining point clouds; finally, a MRF-based optimization is used for post-processing and improving the accuracy of point cloud classification. In order to evaluate the effectiveness and robustness of the proposed method, experiments were performed on three outdoor large-scale point cloud scenarios. Experimental results show that the proposed method can perform effective semantic segmentation on various types of point cloud scenes, with (94.6%, 96.8%, 95.7%), (88.5%, 90.5%, 89.2%) and (95.3%, 95.2%, 95.3%), respectively. In addition, compared with existing advanced methods, it is shown that our method significantly improves the performance of semantic segmentation.

Key words: LiDAR point cloud, semantic segmentation, hierarchical extraction, residual learning, MRF

中图分类号:

P237

蒋腾平, 王永君, 张林淇, 梁冲, 孙剑. 融合CNN和MRF的激光点云层次化语义分割方法[J]. 测绘学报, 2021, 50(2): 215-225.

JIANG Tengping, WANG Yongjun, ZHANG Linqi, LIANG Chong, SUN Jian. A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2): 215-225.

参考文献

[1] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12):1575-1585. DOI:10.11947/j.AGCS.2019.20190465.
[2] 杨必胜, 梁福逊, 黄荣刚. 三维激光扫描点云数据处理研究进展、挑战与趋势[J]. 测绘学报, 2017, 46(10):1509-1516. DOI:10.11947/j.AGCS.2017.20170351. YANG Bisheng, LIANG Fuxun, HUANG Ronggang. Progress, challenges and perspectives of 3D LiDAR point cloud processing[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10):1509-1516. DOI:10.11947/j.AGCS.2017.20170351.
[3] WEINMANN M, JUTZI B, HINZ S, et al. Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 105:286-304.
[4] LANDRIEU L, RAGUET H, VALLET B, et al. A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 132:102-118.
[5] 熊艳, 高仁强, 徐战亚. 机载LiDAR点云数据降维与分类的随机森林方法[J]. 测绘学报, 2018, 47(4):508-518. DOI:10.11947/j.AGCS.2018.20170417. XIONG Yan, GAO Renqiang, XU Zhanya. Random forest method for dimension reduction and point cloud classification based on airborne LiDAR[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(4):508-518. DOI:10.11947/j.AGCS.2018.20170417.
[6] ZHU Zan, GAN Shu,WANG Jianqi, et al. A Novel airborne 3D laser point cloud hole repair algorithm considering topographic features[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3):29-38.
[7] 赵传, 郭海涛, 卢俊, 等. 基于深度残差网络的机载LiDAR点云分类[J]. 测绘学报, 2020, 49(2):202-213. DOI:10.11947/j.AGCS.2020.20190004. ZHAO Chuan, GUO Haitao, LU Jun, et al. Airborne LiDAR point cloud classification based on deep residual network[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(2):202-213. DOI:10.11947/j.AGCS.2020.20190004.
[8] HUA B S, TRAN M K, YEUNG S K. Pointwise convolutional neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018:984-993.
[9] ZHANG Dejun, HE Fazhi, TU Zhigang, et al. Pointwise geometric and semantic learning network on 3D point clouds[J]. Integrated Computer-Aided Engineering, 2020, 27(1):57-75.
[10] LUO Huan, WANG Cheng, WEN Chenglu, et al. Patch-based semantic labeling of road scene using colorized mobile LiDAR point clouds[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(5):1286-1297.
[11] 罗海峰, 方莉娜, 陈崇成, 等. 基于DBN的车载激光点云路侧多目标提取[J]. 测绘学报, 2018, 47(2):234-246. DOI:10.11947/j.AGCS.2018.20170524. LUO Haifeng, FANG Lina, CHEN Chongcheng, et al. Roadside multiple objects extraction from mobile laser scanning point cloud based on DBN[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(2):234-246. DOI:10.11947/j.AGCS.2018.20170524.
[12] WANG Hanyun, WANG Cheng, LUO Huan, et al. 3-D point cloud object detection based on supervoxel neighborhood with Hough forest framework[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(4):1570-1581.
[13] 董震, 杨必胜. 车载激光扫描数据中多类目标的层次化提取方法[J]. 测绘学报, 2015, 44(9):980-987. DOI:10.11947/j.AGCS.2015.20140339. DONG Zhen, YANG Bisheng. Hierarchical extraction of multiple objects from mobile laser scanning data[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(9):980-987. DOI:10.11947/j.AGCS.2015.20140339.
[14] 李明磊, 刘少创, 杨欢, 等. 双层优化的激光雷达点云场景分割方法[J]. 测绘学报, 2018, 47(2):269-274. DOI:10.11947/j.AGCS.2018.20170493. LI Minglei, LIU Shaochuang, YANG Huan, et al. Bilevel optimization for scene segmentation of LiDAR point cloud[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(2):269-274. DOI:10.11947/j.AGCS.2018.20170493.
[15] XU Yusheng, YE Zhen, YAO Wei, et al. Classification of LiDAR point clouds using supervoxel-based detrended feature and perception-weighted graphical model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:72-88.
[16] BABAHAJIANI P, FAN Lixin, GABBOUJ M. Object recognition in 3D point cloud of urban street scene[C]//Computer Vision-ACCV Workshops. Singapore:Springer, 2014:177-190.
[17] YANG Bisheng, DONG Zhen, LIU Yuan, et al. Computing multiple aggregation levels and contextual features for road facilities recognition using mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 126:180-194.
[18] YANG Bisheng, DONG Zhen, ZHAO Gang, et al. Hierarchical extraction of urban objects from mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 99:45-57.
[19] TEO T A, CHIU C M. Pole-like road object detection from mobile LiDAR system using a coarse-to-fine approach[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(10):4805-4818.
[20] LIU Xiaoqiang, CHEN Yanming, LI Shuyi, et al. Hierarchical classification of urban ALS data by using geometry and intensity information[J]. Sensors, 2019, 19(20):4583.
[21] WANG Yongjun, JIANG Tengping, YU Min, et al. Semantic-based building extraction from LiDAR point clouds using contexts and optimization in complex environment[J]. Sensors, 2020, 20:3386.
[22] LI Nan, LIU Chun, PFEIFER N. Improving LiDAR classification accuracy by contextual label smoothing in post-processing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 148:13-31.
[23] WANG Y, JIANG T, LIU J, et al. Hierarchical instance recognition of individual roadside trees in environmentally complex urban areas from UAV laser scanning point clouds[J]. ISPRS International Journal of Geo-Information. 2020; 9(10):595.
[24] ZHANG Liangpei,ZHANG Yun, CHEN Zhenzhong, et al. Splitting and merging based multi-model fitting for point cloud segmentation[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):78-89.
[25] RUSU R B, COUSINS S. 3D is here:Point cloud library (PCL)[C]//2011 IEEE International Conference on Robotics and Automation. Shanghai, China:IEEE, 2011:1-4.
[26] ZHANG Wuming, QI Jianbo, WAN Peng, et al. An easy-to-use airborne LiDAR data filtering method based on cloth simulation[J]. Remote Sensing, 2016, 8(6):501.
[27] XIA Shanbo, WANG Ruisheng. Extraction of residential building instances in suburban areas from mobile LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 144:453-468.
[28] SCHNABEL R, WAHL R, KLEIN R. Efficient RANSAC for point-cloud shape detection[J]. Computer Graphics Forum, 2007, 26(2):214-226.
[29] KOLMOGOROV V, ZABIN R. What energy functions can be minimized via graph cuts?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(2):147-159.
[30] QI C R, SU Hao, MO Kaichun, et al. PointNet:deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:77-85.
[31] ARSHAD S, SHAHZAD M, RIAZ Q, et al. DPRNet:Deep 3D point based residual network for semantic segmentation and classification of 3D point clouds[J]. IEEE Access, 2019, 7:68892-68904.
[32] KLAMBAUER G, UNTERTHINER T, MAYR A, et al. Self-normalizing neural networks[C]//31st Conference on Neural Information Processing Systems. Long Beach, CA:[s.n.], 2017:971-980.
[33] XIANG Binbin, YAO Jian, LU Xiaohu, et al. Segmentation-based classification for 3D point clouds in the road environment[J]. International Journal of Remote Sensing, 2018, 39(19):6182-6212.
[34] HACKEL T, SAVINOV N, LADICKY L, et al. SEMANTIC3D.NET:A new large-scale point cloud classification benchmark[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017, IV-1/W1:91-98.

融合CNN和MRF的激光点云层次化语义分割方法

A LiDAR point cloud hierarchical semantic segmentation method combining CNN and MRF

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	方莉娜, 王爽, 赵志远, 付化胜, 陈崇成. 车载激光点云中交通标线自动分类与矢量化[J]. 测绘学报, 2021, 50(9): 1251-1265.
[2]	邵晓航, 吴杭彬, 刘春, 陈晨, 蔡天池, 程帆瑾. 语素关联约束的动态环境视觉定位优化[J]. 测绘学报, 2021, 50(11): 1478-1486.
[3]	李彩林, 王志勇, 俞路路, 郭宝云. 最邻近曲面约束的近景光学影像与地面激光点云几何配准[J]. 测绘学报, 2020, 49(8): 1014-1022.
[4]	李道纪, 郭海涛, 卢俊, 赵传, 林雨准, 余东行. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8): 1051-1064.
[5]	耿艳磊, 陶超, 沈靖, 邹峥嵘. 高分辨率遥感影像语义分割的半监督全卷积网络法[J]. 测绘学报, 2020, 49(4): 499-508.
[6]	刘嗣超, 武鹏达, 赵占杰, 李成名. 交通监控视频图像语义分割及其拼接方法[J]. 测绘学报, 2020, 49(4): 522-532.
[7]	王忠武, 王志盼, 尤淑撑, 雷帆, 曹里, 杨凯钧. Landsat影像冰川提取的上下文感知语义分割网络法[J]. 测绘学报, 2020, 49(12): 1575-1582.
[8]	徐旺, 游雄, 张威巍, 邓晨. 2D地图的建筑物场景结构提取方法及其在城市增强现实中的应用[J]. 测绘学报, 2020, 49(12): 1619-1629.
[9]	方莉娜, 卢丽靖, 赵志远, 王昀昀, 陈崇成. 车载激光点云道路边界提取的Snake方法[J]. 测绘学报, 2020, 49(11): 1438-1450.
[10]	李雪, 张力, 王庆栋, 艾海滨. 多时相遥感影像语义分割色彩一致性对抗网络[J]. 测绘学报, 2020, 49(11): 1473-1484.
[11]	方莉娜, 黄志文, 罗海峰, 陈崇成. 车载激光扫描数据中实线型交通标线提取[J]. 测绘学报, 2019, 48(8): 960-974.
[12]	左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6): 718-726.
[13]	季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4): 448-459.
[14]	贺浩, 王仕成, 杨东方, 王舒洋, 刘星. 基于Encoder-Decoder网络的遥感影像道路提取方法[J]. 测绘学报, 2019, 48(3): 330-338.
[15]	刘冰, 余旭初, 张鹏强, 谭熊. 联合空-谱信息的高光谱影像深度三维卷积网络分类[J]. 测绘学报, 2019, 48(1): 53-63.