融合点云和多视图的车载激光点云路侧多目标识别

doi:10.11947/j.AGCS.2021.20210246

摘要/Abstract

摘要： 城市环境中的行道树、车辆、杆状交通设施是重要的交通地物，也是智能交通，导航与位置服务，自动驾驶和高精地图等行业应用的核心要素。为了准确识别这些路侧目标，本文提出一种融合点云和多视角图像的深度学习模型PGVNet（point-group-view network），充分利用目标点云数据中空间几何信息及其多视角图像中高级全局特征提升路侧行道树、车辆和杆状设施的分类精度。为了减少视图间的冗余信息并增强显著视图特征，PGVNet模型利用预训练的VGG网络提取多视图特征，对其进行分组赋权获取最优视图特征；采用嵌入注意力机制的融合策略，利用最优视图特征动态调整PGVNet模型对点云不同局部关系的注意力度，学习不同路侧目标的多层次、多尺度显著特征，实现行道树、车辆和杆状交通设施的精确分类。试验采用5份不同车载激光扫描系统获取的不同城市场景数据验证本文方法的有效性，其中行道树、车辆及杆状交通设施分类结果中的准确率、召回率、精度和F₁指数分别达（99.19%、94.27%、93.58%、96.63%）；（94.20%、97.56%、92.02%、95.68%）；（91.48%、98.61%、90.39%、94.87%）。结果表明，本文方法融合多视图全局信息和点云局部结构特征可以有效区分城市场景中的行道树、车辆和杆状交通设施，可为高精度地图中要素构建与矢量化提供数据支撑。

关键词: 车载激光扫描系统, 点云分类, 多视角图像, 深度学习, 注意力机制

Abstract: Accurately identifying roadside objects like trees, cars, and traffic poles from mobile LiDAR point clouds is of great significance for some applications such as intelligent traffic systems, navigation and location services, autonomous driving, and high precision map. In the paper, we proposed a point-group-view network (PGVNet) to classify the roadside objects into trees, cars, traffic poles, and others, which utilize and fuse the advanced global features of multi-view images and the spatial geometry information of point cloud. To reduce redundant information between similar views and highlight salient view features, the PGVNet model employs a hierarchical view-group-shape architecture to split all views into different groups according to their discriminative level, which uses the pre-trained VGG network as the bone network. In view-group-shape architecture, global-level significant features are further generated from group descriptors with their weights. Moreover, an attention-guided fusion network is used to fuse the global features from multi-view images and local geometric features from point clouds. In particular, the global advanced features from multi-view images are quantified and leveraged as the attention mask to further refine the intrinsic correlation and discriminability of the local geometric features from point clouds, which contributions to recognize the roadside objects. We have evaluated the proposed method on five different mobile LiDAR point cloud data. Five test datasets of different urban scenes by different mobile laser scanning systems are used to evaluate the validities of the proposed method. Four accuracy evaluation metrics precision, recall, quality and F_score of trees, cars and traffic poles on the selected testing datasets achieve (99.19%,94.27%,93.58%,96.63%),(94.20%,97.56%,92.02%,95.68%),(91.48%,98.61%,90.39%,94.87%), respectively. Experimental results and comparisons with state-of-the-art methods demonstrate that the PGVNet model is available to effectively identify roadside objects from the mobile LiDAR point cloud, which can provide data support for elements construction and vectorization in high precision map applications.

Key words: mobile laser scanning systems, point cloud classification, multi-view, deep learning, attention mechanism

中图分类号:

P237

方莉娜, 沈贵熙, 游志龙, 郭迎亚, 付化胜, 赵志远, 陈崇成. 融合点云和多视图的车载激光点云路侧多目标识别[J]. 测绘学报, 2021, 50(11): 1558-1573.

FANG Lina, SHEN Guixi, YOU Zhilong, GUO Yingya, FU Huasheng, ZHAO Zhiyuan, CHEN Chongcheng. A joint network of point cloud and multiple views for roadside objects recognition from mobile laser point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1558-1573.

参考文献

[1] 刘经南, 詹骄, 郭迟, 等. 智能高精地图数据逻辑结构与关键技术[J]. 测绘学报, 2019,48(8):939-953. DOI: 10.11947/j.AGCS.2019.20190125. LIU Jingnan, ZHAN Jiao, GUO Chi, et al. Data logic structure and key technologies on intelligent high-precision map[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(8):939-953. DOI: 10.11947/j.AGCS.2019.20190125.
[2] 杨必胜, 魏征, 李清泉, 等. 面向车载激光扫描点云快速分类的点云特征图像生成方法[J]. 测绘学报, 2010,39(5):540-545. YANG Bisheng, WEI Zheng, LI Qingquan, et al. A classification-oriented method of feature image generation for vehicle-borne laser scanning point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2010,39(5):540-545.
[3] 李德仁. 展望5G/6G时代的地球空间信息技术[J]. 测绘学报, 2019,48(12):1475-1481. DOI: 10.11947/j.AGCS.2019.20190437. LI Deren. Towards geospatial information technology in 5G/6G era[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(12):1475-1481. DOI: 10.11947/j.AGCS.2019.20190437.
[4] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019,48(12):1575-1585. DOI: 10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(12):1575-1585. DOI: 10.11947/j.AGCS.2019.20190465.
[5] YANG Bisheng, WEI Zheng, LI Qingquan, et al. Automated extraction of street-scene objects from mobile lidar point clouds[J]. International Journal of Remote Sensing, 2012,33(18):5839-5861.
[6] 李婷, 詹庆明, 喻亮. 基于地物特征提取的车载激光点云数据分类方法[J]. 国土资源遥感, 2012, 24(1):17-21. LI Ting, ZHAN Qingming, YU Liang. A classification method for mobile laser scanning data based on object feature extraction[J]. Remote Sensing for Land & Resources, 2012, 24(1):17-21.
[7] LEHTOMÄKI M, JAAKKOLA A, HYYPPÄ J, et al. Object classification and recognition from mobile laser scanning point clouds in a road environment[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016,54(2):1226-1239.
[8] YU Yongtao, LI J, WEN Chenglu, et al. Bag-of-visual-phrases and hierarchical deep models for traffic sign detection and recognition in mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016,113:106-123.
[9] XIAO Wen, VALLET B, SCHINDLER K, et al. Street-side vehicle detection, classification and change detection using mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016,114:166-178.
[10] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015,521(7553):436-444.
[11] GUAN Haiyan, YU Yongtao, JI Zheng, et al. Deep learning-based tree classification using mobile LiDAR data[J]. Remote Sensing Letters, 2015,6(11):864-873.
[12] 罗海峰, 方莉娜, 陈崇成, 等. 基于DBN的车载激光点云路侧多目标提取[J]. 测绘学报, 2018,47(2):234-246. DOI: 10.11947/j.AGCS.2018.20170524. LUO Haifeng, FANG Lina, CHEN Chongcheng, et al. Roadside multiple objects extraction from mobile laser scanning point cloud based on DBN[J]. Acta Geodaetica et Cartographica Sinica, 2018,47(2):234-246. DOI: 10.11947/j.AGCS.2018.20170524.
[13] SU Hang, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 945-953.
[14] QI C R, SU Hao, NIEßNER M, et al. Volumetric and multi-view CNNs for object classification on 3D data[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 5648-5656.
[15] VISHWANATH K V, GUPTA D, VAHDAT A, et al. ModelNet: Towards a datacenter emulation environment[C]//Proceedings of 2009 IEEE Ninth International Conference on Peer-to-Peer Computing. Seattle, WA, USA: IEEE, 2009: 81-82.
[16] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[17] FENG Yifan, ZHANG Zizhao, ZHAO Xibin, et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 264-272.
[18] YU Tan, MENG Jingjing, YUAN Junsong. Multi-view harmonized bilinear network for 3D object recognition[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 186-194.
[19] CHARLES R Q, HAO Su, MO Kaichun, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 77-85.
[20] BELLO S A, YU Shangshu, WANG Cheng, et al. Review: deep learning on 3D point clouds[J]. Remote Sensing, 2020, 12(11): 1729.DOI:org/10.3390/rs12111729.
[21] QI C R, YI LI, SU HAO, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space [C]//Proceedings of the 31st Conference on Neural Information Processing Systems. Long Beach, CA, USA:NIPS,2017.
[22] JOSEPH-RIVLIN M, ZVIRIN A, KIMMEL R. Momenet: flavor the moments in learning to classify shapes[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE, 2019: 4085-4094.
[23] LI Yangyan, BU Rui, SUN Mingchao, et al. PointCNN: convolution on x-transformed points[C]//Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018). NowYork, NY, USA: Curran Associates, 2018: 820-830.
[24] ZHAO Hengshuang, JIANG Li, FU C W, et al. PointWeb: enhancing local neighborhood features for point cloud processing[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2019: 5560-5568.
[25] THOMAS H, QI C R, DESCHAUD J E, et al. KPConv: flexible and deformable convolution for point clouds[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 6410-6419.
[26] WANG Yue, SUN Yongbin, LIU Ziwei, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 1-12.
[27] QI C R, LIU Wei, WU Chenxia, et al. Frustum PointNets for 3D object detection from RGB-D data[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 918-927.
[28] YOU Haoxuan, FENG Yifan, JI Rongrong, et al. PVNet: a joint convolutional network of point cloud and multi-view for 3D shape recognition[C]//Proceedings of the 26th ACM international conference on Multimedia. New York, NY, USA: ACM, 2018: 4561-4570.
[29] YOU Haoxuan, FENG Yifan, ZHAO Xibin, et al. PVRNet: point-view relation neural network for 3D shape recognition[C]//Proceedings of 2019 AAAI Conference on Artificial Intelligence. Palo Alto, California, USA: AAAI, 2019, 33: 9119-9126.
[30] GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: The KITTI dataset[J]. The International Journal of Robotics Research, 2013,32(11):1231-1237.
[31] DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 2432-2443.
[32] ZHANG Wuming, QI Jianbo, WAN Peng, et al. An easy-to-use airborne LiDAR data filtering method based on cloth simulation[J]. Remote Sensing, 2016,8(6):501.
[33] REINGOLD O. Undirected connectivity in log-space[J]. Journal of the ACM, 2008,55(4):1-24.
[34] YU Y, LI J, GUAN H et al. Automated extraction of 3D trees from mobile LiDAR point clouds[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014, XL-5: 629-632.
[35] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA: IEEE, 2009: 248-255.
[36] ROYNARD X, DESCHAUD J E, GOULETTE F. Paris-Lille-3D: a large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification[J]. The International Journal of Robotics Research, 2018,37(6):545-557.