A joint network of point cloud and multiple views for roadside objects recognition from mobile laser point clouds

doi:10.11947/j.AGCS.2021.20210246

Abstract

Abstract: Accurately identifying roadside objects like trees, cars, and traffic poles from mobile LiDAR point clouds is of great significance for some applications such as intelligent traffic systems, navigation and location services, autonomous driving, and high precision map. In the paper, we proposed a point-group-view network (PGVNet) to classify the roadside objects into trees, cars, traffic poles, and others, which utilize and fuse the advanced global features of multi-view images and the spatial geometry information of point cloud. To reduce redundant information between similar views and highlight salient view features, the PGVNet model employs a hierarchical view-group-shape architecture to split all views into different groups according to their discriminative level, which uses the pre-trained VGG network as the bone network. In view-group-shape architecture, global-level significant features are further generated from group descriptors with their weights. Moreover, an attention-guided fusion network is used to fuse the global features from multi-view images and local geometric features from point clouds. In particular, the global advanced features from multi-view images are quantified and leveraged as the attention mask to further refine the intrinsic correlation and discriminability of the local geometric features from point clouds, which contributions to recognize the roadside objects. We have evaluated the proposed method on five different mobile LiDAR point cloud data. Five test datasets of different urban scenes by different mobile laser scanning systems are used to evaluate the validities of the proposed method. Four accuracy evaluation metrics precision, recall, quality and F_score of trees, cars and traffic poles on the selected testing datasets achieve (99.19%,94.27%,93.58%,96.63%),(94.20%,97.56%,92.02%,95.68%),(91.48%,98.61%,90.39%,94.87%), respectively. Experimental results and comparisons with state-of-the-art methods demonstrate that the PGVNet model is available to effectively identify roadside objects from the mobile LiDAR point cloud, which can provide data support for elements construction and vectorization in high precision map applications.

Key words: mobile laser scanning systems, point cloud classification, multi-view, deep learning, attention mechanism

CLC Number:

P237

FANG Lina, SHEN Guixi, YOU Zhilong, GUO Yingya, FU Huasheng, ZHAO Zhiyuan, CHEN Chongcheng. A joint network of point cloud and multiple views for roadside objects recognition from mobile laser point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1558-1573.

References

[1] 刘经南, 詹骄, 郭迟, 等. 智能高精地图数据逻辑结构与关键技术[J]. 测绘学报, 2019,48(8):939-953. DOI: 10.11947/j.AGCS.2019.20190125. LIU Jingnan, ZHAN Jiao, GUO Chi, et al. Data logic structure and key technologies on intelligent high-precision map[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(8):939-953. DOI: 10.11947/j.AGCS.2019.20190125.
[2] 杨必胜, 魏征, 李清泉, 等. 面向车载激光扫描点云快速分类的点云特征图像生成方法[J]. 测绘学报, 2010,39(5):540-545. YANG Bisheng, WEI Zheng, LI Qingquan, et al. A classification-oriented method of feature image generation for vehicle-borne laser scanning point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2010,39(5):540-545.
[3] 李德仁. 展望5G/6G时代的地球空间信息技术[J]. 测绘学报, 2019,48(12):1475-1481. DOI: 10.11947/j.AGCS.2019.20190437. LI Deren. Towards geospatial information technology in 5G/6G era[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(12):1475-1481. DOI: 10.11947/j.AGCS.2019.20190437.
[4] 杨必胜, 董震. 点云智能研究进展与趋势[J]. 测绘学报, 2019,48(12):1575-1585. DOI: 10.11947/j.AGCS.2019.20190465. YANG Bisheng, DONG Zhen. Progress and perspective of point cloud intelligence[J]. Acta Geodaetica et Cartographica Sinica, 2019,48(12):1575-1585. DOI: 10.11947/j.AGCS.2019.20190465.
[5] YANG Bisheng, WEI Zheng, LI Qingquan, et al. Automated extraction of street-scene objects from mobile lidar point clouds[J]. International Journal of Remote Sensing, 2012,33(18):5839-5861.
[6] 李婷, 詹庆明, 喻亮. 基于地物特征提取的车载激光点云数据分类方法[J]. 国土资源遥感, 2012, 24(1):17-21. LI Ting, ZHAN Qingming, YU Liang. A classification method for mobile laser scanning data based on object feature extraction[J]. Remote Sensing for Land & Resources, 2012, 24(1):17-21.
[7] LEHTOMÄKI M, JAAKKOLA A, HYYPPÄ J, et al. Object classification and recognition from mobile laser scanning point clouds in a road environment[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016,54(2):1226-1239.
[8] YU Yongtao, LI J, WEN Chenglu, et al. Bag-of-visual-phrases and hierarchical deep models for traffic sign detection and recognition in mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016,113:106-123.
[9] XIAO Wen, VALLET B, SCHINDLER K, et al. Street-side vehicle detection, classification and change detection using mobile laser scanning data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016,114:166-178.
[10] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015,521(7553):436-444.
[11] GUAN Haiyan, YU Yongtao, JI Zheng, et al. Deep learning-based tree classification using mobile LiDAR data[J]. Remote Sensing Letters, 2015,6(11):864-873.
[12] 罗海峰, 方莉娜, 陈崇成, 等. 基于DBN的车载激光点云路侧多目标提取[J]. 测绘学报, 2018,47(2):234-246. DOI: 10.11947/j.AGCS.2018.20170524. LUO Haifeng, FANG Lina, CHEN Chongcheng, et al. Roadside multiple objects extraction from mobile laser scanning point cloud based on DBN[J]. Acta Geodaetica et Cartographica Sinica, 2018,47(2):234-246. DOI: 10.11947/j.AGCS.2018.20170524.
[13] SU Hang, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 945-953.
[14] QI C R, SU Hao, NIEßNER M, et al. Volumetric and multi-view CNNs for object classification on 3D data[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 5648-5656.
[15] VISHWANATH K V, GUPTA D, VAHDAT A, et al. ModelNet: Towards a datacenter emulation environment[C]//Proceedings of 2009 IEEE Ninth International Conference on Peer-to-Peer Computing. Seattle, WA, USA: IEEE, 2009: 81-82.
[16] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[17] FENG Yifan, ZHANG Zizhao, ZHAO Xibin, et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 264-272.
[18] YU Tan, MENG Jingjing, YUAN Junsong. Multi-view harmonized bilinear network for 3D object recognition[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 186-194.
[19] CHARLES R Q, HAO Su, MO Kaichun, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 77-85.
[20] BELLO S A, YU Shangshu, WANG Cheng, et al. Review: deep learning on 3D point clouds[J]. Remote Sensing, 2020, 12(11): 1729.DOI:org/10.3390/rs12111729.
[21] QI C R, YI LI, SU HAO, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space [C]//Proceedings of the 31st Conference on Neural Information Processing Systems. Long Beach, CA, USA:NIPS,2017.
[22] JOSEPH-RIVLIN M, ZVIRIN A, KIMMEL R. Momenet: flavor the moments in learning to classify shapes[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea (South): IEEE, 2019: 4085-4094.
[23] LI Yangyan, BU Rui, SUN Mingchao, et al. PointCNN: convolution on x-transformed points[C]//Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018). NowYork, NY, USA: Curran Associates, 2018: 820-830.
[24] ZHAO Hengshuang, JIANG Li, FU C W, et al. PointWeb: enhancing local neighborhood features for point cloud processing[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2019: 5560-5568.
[25] THOMAS H, QI C R, DESCHAUD J E, et al. KPConv: flexible and deformable convolution for point clouds[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 6410-6419.
[26] WANG Yue, SUN Yongbin, LIU Ziwei, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 1-12.
[27] QI C R, LIU Wei, WU Chenxia, et al. Frustum PointNets for 3D object detection from RGB-D data[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 918-927.
[28] YOU Haoxuan, FENG Yifan, JI Rongrong, et al. PVNet: a joint convolutional network of point cloud and multi-view for 3D shape recognition[C]//Proceedings of the 26th ACM international conference on Multimedia. New York, NY, USA: ACM, 2018: 4561-4570.
[29] YOU Haoxuan, FENG Yifan, ZHAO Xibin, et al. PVRNet: point-view relation neural network for 3D shape recognition[C]//Proceedings of 2019 AAAI Conference on Artificial Intelligence. Palo Alto, California, USA: AAAI, 2019, 33: 9119-9126.
[30] GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: The KITTI dataset[J]. The International Journal of Robotics Research, 2013,32(11):1231-1237.
[31] DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 2432-2443.
[32] ZHANG Wuming, QI Jianbo, WAN Peng, et al. An easy-to-use airborne LiDAR data filtering method based on cloth simulation[J]. Remote Sensing, 2016,8(6):501.
[33] REINGOLD O. Undirected connectivity in log-space[J]. Journal of the ACM, 2008,55(4):1-24.
[34] YU Y, LI J, GUAN H et al. Automated extraction of 3D trees from mobile LiDAR point clouds[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2014, XL-5: 629-632.
[35] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA: IEEE, 2009: 248-255.
[36] ROYNARD X, DESCHAUD J E, GOULETTE F. Paris-Lille-3D: a large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification[J]. The International Journal of Robotics Research, 2018,37(6):545-557.