Road markings extraction considering topological structure

doi:10.11947/j.AGCS.2024.20230289

Abstract

Abstract:

Road markings are important traffic sign information, and onboard LiDAR point clouds provide high precision 3D coordinates and reflectance intensity information for their extraction. Due to factors such as scanning distance and target material, the different object may exhibit similar intensity values, causing interference in the extraction of road markings. Wear and aging during road use can also damage the original structure of the markings, resulting in discontinuities after extraction. In addition, the diversity of road markings and their varying occurrence frequencies in practice can lead to low classification accuracy for categories with fewer samples in the segmentation network extraction results. To address these issues, this paper proposes a two-stage segmentation and classification extraction method that accurately extracts various types of markings and has topological robustness. Firstly, a multi-layer perceptron is used to adaptively learn the relationship between intensity and its influencing factors, and to perform intensity correction on the road point clouds. Secondly, the semantic segmentation network link spatial topology net (LST-Net) is proposed to segment all road markings, which captures line structure information using row-column convolution and attention mechanisms, and is trained with topological punishment to determine the positions of markings. Finally, YOLOv5 is used to detect the markings, and a separate classification network is trained to address the issue of sample imbalance in segmentation. Experiments are conducted on three sets of point clouds from different driving scenarios, and the results show that our approach achieves a marking extraction accuracy of 94.1% and a recall rate of 95.6%, demonstrating strong practicality and effectiveness.

Key words: vehicle-mounted LiDAR point clouds, road markings extraction, topological structure, semantic segmentation, object detection

CLC Number:

P237

Jiaxing LIU, Yuchun HUANG, Wenxuan SHI, Xi YE, He YANG. Road markings extraction considering topological structure[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(11): 2213-2227.

Figures/Tables 25

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Tab.1

Fig.12

Fig.13

Fig.14

Fig.15

Tab.2

Tab.3

Fig.16

Tab.4

Fig.17

Tab.5

Fig.18

Tab.6

Tab.7

References 35

[1]	MATHIBELA B, NEWMAN P, POSNER I. Reading the road: road marking classification and interpretation[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(4): 2072-2081.
[2]	李德仁, 洪勇, 王密, 等. 测绘遥感能为智能驾驶做什么?[J]. 测绘学报, 2021, 50(11): 1421-1431.DOI:. doi: 10.11947/j.AGCS.2021.20210280
	LI Deren, HONG Yong, WANG Mi, et al. What can surveying and remote sensing do for intelligent driving?[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(11): 1421-1431.DOI:. doi: 10.11947/j.AGCS.2021.20210280
[3]	WU Tao, RANGANATHAN A. A practical system for road marking detection and recognition[C]//Proceedings of 2012 IEEE Intelligent Vehicles Symposium. Madrid: IEEE, 2012: 25-30.
[4]	刘经南, 吴杭彬, 郭迟, 等. 高精度道路导航地图的进展与思考[J]. 中国工程科学, 2018, 20(2): 99-105.
	LIU Jingnan, WU Hangbin, GUO Chi, et al. Progress and consideration of high precision road navigation map[J]. Strategic Study of CAE, 2018, 20(2): 99-105.
[5]	KIM J G, YOO J H, KOO J C. Road and lane detection using stereo camera[C]//Proceedings of 2018 IEEE International Conference on Big Data and Smart Computing. Shanghai: IEEE, 2018: 649-652.
[6]	AZIMI S M, FISCHER P, KORNER M, et al. Aerial LaneNet: lane-marking semantic segmentation in aerial imagery using wavelet-enhanced cost-sensitive symmetric fully convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(5): 2920-2938.
[7]	YIN Weiling, QIAN Mingyang, WANG Lijun, et al. Road extraction from satellite images with iterative cross-task feature enhancement[J]. Neurocomputing, 2022, 506: 300-310.
[8]	YANG Bisheng, WEI Zheng, LI Qingquan, et al. Automated extraction of street-scene objects from mobile LiDAR point clouds[J]. International Journal of Remote Sensing, 2012, 33(18): 5839-5861.
[9]	SMADJA L, NINOT J, GAVRILOVIC T. Road extraction and environment interpretation from LiDAR sensors[J]. IAPRS, 2010, 38: 281-286.
[10]	TOTH C, PASKAA E, BRZEZINSKA D. Using road pavement markings as ground control for LiDAR data[J]. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2008, 37(Part B1): 189-195.
[11]	HATA A, WOLF D. Road marking detection using LiDAR reflective intensity data and its application to vehicle localization[C]//Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems. Qingdao: IEEE, 2014: 584-589.
[12]	谭凯, 程效军, 张吉星. TLS强度数据的入射角及距离效应改正方法[J]. 武汉大学学报(信息科学版), 2017, 42(2): 223-228.
	TAN Kai, CHENG Xiaojun, ZHANG Jixing. Correction for incidence angle and distance effects on TLS intensity data[J]. Geomatics and Information Science of Wuhan University, 2017, 42(2): 223-228.
[13]	CHEN Xin, KOHLMEYER B, STROILA M, et al. Next generation map making: geo-referenced ground-level LiDAR point clouds for automatic retro-reflective road feature extraction[C]//Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. Seattle: ACM Press, 2009: 488-491.
[14]	WAN Rui, HUANG Yuchun, XIE Rongchang, et al. Combined lane mapping using a mobile mapping system[J]. Remote Sensing, 2019, 11(3): 305.
[15]	HUANG Pengdi, CHENG Ming, CHEN Yiping, et al. Traffic sign occlusion detection using mobile laser scanning point clouds[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(9): 2364-2376.
[16]	JUNG J, CHE Erzhuo, OLSEN M J, et al. Efficient and robust lane marking extraction from mobile LiDAR point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 147: 1-18.
[17]	PAN Yue, YANG Bisheng, LI Shengfu, et al. Automatic road markings extraction, classification and vectorization from mobile laser scanning data[EB/OL]. [2023-07-01].https://isprs-archives.copernicus.org/articles/XLII-2-W13/1089/2019/isprs-archives-XLII-2-W13-1089-2019.pdf.
[18]	YE Chengming, ZHAO He, MA Lingfei, et al. Robust lane extraction from MLS point clouds towards HD maps especially in curve road[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(2): 1505-1518.
[19]	WEN Chenglu, SUN Xiaotian, LI J, et al. A deep learning framework for road marking extraction, classification and completion from mobile laser scanning point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 147: 178-192.
[20]	CHENG Yiting, PATEL A, WEN Chenglu, et al. Intensity thresholding and deep learning based lane marking extraction and lane width estimation from mobile light detection and ranging (LiDAR) point clouds[J]. Remote Sensing, 2020, 12(9): 1379.
[21]	MA Lingfei, LI Ying, LI J, et al. Capsule-based networks for road marking extraction and classification from mobile LiDAR point clouds[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(4): 1981-1995.
[22]	CHEN Siyun, ZHANG Zhenxin, ZHONG Ruofei, et al. A dense feature pyramid network-based deep learning model for road marking instance segmentation using MLS point clouds[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(1): 784-800.
[23]	CHAURASIA A, CULURCIELLO E. LinkNet: exploiting encoder representations for efficient semantic segmentation[C]//Proceedings of 2017 IEEE Visual Communications and Image Processing. St. Petersburg: IEEE, 2017: 1-4.
[24]	PAN Xingang, SHI Jianping, LUO Ping, et al. Spatial as deep: spatial CNN for traffic scene understanding[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. [S.l.]: AAAI, 2018.
[25]	ZHANG Han, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[C]//Proceedings of the 36th International Conference on Machine Learning. Long Beach: [s.n.], 2019: 7354-7363.
[26]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[27]	SASAKI K, IIZUKA S, SIMO-SERRA E, et al. Joint gap detection and inpainting of line drawings[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5725-5733.
[28]	CHENG Yiting, LIN Yichun, HABIB A. Generalized LiDAR intensity normalization and its positive impact on geometric and learning-based lane marking detection[J]. Remote Sensing, 2022, 14(17): 4393.
[29]	BAI Jie, NIU Zheng, GAO Shuai, et al. An exploration, analysis, and correction of the distance effect on terrestrial hyperspectral LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 198: 60-83.
[30]	ZHENG Tu, FANG Hao, ZHANG Yi, et al. RESA: recurrent feature-shift aggregator for lane detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(4): 3547-3554.
[31]	FENG Zhengyang, GUO Shaohua, TAN Xin, et al. Rethinking efficient lane detectionvia curve modeling[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17062-17070.
[32]	WANG Jinsheng, MA Yinchao, HUANG Shaofei, et al. A keypoint-based global association network for lane detection[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1392-1401.
[33]	MOSINSKA A, MARQUEZ-NEILA P, KOZINSKI M, et al. Beyond the pixel-wise loss for topology-aware delineation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018.
[34]	SHIT S, PAETZOLD J C, SEKUBOYINA A, et al. clDice—a novel topology-preserving loss function for tubular structure segmentation[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021.
[35]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of 2015 MICCAI. Cham: Springer International Publishing, 2015: 234-241.

路段名称	数据总量	训练集	测试集
路段1	1856	1299	557
路段2	2560	1792	768
路段3	4800	3360	1440

试验数据	网络结构	精度	召回率	F₁值	IOU
路段1	RESA^[30]	0.883 1	0.925 2	0.90 36	0.824 7
路段1	LST-Net	0.938 9	0.967 9	0.953 2	0.910 5
路段2	RESA^[30]	0.876 0	0.881 1	0.878 5	0.783 3
路段2	LST-Net	0.941 5	0.948 8	0.945 2	0.896 2
路段3	RESA^[30]	0.885 6	0.899 6	0.892 6	0.806 4
路段3	LST-Net	0.942 7	0.953 3	0.947 9	0.901 1

网络结构名称	编解码部分	注意力模块	行列卷积模块
A	√
B	√	√
C	√		√
D（LST-Net）	√	√	√

网络结构名称	精度	召回率	F₁值	IOU
A	0.927 5	0.606 2	0.733 2	0.575 7
B	0.906 4	0.848 1	0.876 3	0.779 3
C	0.882 4	0.825 6	0.853 1	0.739 7
D（LSA-Net）	0.890 1	0.902 2	0.896 1	0.811 9

所用损失函数	精度	召回率	F₁值	IOU
Dice损失	0.890 1	0.902 2	0.896 1	0.811 9
文献[33]的损失函数	0.879 8	0.861 5	0.870 5	0.770 4
文献[34]的损失函数	0.901 4	0.894 9	0.898 1	0.815 6
本文提出的损失函数	0.941 5	0.948 8	0.945 2	0.896 2