Multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning

doi:10.11947/j.AGCS.2023.20220239

Abstract

Abstract: Multi-pedestrian trajectory prediction is one of the key factors in integrating urban geographic information system and intelligent transportation. To address the problems of insufficient training data, difficult labeling, and low accuracy of pedestrian trajectory prediction in multi-view scenes for existing methods, we propose a novel multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning. First, a simulation simulator is used to generate the required multi-view pedestrian trajectory annotation data. Then, we mix up the trajectory of the selected view and the adversarial trajectory by a convex combination function to generate the enhanced adversarial trajectory. Next, an advanced detection and tracking algorithm is used to encode and track pedestrian appearance information. Furthermore, the enhanced trajectory and coding information are used as the feature input of a graph attention recurrent neural network to model pedestrian interaction. Finally, the pedestrian trajectory is decoded by a position decoder to extract pedestrian motion characteristics, and multi-pedestrian trajectory prediction is completed. The ADE and FDE accuracies of our method on the ETH/UCY fixed-view dataset are 0.41 and 0.82, respectively. The ADE accuracy on the ActEV/VIRAT and Argoverse multi-view datasets is 17.74 and 65.4, and the FDE accuracy is 34.96 and 172.8.

Key words: 3D simulation, deep learning, trajectory prediction, object tracking, urban geographic information system

CLC Number:

P208

CAO Xingwen, ZHENG Hongwei, LIU Ying, WU Mengquan, WANG Lingyue, BAO Anming, CHEN Xi. Multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(9): 1595-1608.

References

[1] SHANG Yang, SUN Xiaoliang, ZHANG Yueqiang, et al. Research on 3D target pose tracking and modeling[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 60-69.
[2] 张旭, 郝向阳, 李建胜, 等. 监控视频中动态目标与地理空间信息的融合与可视化方法[J]. 测绘学报, 2019, 48(11): 1415-1423. DOI: 10.11947/j.AGCS.2019.20180572. ZHANG Xu, HAO Xiangyang, LI Jiansheng, et al. Fusion and visualization method of dynamic targets in surveillance video with geospatial information[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(11): 1415-1423. DOI: 10.11947/j.AGCS.2019.20180572.
[3] 李德仁, 胡庆武. 基于可量测实景影像的空间信息服务[J]. 武汉大学学报(信息科学版), 2007, 32(5): 377-380, 418. LI Deren, HU Qingwu. Digital measurable image based geo-spatial information service[J]. Geomatics and Information Science of Wuhan University, 2007, 32(5): 377-380, 418.
[4] CHAI Yuning, SAPP B, BANSAL M, et al. MultiPath: multiple probabilistic anchor trajectory hypotheses for behavior prediction[C]//Proceedings of 2019 Conference on Robot Learning. Osaka: Robot Learning Foundation, Inc, 2019:86-99.
[5] LIANG Junwei, JIANG Lu, NIEBLES J C, et al. Peeking into the future: predicting future person activities and locations in videos[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5718-5727.
[6] RHINEHART N, KITANI K M, VERNAZA P. R2P2: a reparameterized pushforward policy for diverse, precise generative path forecasting[C]//Proceedings of 2018 ECCV. Cham: Springer, 2018: 794-811.
[7] 李景文, 韦晶闪, 姜建武, 等. 多视角监控视频中动态目标的时空信息提取方法[J]. 测绘学报, 2022, 51(3): 388-400. DOI: 10.11947/j.AGCS.2022.20200507. LI Jingwen, WEI Jingshan, JIANG Jianwu, et al. Spatio-temporal information extraction method for dynamic targets in multi-perspective surveillance video[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(3): 388-400. DOI: 10.11947/j.AGCS.2022.20200507.
[8] CHOI W, SAVARESE S. A unified framework for multi-target tracking and collective activity recognition[C]//Proceedings of 2012 European Conference on Computer Vision. Berlin: Springer, 2012: 215-230.
[9] HELBING D, MOLNÁR P. Social force model for pedestrian dynamics[J]. Physical Review E, 1995, 51(5): 4282-4286.
[10] ALAHI A, GOEL K, RAMANATHAN V, et al. Social LSTM: human trajectory prediction in crowded spaces[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 961-971.
[11] GUPTA A, JOHNSON J, LI Feifei, et al. Social GAN: socially acceptable trajectories with generative adversarial networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2255-2264.
[12] SADEGHIAN A, KOSARAJU V, SADEGHIAN A, et al. SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 1349-1358.
[13] 张志远, 刁英华. 结合社会特征和注意力的行人轨迹预测模型[J]. 西安电子科技大学学报, 2020, 47(1): 10-17, 79. ZHANG Zhiyuan, DIAO Yinghua. Pedestrian trajectory prediction model with social features and attention[J]. Journal of Xidian University, 2020, 47(1): 10-17, 79.
[14] VEMULA A, MUELLING K, OH J. Social attention: modeling attention in human crowds[C]//Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane: IEEE, 2018: 4601-4607.
[15] HUANG Yingfan, BI Huikun, LI Zhaoxin, et al. STGAT: modeling spatial-temporal interactions for human trajectory prediction[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 6271-6280.
[16] YU Cunjun, MA Xiao, REN Jiawei, et al. Spatio-temporal graph transformer networks for pedestrian trajectory prediction[C]//Proceedings of 2020 European Conference on Computer Vision. Cham: Springer, 2020: 507-523.
[17] OH S, HOOGS A, PERERA A, et al. AVSS 2011 demo session: a large-scale benchmark dataset for event recognition in surveillance video[C]//Proceedings of 2011 Conference on Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011: 3153-3160.
[18] LI Yuke. Which way are You going? Imitative decision learning for path forecasting in dynamic scenes[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 294-303.
[19] CHANG Mingfang, LAMBERT J, SANGKLOY P, et al. Argoverse: 3D tracking and forecasting with rich maps[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 8740-8749.
[20] GAIDON A, WANG Qiao, CABON Y, et al. Virtual worlds as proxy for multi-object tracking analysis[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4340-4349.
[21] WRENNINGE M, UNGER J. Synscapes: a photorealistic synthetic dataset for street scene parsing[EB/OL].[2023-08-22]. https://arxiv.org/abs/1810.08705.
[22] LIANG Junwei, JIANG Lu, MURPHY K, et al. The garden of forking paths: towards multi-future trajectory prediction[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 10505-10515.
[23] DOSOVITSKIY A, ROS G, CODEVILLA F, et al. CARLA: an open urban driving simulator[EB/OL].[2023-08-22]. https://arxiv.org/abs/1711.03938.
[24] ZHANG Yi, WEI Xinyue, QIU Weichao, et al. RSA: randomized simulation as augmentation for robust human action recognition[EB/OL].[2023-08-22]. https://arxiv.org/abs/1912.01180.
[25] LI Lihuan, PAGNUCCO M, SONG Yang. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 2221-2231.
[26] LIANG Junwei, JIANG Lu, HAUPTMANN A. SimAug: learning robust representations from simulation for trajectory prediction[C]//Proceedings of 2020 European Conference on Computer Vision. Cham: Springer, 2020: 275-292.
[27] SHIRAZI M S, MORRIS B T. Trajectory prediction of vehicles turning at intersections using deep neural networks[J]. Machine Vision and Applications, 2019, 30(6): 1097-1109.
[28] YOON Y C, KIM D Y, SONG Y M, et al. Online multiple pedestrians tracking using deep temporal appearance matching association[J]. Information Sciences, 2021, 561: 326-351.
[29] LI J, CHEN C, CHENG T. Motion prediction and robust tracking of a dynamic and temporarily-occluded target by an unmanned aerial vehicle[J]. IEEE Transactions on Control Systems Technology, 2021, 29(4): 1623-1635.
[30] ZHANG Hongyi, CISSE M, DAUPHIN Y N, et al. Mixup: beyond empirical risk minimization[EB/OL].[2023-08-22]. https://arxiv.org/abs/1710.09412.
[31] HU Han, GU Jiayuan, ZHANG Zheng, et al. Relation networks for object detection[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3588-3597.
[32] WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric[C]//Proceedings of 2017 IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 3645-3649.
[33] AWAD G, BUTT A, CURTIS K, et al. TRECVID 2018: benchmarking video activity detection, video captioning and matching, video storytelling linking and video search[EB/OL]. [2023-08-22]. http://www.tara.tcd.ie/bitstream/handle/2262/96111/tv18overview.pdf?sequence=1&isAllowed=y.
[34] ZHOU Bolei, ZHAO Hang, PUIG X, et al. Scene parsing through ADE20K dataset[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5122-5130.
[35] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[36] JIANG Lu, ZHOU Zhengyuan, LEUNG T, et al. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels[EB/OL].[2023-08-22]. https://arxiv.org/abs/1712.05055.
[37] KURAKIN A, GOODFELLOW I J, BENGIO S. Adversarial examples in the physical world[EB/OL]. [2023-08-22].https://arxiv.org/abs/1607.02533.
[38] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[39] PELLEGRINI S, ESS A, SCHINDLER K, et al. You'll never walk alone: modeling social behavior for multi-target tracking[C]//Proceedings of 2009 IEEE International Conference on Computer Vision. Kyoto: IEEE, 2009: 261-268.
[40] LERNER A, CHRYSANTHOU Y, LISCHINSKI D. Crowds by example[J]. Computer Graphics Forum, 2007, 26(3): 655-664.