多行人轨迹多视角三维仿真视频学习预测法

doi:10.11947/j.AGCS.2023.20220239

测绘学报 ›› 2023, Vol. 52 ›› Issue (9): 1595-1608.doi: 10.11947/j.AGCS.2023.20220239

多行人轨迹多视角三维仿真视频学习预测法

曹兴文^1,2, 郑宏伟^1,2, 刘英^1,2, 吴孟泉³, 王灵玥^1,2, 包安明^1,2, 陈曦^1,2

1. 中国科学院新疆生态与地理研究所荒漠与绿洲生态国家重点实验室, 新疆乌鲁木齐 830011;
2. 中国科学院大学资源与环境学院, 北京 100049;
3. 鲁东大学资源与环境工程学院, 山东烟台 264025

收稿日期:2022-04-06 修回日期:2023-06-08 发布日期:2023-10-12
通讯作者: 郑宏伟 E-mail:hzheng@ms.xjb.ac.cn
作者简介:曹兴文(1997-),男,硕士生,研究方向为行人轨迹预测。E-mail:caoxingwen21@mails.ucas.ac.cn
基金资助:
新疆维吾尔自治区重点研发专项(2022B03001-3;2022B03021-3);国家自然科学基金(NSFCU-U1803120);中国科学院“西部之光”人才培养计划(2021-XBQNXZ-012);国家自然科学基金(42071385)

Multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning

CAO Xingwen^1,2, ZHENG Hongwei^1,2, LIU Ying^1,2, WU Mengquan³, WANG Lingyue^1,2, BAO Anming^1,2, CHEN Xi^1,2

1. State Key Laboratory of Desert and Oasis Ecology, Xinjiang Institute of Ecology and Geography, Chinese Academy of Sciences, Urumqi 830011, China;
2. College of Resources and Environment, University of Chinese Academy of Sciences, Beijing 100049, China;
3. School of Resources and Environmental Engineering, Ludong University, Yantai 264025, China

Received:2022-04-06 Revised:2023-06-08 Published:2023-10-12
Supported by:
The Key Research and Development Program of Xinjiang Uygur Autonomous Region (Nos. 2022B03001-3;2022B03021-3);The National Natural Science Foundation of China (No. NSFCU-U1803120);The Western Light Talents Training Program of CAS (No. 2021-XBQNXZ-012);The National Natural Science Foundation of China (No. 42071385)

摘要/Abstract

摘要： 多行人轨迹预测是城市地理信息系统与智能交通融合的关键因素之一。针对现有行人轨迹预测方法训练数据量不足、标注难、对多视角场景轨迹预测精度低等问题,本文提出一种多行人轨迹多视角三维仿真视频学习预测方法。首先,通过仿真模拟器生成所需多视角行人轨迹标注数据,利用凸函数组合原始视角对抗轨迹和选定的多视角轨迹,生成增强对抗轨迹,接着使用检测跟踪算法对行人特征信息进行编码;然后,将增强轨迹和编码信息作为图注意力循环神经网络的特征输入,对行人交互信息建模;最后,通过位置解码器对行人轨迹进行解码并提取行人运动特征,完成多行人轨迹预测。本文方法在ETH/UCY固定视角数据集上的ADE和FDE精度分别为0.41和0.82。在ActEV/VIRAT和Argoverse多视角数据集上的ADE精度为17.74和65.4,FDE精度为34.96和172.8。

关键词: 三维仿真, 深度学习, 轨迹预测, 目标跟踪, 城市地理信息系统

Abstract: Multi-pedestrian trajectory prediction is one of the key factors in integrating urban geographic information system and intelligent transportation. To address the problems of insufficient training data, difficult labeling, and low accuracy of pedestrian trajectory prediction in multi-view scenes for existing methods, we propose a novel multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning. First, a simulation simulator is used to generate the required multi-view pedestrian trajectory annotation data. Then, we mix up the trajectory of the selected view and the adversarial trajectory by a convex combination function to generate the enhanced adversarial trajectory. Next, an advanced detection and tracking algorithm is used to encode and track pedestrian appearance information. Furthermore, the enhanced trajectory and coding information are used as the feature input of a graph attention recurrent neural network to model pedestrian interaction. Finally, the pedestrian trajectory is decoded by a position decoder to extract pedestrian motion characteristics, and multi-pedestrian trajectory prediction is completed. The ADE and FDE accuracies of our method on the ETH/UCY fixed-view dataset are 0.41 and 0.82, respectively. The ADE accuracy on the ActEV/VIRAT and Argoverse multi-view datasets is 17.74 and 65.4, and the FDE accuracy is 34.96 and 172.8.

Key words: 3D simulation, deep learning, trajectory prediction, object tracking, urban geographic information system

中图分类号:

P208

曹兴文, 郑宏伟, 刘英, 吴孟泉, 王灵玥, 包安明, 陈曦. 多行人轨迹多视角三维仿真视频学习预测法[J]. 测绘学报, 2023, 52(9): 1595-1608.

CAO Xingwen, ZHENG Hongwei, LIU Ying, WU Mengquan, WANG Lingyue, BAO Anming, CHEN Xi. Multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(9): 1595-1608.

参考文献

[1] SHANG Yang, SUN Xiaoliang, ZHANG Yueqiang, et al. Research on 3D target pose tracking and modeling[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2): 60-69.
[2] 张旭, 郝向阳, 李建胜, 等. 监控视频中动态目标与地理空间信息的融合与可视化方法[J]. 测绘学报, 2019, 48(11): 1415-1423. DOI: 10.11947/j.AGCS.2019.20180572. ZHANG Xu, HAO Xiangyang, LI Jiansheng, et al. Fusion and visualization method of dynamic targets in surveillance video with geospatial information[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(11): 1415-1423. DOI: 10.11947/j.AGCS.2019.20180572.
[3] 李德仁, 胡庆武. 基于可量测实景影像的空间信息服务[J]. 武汉大学学报(信息科学版), 2007, 32(5): 377-380, 418. LI Deren, HU Qingwu. Digital measurable image based geo-spatial information service[J]. Geomatics and Information Science of Wuhan University, 2007, 32(5): 377-380, 418.
[4] CHAI Yuning, SAPP B, BANSAL M, et al. MultiPath: multiple probabilistic anchor trajectory hypotheses for behavior prediction[C]//Proceedings of 2019 Conference on Robot Learning. Osaka: Robot Learning Foundation, Inc, 2019:86-99.
[5] LIANG Junwei, JIANG Lu, NIEBLES J C, et al. Peeking into the future: predicting future person activities and locations in videos[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5718-5727.
[6] RHINEHART N, KITANI K M, VERNAZA P. R2P2: a reparameterized pushforward policy for diverse, precise generative path forecasting[C]//Proceedings of 2018 ECCV. Cham: Springer, 2018: 794-811.
[7] 李景文, 韦晶闪, 姜建武, 等. 多视角监控视频中动态目标的时空信息提取方法[J]. 测绘学报, 2022, 51(3): 388-400. DOI: 10.11947/j.AGCS.2022.20200507. LI Jingwen, WEI Jingshan, JIANG Jianwu, et al. Spatio-temporal information extraction method for dynamic targets in multi-perspective surveillance video[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(3): 388-400. DOI: 10.11947/j.AGCS.2022.20200507.
[8] CHOI W, SAVARESE S. A unified framework for multi-target tracking and collective activity recognition[C]//Proceedings of 2012 European Conference on Computer Vision. Berlin: Springer, 2012: 215-230.
[9] HELBING D, MOLNÁR P. Social force model for pedestrian dynamics[J]. Physical Review E, 1995, 51(5): 4282-4286.
[10] ALAHI A, GOEL K, RAMANATHAN V, et al. Social LSTM: human trajectory prediction in crowded spaces[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 961-971.
[11] GUPTA A, JOHNSON J, LI Feifei, et al. Social GAN: socially acceptable trajectories with generative adversarial networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2255-2264.
[12] SADEGHIAN A, KOSARAJU V, SADEGHIAN A, et al. SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 1349-1358.
[13] 张志远, 刁英华. 结合社会特征和注意力的行人轨迹预测模型[J]. 西安电子科技大学学报, 2020, 47(1): 10-17, 79. ZHANG Zhiyuan, DIAO Yinghua. Pedestrian trajectory prediction model with social features and attention[J]. Journal of Xidian University, 2020, 47(1): 10-17, 79.
[14] VEMULA A, MUELLING K, OH J. Social attention: modeling attention in human crowds[C]//Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane: IEEE, 2018: 4601-4607.
[15] HUANG Yingfan, BI Huikun, LI Zhaoxin, et al. STGAT: modeling spatial-temporal interactions for human trajectory prediction[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019: 6271-6280.
[16] YU Cunjun, MA Xiao, REN Jiawei, et al. Spatio-temporal graph transformer networks for pedestrian trajectory prediction[C]//Proceedings of 2020 European Conference on Computer Vision. Cham: Springer, 2020: 507-523.
[17] OH S, HOOGS A, PERERA A, et al. AVSS 2011 demo session: a large-scale benchmark dataset for event recognition in surveillance video[C]//Proceedings of 2011 Conference on Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011: 3153-3160.
[18] LI Yuke. Which way are You going? Imitative decision learning for path forecasting in dynamic scenes[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 294-303.
[19] CHANG Mingfang, LAMBERT J, SANGKLOY P, et al. Argoverse: 3D tracking and forecasting with rich maps[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 8740-8749.
[20] GAIDON A, WANG Qiao, CABON Y, et al. Virtual worlds as proxy for multi-object tracking analysis[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4340-4349.
[21] WRENNINGE M, UNGER J. Synscapes: a photorealistic synthetic dataset for street scene parsing[EB/OL].[2023-08-22]. https://arxiv.org/abs/1810.08705.
[22] LIANG Junwei, JIANG Lu, MURPHY K, et al. The garden of forking paths: towards multi-future trajectory prediction[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 10505-10515.
[23] DOSOVITSKIY A, ROS G, CODEVILLA F, et al. CARLA: an open urban driving simulator[EB/OL].[2023-08-22]. https://arxiv.org/abs/1711.03938.
[24] ZHANG Yi, WEI Xinyue, QIU Weichao, et al. RSA: randomized simulation as augmentation for robust human action recognition[EB/OL].[2023-08-22]. https://arxiv.org/abs/1912.01180.
[25] LI Lihuan, PAGNUCCO M, SONG Yang. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 2221-2231.
[26] LIANG Junwei, JIANG Lu, HAUPTMANN A. SimAug: learning robust representations from simulation for trajectory prediction[C]//Proceedings of 2020 European Conference on Computer Vision. Cham: Springer, 2020: 275-292.
[27] SHIRAZI M S, MORRIS B T. Trajectory prediction of vehicles turning at intersections using deep neural networks[J]. Machine Vision and Applications, 2019, 30(6): 1097-1109.
[28] YOON Y C, KIM D Y, SONG Y M, et al. Online multiple pedestrians tracking using deep temporal appearance matching association[J]. Information Sciences, 2021, 561: 326-351.
[29] LI J, CHEN C, CHENG T. Motion prediction and robust tracking of a dynamic and temporarily-occluded target by an unmanned aerial vehicle[J]. IEEE Transactions on Control Systems Technology, 2021, 29(4): 1623-1635.
[30] ZHANG Hongyi, CISSE M, DAUPHIN Y N, et al. Mixup: beyond empirical risk minimization[EB/OL].[2023-08-22]. https://arxiv.org/abs/1710.09412.
[31] HU Han, GU Jiayuan, ZHANG Zheng, et al. Relation networks for object detection[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3588-3597.
[32] WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric[C]//Proceedings of 2017 IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 3645-3649.
[33] AWAD G, BUTT A, CURTIS K, et al. TRECVID 2018: benchmarking video activity detection, video captioning and matching, video storytelling linking and video search[EB/OL]. [2023-08-22]. http://www.tara.tcd.ie/bitstream/handle/2262/96111/tv18overview.pdf?sequence=1&isAllowed=y.
[34] ZHOU Bolei, ZHAO Hang, PUIG X, et al. Scene parsing through ADE20K dataset[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5122-5130.
[35] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[36] JIANG Lu, ZHOU Zhengyuan, LEUNG T, et al. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels[EB/OL].[2023-08-22]. https://arxiv.org/abs/1712.05055.
[37] KURAKIN A, GOODFELLOW I J, BENGIO S. Adversarial examples in the physical world[EB/OL]. [2023-08-22].https://arxiv.org/abs/1607.02533.
[38] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[39] PELLEGRINI S, ESS A, SCHINDLER K, et al. You'll never walk alone: modeling social behavior for multi-target tracking[C]//Proceedings of 2009 IEEE International Conference on Computer Vision. Kyoto: IEEE, 2009: 261-268.
[40] LERNER A, CHRYSANTHOU Y, LISCHINSKI D. Crowds by example[J]. Computer Graphics Forum, 2007, 26(3): 655-664.

多行人轨迹多视角三维仿真视频学习预测法

Multi-pedestrian trajectory prediction method based on multi-view 3D simulation video learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	江宝得, 黄威, 许少芬, 巫勇. 融合分散自适应注意力机制的多尺度遥感影像建筑物实例细化提取[J]. 测绘学报, 2023, 52(9): 1504-1514.
[2]	杜培军, 张伟, 张鹏, 林聪, 郭山川, 胡泽周. 一种联合空谱特征的高光谱影像分类胶囊网络[J]. 测绘学报, 2023, 52(7): 1090-1104.
[3]	余旭初, 刘冰, 薛志祥. 高光谱地物要素识别潜力分析与前景展望[J]. 测绘学报, 2023, 52(7): 1115-1125.
[4]	杭仁龙, 李成相, 刘青山. 谱间对比学习的高光谱图像无监督特征提取[J]. 测绘学报, 2023, 52(7): 1164-1174.
[5]	胡鑫, 王心宇, 钟燕飞. 基于自适应上下文聚合网络的双高遥感影像分类[J]. 测绘学报, 2023, 52(7): 1175-1186.
[6]	张蒙蒙, 李伟, 刘欢, 赵旭东, 陶然. 基于形态变换与空间逻辑聚合的高光谱森林树种分类[J]. 测绘学报, 2023, 52(7): 1202-1211.
[7]	宋尚真, 杨怡欣, 王会峰, 王晓艳, 荣生辉, 周慧鑫. 高光谱图像稀疏约束与自编码器特征提取相结合的异常检测方法[J]. 测绘学报, 2023, 52(6): 932-943.
[8]	邓敏, 罗斌, 唐建波, 姚志鹏, 刘国平, 温翔, 胡润波, 柴华, 胡文柯. 顾及轨迹密度分布异质性的道路交叉口提取方法[J]. 测绘学报, 2023, 52(6): 1000-1009.
[9]	马梦锴, 董箭, 唐露露, 彭认灿, 周寅飞, 王芳. 兼顾要素正确分类及精准定位的栅格海图水深注记自动提取方法[J]. 测绘学报, 2023, 52(6): 1022-1036.
[10]	刘帅, 李笑迎, 于梦, 邢光龙. 高分辨率遥感图像双解耦语义分割网络模型[J]. 测绘学报, 2023, 52(4): 638-647.
[11]	张永显, 马国锐, 崔志祥, 张志军. 面向大视角差的无人机影像序列学习型特征匹配[J]. 测绘学报, 2023, 52(2): 230-243.
[12]	卜丽静, 吴文玉, 张正鹏, 杨银. 先验引导的多降质特征“珞珈一号”夜光遥感影像质量提升方法[J]. 测绘学报, 2023, 52(2): 244-259.
[13]	孙根云, 王鑫, 安娜, 张爱竹. 基于多源遥感的大尺度高分辨率不透水面深度学习提取方法[J]. 测绘学报, 2023, 52(2): 272-282.
[14]	禹文豪, 魏铖, 陈佳鑫. 面向不平衡POI类别分布的电子地图多层次标签预测模型[J]. 测绘学报, 2023, 52(2): 318-328.
[15]	杨敏, 陈果, 李连营, 黄浩然, 苗静, 晏雄锋. 序列卷积神经网络支持下线状地图目标的分段方法[J]. 测绘学报, 2023, 52(1): 108-116.