Detecting pedestrian intention using EEG signals in navigation

doi:10.11947/j.AGCS.2024.20230444

Abstract

Abstract:

The automatic recognition of pedestrian intentions is a difficult issue in location-based services, which is crucial for establishing intelligent navigation services and new human-computer interaction method. Currently, using behavior patterns to estimate pedestrian intentions has become a mainstream solution, but this approach relies on multiple sensors and has time delays. This article proposes a pedestrian intention detection method based on brain imaging technology, which interprets pedestrian turning intentions through multi-channel, high-resolution EEG signals. Firstly, according to the standard motor imagery paradigm, EEG samples corresponding to four types of intentions within road intersection scenes were collected, including straight ahead, stop, left turn, and right turn. Then, by fusing the features of EEG in time-frequency domain, spatial domain, and functional connectivity domain, the spatiotemporal functional connectivity networks (STFCNs) of EEG are constructed to express the process of EEG activity, facilitating the capture of EEG features highly related to the intent. Finally, a graph convolutional neural network was used to encode the STFCNs, completing the mapping from EEG to four types of navigation intentions. The experimental results show that the average accuracy (F₁ score) of detecting four types of intentions using a short time window (1 s) is 0.443±0.062, and the highest accuracy can reach 0.571. The average accuracy with a long time window (6 s) is 0.525±0.084, and the highest accuracy is 0.665. The detection accuracy of this method is slightly better than other classification algorithms, and its detection ability for forward and stop intentions is excellent, up to 0.740 and 0.700.

Key words: pedestrian navigation, intention detection, EEG, GCN

CLC Number:

P228

Zhixiang FANG, Lubin WANG. Detecting pedestrian intention using EEG signals in navigation[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(9): 1829-1841.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Tab.1

Fig.7

Fig.8

Fig.9

Tab.2

Fig.10

Fig.11

References 36

[1]	方志祥, 徐虹, 萧世伦, 等. 绝对空间定位到相对空间感知的行人导航研究趋势[J]. 武汉大学学报(信息科学版), 2018, 43(12):2173-2182.
	FANG Zhixiang, XU Hong, XIAO Shilun, et al. Pedestrian navigation research trend: from absolute space to relative space-based approach[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):2173-2182.
[2]	李德毅, 赵菲, 刘萌, 等. 自动驾驶量产的难点分析及展望[J]. 武汉大学学报(信息科学版), 2018, 43(12):1775-1779.
	LI Deyi, ZHAO Fei, LIU Meng, et al. Difficulty analysis and prospect of autonomous vehicle mass production[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12):1775-1779.
[3]	吕超, 崔格格, 孟相浩, 等. 基于图表示的智能车行人意图识别方法[J]. 北京理工大学学报, 2022, 42(7):688-695.
	LÜ Chao, CUI Gege, MENG Xianghao, et al. Graph representation method for pedestrian intention recognition of intelligent vehicle[J]. Transactions of Beijing Institute of Technology, 2022, 42(7):688-695.
[4]	方志祥, 罗浩, 李灵. 有限状态自动机辅助的行人导航状态匹配算法[J]. 测绘学报, 2017, 46(3):371-380. DOI: 10.11947/j.AGCS.2017.20160530.
	FANG Zhixiang, LUO Hao, LI Ling. A finite state machine aided pedestrian navigation state matching algorithm[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(3):371-380. DOI: 10.11947/j.AGCS.2017.20160530.
[5]	DOSHI A, TRIVEDI M M. On the roles of eye gaze and head dynamics in predicting driver's intent to change lanes[J]. IEEE Transactions on Intelligent Transportation Systems, 2009, 10(3):453-462.
[6]	SHI Bowen, DONG Weihua, ZHAN Zhicheng. AdaFI-FCN: an adaptive feature integration fully convolutional network for predicting driver's visual attention[J]. Geo-spatial Information Science, 2022, 27(4):1-17.
[7]	HEI Qiaosong, DONG Weihua, SHI Bowen. Detecting dynamic visual attention in augmented reality aided navigation environment based on a multi-feature integration fully convolutional network[J]. Cartography and Geographic Information Science, 2023, 50(1):63-78.
[8]	RAJWAL S, AGGARWAL S. Convolutional neural network-based EEG signal analysis: a systematic review[J]. Archives of Computational Methods in Engineering, 2023, 30(6):3585-3615.
[9]	WEI Ying, ZHOU Jun, WANG Yin, et al. A review of algorithm & hardware design for AI-based biomedical applications[J]. IEEE Transactions on Biomedical Circuits and Systems, 2020, 14(2):145-163.
[10]	KHALIFA Y, MANDIC D, SEJDIĆ E. A review of hidden Markov models and recurrent neural networks for event detection and localization in biomedical signals[J]. Information Fusion, 2021, 69:52-72.
[11]	ZABIHI S M, BEAUCHEMIN S S, BAUER M A. Real-time driving manoeuvre prediction using IO-HMM and driver cephalo-ocular behaviour[C]//Proceedings of 2017 IEEE Intelligent Vehicles Symposium. Los Angeles: IEEE, 2017: 875-880.
[12]	张杨松, 卓彦, 尧德中. 脑电磁成像进展及展望[J]. 中国科学：生命科学, 2020, 50(11):1268-1284.
	ZHANG Yangsong, ZHUO Yan, YAO Dezhong. Progresses and prospects of brain electromagnetic imaging[J]. Scientia Sinica Vitae, 2020, 50(11):1268-1284.
[13]	NICOLAS-ALONSO L F, GOMEZ-GIL J. Brain computer interfaces, a review[J]. Sensors (Basel, Switzerland), 2012, 12(2):1211-1279.
[14]	董卫华, 廖华, 詹智成, 等. 2008年以来地图学眼动与视觉认知研究新进展[J]. 地理学报, 2019, 74(3):599-614.
	DONG Weihua, LIAO Hua, ZHAN Zhicheng, et al. New research progress of eye tracking-based map cognition in cartography since 2008[J]. Acta Geographica Sinica, 2019, 74(3):599-614.
[15]	STOJIC F, CHAU T. Nonspecific visuospatial imagery as a novel mental task for online EEG-based BCI control[J]. International Journal of Neural Systems, 2020, 30(6):2050026.
[16]	SOUZA R H C E, NAVES E L M. Attention detection in virtual environments using EEG signals: a scoping review[J]. Frontiers in Physiology, 2021, 12:727840.
[17]	KOHLI V, TRIPATHI U, CHAMOLA V, et al. A review on virtual reality and augmented reality use-cases of brain computer interface based applications for smart cities[J]. Microprocessors and Microsystems, 2022, 88:104392.
[18]	WUNDERLICH A, GRAMANN K. Eye movement-related brain potentials during assisted navigation in real-world environments[J]. The European Journal of Neuroscience, 2021, 54(12):8336-8354.
[19]	LIN C T, KING J T, SINGH A K, et al. Voice navigation effects on real-world lane change driving analysis using an electroencephalogram[J]. IEEE Access, 2018, 6:26483-26492.
[20]	YU Yang, LIU Yadong, JIANG Jun, et al. An asynchronous control paradigm based on sequential motor imagery and its application in wheelchair navigation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(12):2367-2375.
[21]	BOASHASH B, KHAN N A, BEN-JABEUR T. Time-frequency features for pattern recognition using high-resolution TFDs: a tutorial review[J]. Digital Signal Processing, 2015, 40:1-30.
[22]	王韬, 柯余峰, 王宁慈, 等. 空间滤波方法在脑-机接口中的应用及研究进展[J]. 中国生物医学工程学报, 2019, 38(5):599-608.
	WANG Tao, KE Yufeng, WANG Ningci, et al. Application and research development of spatial filtering method in brain-computer interfaces[J]. Chinese Journal of Biomedical Engineering, 2019, 38(5):599-608.
[23]	HE Bin, ASTOLFI L, VALDES-SOSA P A, et al. Electrophysiological brain connectivity: theory and implementation[J]. IEEE Transactions on Bio-Medical Engineering, 2019:2913928.
[24]	GRAMFORT A, LUESSI M, LARSON E, et al. MEG and EEG data analysis with MNE-Python[J]. Frontiers in Neuroscience, 2013, 7:267.
[25]	DHARMAPRANI D, NGUYEN H K, LEWIS T W, et al. A comparison of independent component analysis algorithms and measures to discriminate between EEG and artifact components[C]//Proceedings of 2016 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Orlando: IEEE, 2016: 825-828.
[26]	CAMPOS VIOLA F, THORNE J, EDMONDS B, et al. Semi-automatic identification of independent components representing EEG artifact[J]. Clinical Neurophysiology, 2009, 120(5):868-877.
[27]	DAMMERS J, SCHIEK M, BOERS F, et al. Integration of amplitude and phase statistics for complete artifact removal in independent components of neuromagnetic recordings[J]. IEEE Transactions on Biomedical Engineering, 2008, 55(10):2353-2362.
[28]	罗志增, 鲁先举, 周莹. 基于脑功能连接网络和样本熵的脑电信号特征提取[J]. 电子与信息学报, 2021, 43(2):412-418.
	LUO Zhizeng, LU Xianju, ZHOU Ying. EEG feature extraction based on brain function network and sample entropy[J]. Journal of Electronics & Information Technology, 2021, 43(2):412-418.
[29]	GROSSE-WENTRUP M, BUSS M. Multiclass common spatial patterns and information theoretic feature extraction[J]. IEEE Transactions on Biomedical Engineering, 2008, 55(8):1991-2000.
[30]	LI Xuan, WU Yunqiao, WEI Mengting, et al. A novel index of functional connectivity: phase lag based on Wilcoxon signed rank test[J]. Cognitive Neurodynamics, 2021, 15(4):621-636.
[31]	STAM C J, NOLTE G, DAFFERTSHOFER A. Phase lag index: assessment of functional connectivity from multi channel EEG and MEG with diminished bias from common sources[J]. Human Brain Mapping, 2007, 28(11):1178-1193.
[32]	VINCK M, OOSTENVELD R, VAN WINGERDEN M, et al. An improved index of phase-synchronization for electrophysiological data in the presence of volume-conduction, noise and sample-size bias[J]. NeuroImage, 2011, 55(4):1548-1565.
[33]	HAMILTON WL, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York: Curran Associates Inc, 2017: 1025-1035.
[34]	DIEHL F, BRUNNER T, LE TM, KNOLL A. Towards graph pooling by edge contraction[C]//Proceedings of 2019 ICML Workshop on Learning and Reasoning with Graph-structured Data. New York: ACM Press, 2019.
[35]	SU Zidong, HU Zehui, LI Yangding. Hierarchical graph representation learning with local capsule pooling[C]//Proceedings of 2021 ACM Multimedia Asia. Gold Coast: ACM Press, 2021: 1-7.
[36]	LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5):056013.

样本规模的组别	L/s	L_T/s	N_T	N_S
1	1	1	1	1920
2	3	1	5	640
3	3	2	2	640
4	3	3	1	640
5	6	1	11	320
6	6	2	5	320
7	6	4	2	320
8	6	6	1	320

模型	右转		左转		直行		停止
模型	精确率	召回率	精确率	召回率	精确率	召回率	精确率	召回率
EEGNet	50.6±10.2	49.3±12.8	49.4±12.3	49.7±10.4	53.7±9.3	54.1±12.3	54.5±11.8	55.1±11.7
GCN	50.4±9.1	48.6±11.6	47.4±14.1	48.4±9.9	55.3±10.4	55.9±13.6	55.4±14.4	56.1±13.5
STFCN_CSP	49.1±7.5	45.2±7.8	46.7±9.6	46.9±11.5	54.2±9.8	55.8±12.0	54.1±10.7	56.1±10.5
STFCN_Freq	35.9±11.6	35.1±9.7	36.8±9.0	35.6±9.9	37.4±10.2	38.4±10.3	37.6±9.4	36.3±8.6
MLP	46.6±10.7	45.2±10.4	46.7±8.6	44.8±10.4	49.2±9.7	48.3±10.7	50.2±10.8	49.6±10.7
本文模型	50.2±6.4	47.5±7.5	48.6±9.2	48.9±10.3	55.6±8.6	56.7±11.9	55.5±11.3	56.7±11.4