Image semantic segmentation and stitching method of traffic monitoring video

doi:10.11947/j.AGCS.2020.20190224

Abstract

Abstract: As the extension of image stitching, video stitching plays an important role in scene monitoring, target recognition and so on. Traditional video stitching methods are mostly suitable for the videos with large overlapping regions and only geometric features of images are considered in feature matching. When dealing with traffic monitoring videos, existing methods often leads to stitching failure or large distortion because of the overlap region between different cameras is small and the angle between the main optical axes is large. Hence, an image semantic segmentation and stitching method of traffic monitoring video is proposed in this paper. First, the edge angular second-order difference histogram algorithm is proposed to recognize the orthophoto image automatically in the multi-video intersection area, and the orthophoto image is used as the stitching background image. Second, the orthophoto image and video image are segmented semantically based on fully convolutional network (FCN), and traffic thematic features are extracted separately. Finally, the traffic thematic features are used as constraints for feature point matching, and each traffic monitoring image is matched to the orthophoto image to realize regional video stitching. The experimental of real video data from a city in Shandong Province show that the proposed method obtain better stitching images for monitoring videos with smaller overlap areas, and effectively improve the accuracy of feature point matching.

Key words: traffic monitoring video stitching, SIFT feature matching, edge angular second-order difference histogram, fully convolutional network, semantic segmentation

CLC Number:

P227

LIU Sichao, WU Pengda, ZHAO Zhanjie, LI Chengming. Image semantic segmentation and stitching method of traffic monitoring video[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(4): 522-532.

References

[1] NIE Yongwei, SU Tan, ZHANG Zhensong, et al. Dynamic video stitching via shakiness removing[J]. IEEE Transactions on Image Processing, 2018, 27(1):164-178.
[2] KRISHNAKUMAR K, GANDHI S I. Video stitching using interacting multiple model based feature tracking[J]. Multimedia Tools and Applications, 2019, 78(2):1375-1397.
[3] 杨毅, 王冬生, 宋文杰, 等. 融合图像语义的动态视频拼接方法[J]. 系统工程与电子技术, 2018, 40(12):2845-2854. YANG Yi, WANG Dongsheng, SONG Wenjie, et al. Dynamic video stitching method embracing image semantics[J]. Systems Engineering and Electronics, 2018, 40(12):2845-2854.
[4] MORAVEC H P. Obstacle avoidance and navigation in the real world by a seeing robot rover[D]. Stanford:Stanford University, 1980.
[5] LOWE D G. Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra:IEEE, 1999:1150-1157.
[6] HAGGUI O, TADONKI C, LACASSAGNE L, et al. Harris corner detection on a NUMA manycore[J]. Future Generation Computer Systems, 2018, 88:442-452.
[7] LI Jian, GUO Binxuan, PENG Zhe. Image matching of junction-type feature points based on the third-order moment[J]. Optik, 2015, 126(4):411-416.
[8] KE Yan, SUKTHANKAR R. PCA-SIFT:a more distinctive representation for local image descriptors[C]//Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE, 2004:II-506-II-513.
[9] 杨宇博, 程承旗. 基于分块Harris特征的航拍视频拼接方法[J]. 北京大学学报(自然科学版), 2013, 49(4):657-661. YANG Yubo, CHENG Chengqi. Block Harris based mosaic method for aerial video[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2013, 49(4):657-661.
[10] 刘畅, 金立左, 费树岷, 等. 固定多摄像头的视频拼接技术[J]. 数据采集与处理, 2014, 29(1):126-133. LIU Chang, JIN Lizuo, FEI Shumin, et al. Video stitching technology based on fixed multi-camera[J]. Journal of Data Acquisition and Processing, 2014, 29(1):126-133.
[11] 苗立刚. 视频监控中的图像拼接与合成算法研究[J]. 仪器仪表学报, 2009, 30(4):857-861. MIAO Ligang. Image mosaicing and compositing algorithm for video surveillance[J]. Chinese Journal of Scientific Instrument, 2009, 30(4):857-861.
[12] HSU S, SAAWHNEY H S, KUMAR R. Automated mosaics via topology inference[J]. IEEE Computer Graphics and Applications, 2002, 22(2):44-54.
[13] 杨涛, 张艳宁, 张秀伟, 等. 基于场景复杂度与不变特征的航拍视频实时配准算法[J]. 电子学报, 2010, 38(5):1069-1077. YANG Tao, ZHANG Yanning, ZHANG Xiuwei, et al. Scene complexity and invariant feature based real-time aerial video registration algorithm[J]. Acta Electronica Sinica, 2010, 38(5):1069-1077.
[14] ACHARYA T, RAY A K. 数字图像处理:原理与应用[M]. 田浩, 葛秀慧, 王顶, 译.北京:清华大学出版社, 2007. ACHARYA T, RAY A K. Principle and application of digital:image processing[M]. TIAN Hao, GE Xiuhui, WANG Ding, trans. Beijing:Tsinghua University Press, 2007.
[15] SALMAN N. Image segmentation based on watershed and edge detection techniques[J]. The International Arab Journal of Information Technology, 2006, 3(2):104-110.
[16] 刘嗣超, 李成名, 孙林. 基于边缘角度二阶差分直方图的城市遥感图像检索方法[J]. 山东科技大学学报(自然科学版), 2019, 38(2):36-43. LIU Sichao, LI Chengming, SUN Lin. Remote sensing image retrieval based on histogram of edge angular second-order difference[J]. Journal of Shandong University of Science and Technology (Natural Science), 2019, 38(2):36-43.
[17] 包倩, 郭平. 基于直方图的遥感图像相似性检索方法比较[J]. 遥感学报, 2006, 10(6):893-900. BAO Qian, GUO Ping. Comparative studies on similarity measures for remote sensing image retrieval based on histogram[J]. Journal of Remote Sensing, 2006, 10(6):893-900.
[18] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017, 39(4):640-651.
[19] 戴玉超, 张静, PORIKLI F, 等. 深度残差网络的多光谱遥感图像显著目标检测[J]. 测绘学报, 2018, 47(6):873-881. DAI Yuchao, ZHANG Jing, PORIKLI F, et al. Salient object detection from multi-spectral remote sensing images with deep residual network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):873-881.
[20] KRUTHIVENTI S S S, AYUSH K, BABU R V. Deepfix:a fully convolutional neural network for predicting human eye fixations[J]. IEEE Transactions on Image Processing, 2017, 26(9):4446-4456.
[21] NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE,2015:1520-1528.
[22] ZHOU Hao, ZHANG Jun, LEI Jun, et al. Image semantic segmentation based on FCN-CRF model[C]//Proceedings of 2016 International Conference on Image, Vision and Computing (ICIVC). Portsmouth:IEEE, 2016:9-14.
[23] PALENICHKA R M, LAKHSSASSI A, ZAREMBA M B. Outlier-resistant dissimilarity measure for feature-based image matching[C]//Proceedings of the 20th International Conference on Pattern Recognition. Istanbul:IEEE, 2010:846-849.
[24] GRUNDLAND M, DODGSON N A. Color histogram specification by histogram warping[C]//Color Imaging X:Processing, Hardcopy, and Applications. San Jose:SPIE, 2005:610-621.
[25] XIAO Bin, TANG Han, JIANG Yanjun, et al. Brightness and contrast controllable image enhancement based on histogram specification[J]. Neurocomputing, 2018, 275:2798-2809.