交通监控视频图像语义分割及其拼接方法

doi:10.11947/j.AGCS.2020.20190224

测绘学报 ›› 2020, Vol. 49 ›› Issue (4): 522-532.doi: 10.11947/j.AGCS.2020.20190224

交通监控视频图像语义分割及其拼接方法

刘嗣超¹, 武鹏达², 赵占杰², 李成名^1,2

1. 山东科技大学测绘科学与工程学院, 山东青岛 266590;
2. 中国测绘科学研究院, 北京 100830

收稿日期:2019-06-04 修回日期:2019-10-14 发布日期:2020-04-17
通讯作者: 武鹏达 E-mail:wupd@casm.ac.cn
作者简介:刘嗣超(1994-)男,硕士,研究方向为图像处理、深度学习和三维可视化表达。E-mail:liusc_sdust@126.com
基金资助:
国家自然科学基金（41871375）；中国测绘科学研究院基本科研业务费（AR 1909；AR 1916；AR 1917；AR 1935）；国家重点研发计划（2018YFB2100702）

Image semantic segmentation and stitching method of traffic monitoring video

LIU Sichao¹, WU Pengda², ZHAO Zhanjie², LI Chengming^1,2

1. College of Geomatics, Shandong University of Science and Technology, Qingdao 266590, China;
2. Chinese Academy of Surveying and Mapping, Beijing 100830, China

Received:2019-06-04 Revised:2019-10-14 Published:2020-04-17
Supported by:
The National Natural Science Foundation of China (No. 41871375);The Basal Research Fund of CASM (Nos. AR 1909;AR 1916;AR 1917;AR 1935);The National Key Research and Development Project (2018YFB2100702)

摘要/Abstract

摘要： 视频拼接是图像拼接的外延，在场景监控、目标识别等应用中发挥着重要作用。传统视频拼接算法多要求视频间具有较大重叠区域且特征点匹配过程中只顾及图像几何特征，当处理交通监控视频时，会因不同摄像头之间重叠区域极小或主光轴之间夹角较大而导致无法拼接或图像变形较大。为此，本文提出一种交通监控视频图像语义分割及其拼接方法。首先，利用边缘角度二阶差分直方图算法自动识别多视频交汇区域的正射影像，并将其作为拼接背景图像；然后，基于全卷积神经网络对正射影像和视频图像进行语义分割，提取图像中的交通专题语义；最后，以交通专题语义作为约束进行特征点匹配，将各个交通监控视频匹配至背景正射影像，实现监控区域视频拼接。采用山东省某市实际视频数据进行试验验证，结果表明对于重叠区域较小的监控视频，本文方法可获得较好地拼接图像，同时可有效提高特征点匹配的准确度。

关键词: 交通监控视频拼接, SIFT特征匹配, 边缘角度二阶差分直方图, 全卷积神经网络, 语义分割

Abstract: As the extension of image stitching, video stitching plays an important role in scene monitoring, target recognition and so on. Traditional video stitching methods are mostly suitable for the videos with large overlapping regions and only geometric features of images are considered in feature matching. When dealing with traffic monitoring videos, existing methods often leads to stitching failure or large distortion because of the overlap region between different cameras is small and the angle between the main optical axes is large. Hence, an image semantic segmentation and stitching method of traffic monitoring video is proposed in this paper. First, the edge angular second-order difference histogram algorithm is proposed to recognize the orthophoto image automatically in the multi-video intersection area, and the orthophoto image is used as the stitching background image. Second, the orthophoto image and video image are segmented semantically based on fully convolutional network (FCN), and traffic thematic features are extracted separately. Finally, the traffic thematic features are used as constraints for feature point matching, and each traffic monitoring image is matched to the orthophoto image to realize regional video stitching. The experimental of real video data from a city in Shandong Province show that the proposed method obtain better stitching images for monitoring videos with smaller overlap areas, and effectively improve the accuracy of feature point matching.

Key words: traffic monitoring video stitching, SIFT feature matching, edge angular second-order difference histogram, fully convolutional network, semantic segmentation

中图分类号:

P227

刘嗣超, 武鹏达, 赵占杰, 李成名. 交通监控视频图像语义分割及其拼接方法[J]. 测绘学报, 2020, 49(4): 522-532.

LIU Sichao, WU Pengda, ZHAO Zhanjie, LI Chengming. Image semantic segmentation and stitching method of traffic monitoring video[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(4): 522-532.

参考文献

[1] NIE Yongwei, SU Tan, ZHANG Zhensong, et al. Dynamic video stitching via shakiness removing[J]. IEEE Transactions on Image Processing, 2018, 27(1):164-178.
[2] KRISHNAKUMAR K, GANDHI S I. Video stitching using interacting multiple model based feature tracking[J]. Multimedia Tools and Applications, 2019, 78(2):1375-1397.
[3] 杨毅, 王冬生, 宋文杰, 等. 融合图像语义的动态视频拼接方法[J]. 系统工程与电子技术, 2018, 40(12):2845-2854. YANG Yi, WANG Dongsheng, SONG Wenjie, et al. Dynamic video stitching method embracing image semantics[J]. Systems Engineering and Electronics, 2018, 40(12):2845-2854.
[4] MORAVEC H P. Obstacle avoidance and navigation in the real world by a seeing robot rover[D]. Stanford:Stanford University, 1980.
[5] LOWE D G. Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra:IEEE, 1999:1150-1157.
[6] HAGGUI O, TADONKI C, LACASSAGNE L, et al. Harris corner detection on a NUMA manycore[J]. Future Generation Computer Systems, 2018, 88:442-452.
[7] LI Jian, GUO Binxuan, PENG Zhe. Image matching of junction-type feature points based on the third-order moment[J]. Optik, 2015, 126(4):411-416.
[8] KE Yan, SUKTHANKAR R. PCA-SIFT:a more distinctive representation for local image descriptors[C]//Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE, 2004:II-506-II-513.
[9] 杨宇博, 程承旗. 基于分块Harris特征的航拍视频拼接方法[J]. 北京大学学报(自然科学版), 2013, 49(4):657-661. YANG Yubo, CHENG Chengqi. Block Harris based mosaic method for aerial video[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2013, 49(4):657-661.
[10] 刘畅, 金立左, 费树岷, 等. 固定多摄像头的视频拼接技术[J]. 数据采集与处理, 2014, 29(1):126-133. LIU Chang, JIN Lizuo, FEI Shumin, et al. Video stitching technology based on fixed multi-camera[J]. Journal of Data Acquisition and Processing, 2014, 29(1):126-133.
[11] 苗立刚. 视频监控中的图像拼接与合成算法研究[J]. 仪器仪表学报, 2009, 30(4):857-861. MIAO Ligang. Image mosaicing and compositing algorithm for video surveillance[J]. Chinese Journal of Scientific Instrument, 2009, 30(4):857-861.
[12] HSU S, SAAWHNEY H S, KUMAR R. Automated mosaics via topology inference[J]. IEEE Computer Graphics and Applications, 2002, 22(2):44-54.
[13] 杨涛, 张艳宁, 张秀伟, 等. 基于场景复杂度与不变特征的航拍视频实时配准算法[J]. 电子学报, 2010, 38(5):1069-1077. YANG Tao, ZHANG Yanning, ZHANG Xiuwei, et al. Scene complexity and invariant feature based real-time aerial video registration algorithm[J]. Acta Electronica Sinica, 2010, 38(5):1069-1077.
[14] ACHARYA T, RAY A K. 数字图像处理:原理与应用[M]. 田浩, 葛秀慧, 王顶, 译.北京:清华大学出版社, 2007. ACHARYA T, RAY A K. Principle and application of digital:image processing[M]. TIAN Hao, GE Xiuhui, WANG Ding, trans. Beijing:Tsinghua University Press, 2007.
[15] SALMAN N. Image segmentation based on watershed and edge detection techniques[J]. The International Arab Journal of Information Technology, 2006, 3(2):104-110.
[16] 刘嗣超, 李成名, 孙林. 基于边缘角度二阶差分直方图的城市遥感图像检索方法[J]. 山东科技大学学报(自然科学版), 2019, 38(2):36-43. LIU Sichao, LI Chengming, SUN Lin. Remote sensing image retrieval based on histogram of edge angular second-order difference[J]. Journal of Shandong University of Science and Technology (Natural Science), 2019, 38(2):36-43.
[17] 包倩, 郭平. 基于直方图的遥感图像相似性检索方法比较[J]. 遥感学报, 2006, 10(6):893-900. BAO Qian, GUO Ping. Comparative studies on similarity measures for remote sensing image retrieval based on histogram[J]. Journal of Remote Sensing, 2006, 10(6):893-900.
[18] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017, 39(4):640-651.
[19] 戴玉超, 张静, PORIKLI F, 等. 深度残差网络的多光谱遥感图像显著目标检测[J]. 测绘学报, 2018, 47(6):873-881. DAI Yuchao, ZHANG Jing, PORIKLI F, et al. Salient object detection from multi-spectral remote sensing images with deep residual network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):873-881.
[20] KRUTHIVENTI S S S, AYUSH K, BABU R V. Deepfix:a fully convolutional neural network for predicting human eye fixations[J]. IEEE Transactions on Image Processing, 2017, 26(9):4446-4456.
[21] NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE,2015:1520-1528.
[22] ZHOU Hao, ZHANG Jun, LEI Jun, et al. Image semantic segmentation based on FCN-CRF model[C]//Proceedings of 2016 International Conference on Image, Vision and Computing (ICIVC). Portsmouth:IEEE, 2016:9-14.
[23] PALENICHKA R M, LAKHSSASSI A, ZAREMBA M B. Outlier-resistant dissimilarity measure for feature-based image matching[C]//Proceedings of the 20th International Conference on Pattern Recognition. Istanbul:IEEE, 2010:846-849.
[24] GRUNDLAND M, DODGSON N A. Color histogram specification by histogram warping[C]//Color Imaging X:Processing, Hardcopy, and Applications. San Jose:SPIE, 2005:610-621.
[25] XIAO Bin, TANG Han, JIANG Yanjun, et al. Brightness and contrast controllable image enhancement based on histogram specification[J]. Neurocomputing, 2018, 275:2798-2809.

交通监控视频图像语义分割及其拼接方法

Image semantic segmentation and stitching method of traffic monitoring video

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 11

编辑推荐

Metrics

本文评价

[1]	蒋腾平, 王永君, 张林淇, 梁冲, 孙剑. 融合CNN和MRF的激光点云层次化语义分割方法[J]. 测绘学报, 2021, 50(2): 215-225.
[2]	邵晓航, 吴杭彬, 刘春, 陈晨, 蔡天池, 程帆瑾. 语素关联约束的动态环境视觉定位优化[J]. 测绘学报, 2021, 50(11): 1478-1486.
[3]	李道纪, 郭海涛, 卢俊, 赵传, 林雨准, 余东行. 遥感影像地物分类多注意力融和U型网络法[J]. 测绘学报, 2020, 49(8): 1051-1064.
[4]	耿艳磊, 陶超, 沈靖, 邹峥嵘. 高分辨率遥感影像语义分割的半监督全卷积网络法[J]. 测绘学报, 2020, 49(4): 499-508.
[5]	王忠武, 王志盼, 尤淑撑, 雷帆, 曹里, 杨凯钧. Landsat影像冰川提取的上下文感知语义分割网络法[J]. 测绘学报, 2020, 49(12): 1575-1582.
[6]	徐旺, 游雄, 张威巍, 邓晨. 2D地图的建筑物场景结构提取方法及其在城市增强现实中的应用[J]. 测绘学报, 2020, 49(12): 1619-1629.
[7]	李雪, 张力, 王庆栋, 艾海滨. 多时相遥感影像语义分割色彩一致性对抗网络[J]. 测绘学报, 2020, 49(11): 1473-1484.
[8]	左宗成, 张文, 张东映. 融合可变形卷积与条件随机场的遥感影像语义分割方法[J]. 测绘学报, 2019, 48(6): 718-726.
[9]	季顺平, 魏世清. 遥感影像建筑物提取的卷积神经元网络与开源数据集方法[J]. 测绘学报, 2019, 48(4): 448-459.
[10]	贺浩, 王仕成, 杨东方, 王舒洋, 刘星. 基于Encoder-Decoder网络的遥感影像道路提取方法[J]. 测绘学报, 2019, 48(3): 330-338.
[11]	胡翔云, 巩晓雅, 张觅. 变分法遥感影像人工地物自动检测[J]. 测绘学报, 2018, 47(6): 780-789.