抗视差的宽基线弱纹理影像自动拼接算法

doi:10.11947/j.AGCS.2023.20220417

摘要/Abstract

摘要： 针对目前带有视差突变的宽基线弱纹理影像拼接效果差及需要人工干预的问题,本文从影像匹配和影像配准两方面进行改进,提出了面向宽基线弱纹理影像的抗视差全自动拼接算法。首先,采用融合了影像视角几何纠正的局部特征变换模型,由粗到精地实现弱纹理特征的准密集对应;然后,基于匹配点和深度神经网络,泛化学习宽基线影像间的可靠透视变换,以获取全局配准视差,局部视差则通过薄板样条函数来精确拟合;接着,将影像拼接结果的多边形边界进行规则化处理,通过全卷积网络将其训练为规则化矩形,在有效剔除空白区域的同时,最大限度地保留影像拼接内容;最后,选取4组无人机和地面近景宽基线弱纹理立体像对进行测试,并将本文算法的影像匹配及配准各阶段结果分别与现有代表性算法结果进行对比。试验结果表明,本文算法在匹配点数目、匹配精度及影像拼接质量等方面具有显著优势,并能够在影像弱纹理区域及视差突变场景表现出较好的稳定性。

关键词: 影像匹配, 视差突变, 宽基线弱纹理影像, 深度神经网络, 自动拼接

Abstract: Based on the available algorithm, it is a tough work to achieve stitching of wide-baseline weak-texture images with parallax discontinuity. As a result, the stitching task usually requires manual intervention. For this, we modify the critical steps of image matching and image registration, and propose an anti-parallax automatic stitching algorithm for wide-baseline weak-texture images. First, we obtain the quasi-dense correspondence of weak-texture features from coarse to fine, based on the local feature transformers model incorporating the geometric correction of the image perspective. Next, based on matching points and deep neural network (DNN), the reliable perspective transform between wide-baseline images can be learned to eliminate global registration disparity, and then the local left disparities are precisely fitted by thin plate spline (TPS) function. Furthermore, the polygon boundary of the image stitching result is regularized, and it is trained as a regularized rectangle through a fully convolutional network, which effectively removes the blank area and preserves the content of the image stitching to the maximum extent. Finally, four groups of UAV and ground close-range wide-baseline stereo image pairs with weak-textures are selected and tested, and the results of image matching and registration stages of our method are respectively compared with the results of the existing representative algorithms. The experimental results verify that our method has significant advantages in the number of matching points, accuracy and stitching quality, and show good stability at the weak-texture and parallax discontinuity regions of the images.

Key words: image matching, disparity discontinuity, wide-baseline weak-texture images, deep neural network, automatic stitching

中图分类号:

P237

姚国标, 黄鹏飞, 龚健雅, 孟飞, 张进. 抗视差的宽基线弱纹理影像自动拼接算法[J]. 测绘学报, 2023, 52(9): 1515-1527.

YAO Guobiao, HUANG Pengfei, GONG Jianya, MENG Fei, ZHANG Jin. The automatic stitching algorithm with anti-parallax for wide-baseline weak-texture images[J]. Acta Geodaetica et Cartographica Sinica, 2023, 52(9): 1515-1527.

参考文献

[1] VIGUIER R, LIN C C, ALIAKBARPOUR H, et al. Automatic video content summarization using geospatial mosaics of aerial imagery[C]// Proceedings of 2015 IEEE International Symposium on Multimedia. Miami: IEEE 2015: 249-253.
[2] GUPTA V, BUSTAMANTE M, FREDRIKSSON A, et al. Improving left ventricular segmentation in four-dimensional flow MRI using intramodality image registration for cardiac blood flow analysis[J]. Magnetic Resonance in Medicine, 2018, 79(1): 554-560.
[3] 杨笛航. 基于多全景相机拼接的虚拟现实和实景交互系统[D]. 杭州: 浙江大学, 2017. YANG Dihang. VR and real interaction system based on multi panoramic camera[D]. Hangzhou: Zhejiang University, 2017.
[4] 程争刚, 张利. 一种基于无人机位姿信息的航拍图像拼接方法[J]. 测绘学报, 2016, 45(6):698-705. DOI: 10.11947/j.AGCS.2016.20150567. CHENG Zhenggang, ZHANG Li. An aerial image mosaic method based on UAV position and attitude information[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(6): 698-705. DOI: 10.11947/j.AGCS.2016.20150567.
[5] 孙商文, 刘宇, 徐昭洪, 等. 基于全局最优变换矩阵的图像拼接方法[J]. 兵器装备工程学报, 2020, 41(11): 207-211. SUN Shangwen, LIU Yu, XU Zhaohong, et al. Image stitching based on global optimal homography matrix[J]. Journal of Ordnance Equipment Engineering, 2020, 41(11): 207-211.
[6] 姚万业, 鲍珣, 刘彤宇. 多相机图像拼接算法的改进[J]. 科学技术与工程, 2019, 19(28): 227-232. YAO Wanye, BAO Xun, LIU Tongyu. The improvement of multi-camera image mosaic algorithm[J]. Science Technology and Engineering, 2019, 19(28): 227-232.
[7] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[8] ROSTEN E, PORTER R, DRUMMOND T. Faster and better:a machine learning approach to corner detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(1): 105-119.
[9] CALONDER M, LEPETIT V, STRECHA C, et al. BRIEF: binary robust independent elementary features[C]// Proceedings of 2020 Computer Vision. Berlin:Springer, 2020: 778-792.
[10] RUBLEE E, RABAUD V, KONOLIGE K,et al. ORB: an efficient alternative to SIFT or SURF[C]// Proceedings of 2011 IEEE International Conference on Computer Vision. Barcelona: IEEE, 2011: 2564-2571.
[11] MOREL J M, YU Guoshen. ASIFT: a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2): 438-469.
[12] XU Guili, WU Quan, CHENG Yuehua, et al. A robust deformed image matching method for multi-source image matching[J]. Infrared Physics & Technology, 2021, 115: 103691.
[13] ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks[C]// Proceedings of 2011 IEEE Conferenceon Computer Vision and Pattern Recognition.Boston:IEEE, 2015:4353-4361.
[14] TIAN Yurun, FAN Bin, WU Fuchao. L2-net: deep learning of discriminative patch descriptor in euclidean space[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu: IEEE, 2017: 661-669.
[15] MISHCHUK A, MISHKIN D, RADENOVIC F, et al. Working hard to know your neighbor's margins: local descriptor learning loss[C]//Proceedings of 2017 Advances in Neural Information Processing Systems. Long Beach:ACM Press,2017: 4826-4837.
[16] MISHKIN D,RADENOVIC' F, MATAS J. Repeatability is not enough: learning affine regions via discriminability[C]//Proceedings of 2018 Computer Vision.Cham:Springer, 2018: 287-304.
[17] CHEN Lin, HEIPKE C. Deep learning feature representation for image matching under large viewpoint and viewing direction change[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 94-112.
[18] LUO Zixin, SHEN Tianwei, ZHOU Lei, et al. ContextDesc: local descriptor augmentation with cross-modality context [C]// Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2527-2536.
[19] DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description[C]// Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City: IEEE, 2018: 224-236.
[20] SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proceedings of 2020 IEEE Conference on Computer Vision and Pattern Recognition.Seattle: IEEE, 2020: 4937-4946.
[21] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach:ACM Press,2017:6000-6010.
[22] SUN Jiaming, SHEN Zehong, WANG Yuang, et al. LoFTR: detector-free local feature matching with transformers[C]//Proceedings of 2021 IEEE Conference on Computer Vision and Pattern Recognition.Nashville:IEEE, 2021: 8918-8927.
[23] 裴红星, 刘金达, 葛佳隆, 等. 图像拼接技术综述[J]. 郑州大学学报(理学版), 2019, 51(4): 1-10, 29. PEI Hongxing, LIU Jinda, GE Jialong, et al. A review on image mosaicing techniques[J]. Journal of Zhengzhou University (Natural Science Edition), 2019, 51(4): 1-10, 29.
[24] BROWN M, LOWE D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision, 2007, 74(1): 59-73.
[25] GAO Junhong, KIM S J, BROWN M S. Constructing image panoramas using dual-homography warping[C]// Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011: 49-56.
[26] ZARAGOZA J, CHIN T J, BROWN M S, et al. As-projective-as-possible image stitching with moving DLT[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE, 2013: 2339-2346.
[27] CHANG Chehan, SATO Y, CHUANG Y Y. Shape-preserving half-projective warps for image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014:3254-3261.
[28] LIN C C, PANKANTI S U,RAMAMURTHY K N, et al. Adaptive as-natural-as-possible image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition.Columbus: IEEE, 2014:3254-3261.
[29] LI Jing, WANG Zhengming, LAI Shiming, et al. Parallax-tolerant image stitching based on robust elastic warping[J]. IEEE Transactions on Multimedia, 2018, 20(7): 1672-1687.
[30] BOOKSTEIN F L. Principal warps: thin-plate splines and the decomposition of deformations[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1989, 11(6): 567-585.
[31] ROHR K, STIEHL H S, SPRENGEL R, et al. Point-based elastic registration of medical image data using approximating thin-plate splines[C]//Proceedings of 1996 International Conference on Visualization in Biomedical Computing. Berlin: Springer, 1996: 297-306.
[32] LI Jing, DENG Baosong, TANG Rongfu, et al. Local-adaptive image alignment based on triangular facet approximation[J]. IEEE Transactions on Image Processing, 2020, 29: 2356-2369.
[33] JIA Qi, LI Zhengjun, FAN Xin, et al. Leveraging line-point consistence to preserve structures for wide parallax image stitching[C]// Proceedings of 2021 IEEE Conference on Computer Vision and Pattern Recognition.Nashville:IEEE, 2021: 12181-12190.
[34] XIU Chunbo, FANG Jingyao, ZHANG Jiang. Image stitching method based on adaptive weighted fusion[C]// Proceedings of the 33rd Chinese Control and Decision Conference.Kunming:IEEE,2021: 3099-3103.
[35] DENG Danjun. Smooth stitching method for the texture seams of remote sensing images based on gradient structure information[J]. Processes, 2021, 9(10): 1689.
[36] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas: IEEE, 2016: 770-778.
[37] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//Proceedings of the 16th European Conference. Glasgow: Springer, 2020: 213-229.
[38] DETONE D, MALISIEWICZ T, RABINOVICH A. Deep image homography estimation[EB/OL]. [2022-07-03]. https://arxiv.org/abs/1706.03762.
[39] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of 2014 Computer Vision.Zurich: Springer,2014: 740-755.
[40] YE Yuanxin, SHAN Jie. A local descriptor based registration method for multispectral remote sensing images with non-linear intensity differences[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 90: 83-95.
[41] NIE Lang, LIN Chunyu, LIAO Kang, et al. Deep rectangling for image stitching: a learning baseline[C]// Proceedings of 2022 IEEE Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022:5730-5738.
[42] KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL].[2022-07-03].https://arxiv.org/abs/1412.6980.
[43] LI Zhengqi, SNAVELY N. MegaDepth:learning single-view depth prediction from Internet photos[C]// Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE, 2018:2041-2050.
[44] 姚国标, 邓喀中, 张力, 等. 融合互补仿射不变特征的倾斜立体影像高精度自动配准方法[J]. 测绘学报, 2013, 42(6): 869-876, 883. YAO Guobiao, DENG Kazhong, ZHANG Li, et al. An automated registration method with high accuracy for oblique stereo images based on complementary affine invariant features[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(6): 869-876, 883.
[45] 罗楠, 孙权森, 耿蕾蕾, 等. 一种扩展SURF描述符及其在遥感图像配准中的应用[J]. 测绘学报, 2013, 42(3): 383-388. LUO Nan, SUN Quansen, GENG Leilei, et al. An extended SURF descriptor and its application in remote sensing images registration[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(3): 383-388.
[46] MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a“completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20(3): 209-212.