Research developments and prospects on dense image matching in photogrammetry

doi:10.11947/j.AGCS.2019.20190453

Abstract

Abstract: The general workflow for dense matching of aerial images is given in this paper. Dense matching is divided into two categories, namely, those that utilize local matching algorithms and global matching algorithms, respectively. The key technologies of the two methods are analyzed in details. Concerns in theory, technology, universality and practicability are proposed. We hope it will be helpful to the related research on dense matching.

Key words: aerial image, dense image matching, local matching algorithm, global matching algorithm, optical flow field-based method, deep learning-based method

CLC Number:

P237

YUAN Xiuxiao, YUAN Wei, XU Shu, JI Yanhua. Research developments and prospects on dense image matching in photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12): 1542-1550.

References

[1] 闫利, 费亮, 陈长海, 等. 利用网络图进行高分辨率航空多视影像密集匹配[J]. 测绘学报, 2016, 45(10):1171-1181. DOI:10.11947/j.AGCS.2016.20160068. YAN Li, FEI Liang, CHEN Changhai, et al. A multi-view dense matching algorithm of high-resolution aerial images based on graph network[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(10):1171-1181. DOI:10.11947/j.AGCS.2016.20160068.
[2] 朱庆, 陈崇泰, 胡翰, 等. 顾及纹理特征的航空影像自适应密集匹配方法[J]. 测绘学报, 2017, 46(1):62-72. DOI:10.11947/j.AGCS.2017.20150608. ZHU Qing, CHEN Chongtai, HU Han, et al. An adaptive dense matching method for airborne images using texture information[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(1):62-72. DOI:10.11947/j.AGCS.2017.20150608.
[3] YUAN Xiuxiao, CHEN Shiyu, YUAN Wei, et al. Poor textural image tie point matching via graph theory[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 129:21-31.
[4] TORRESANI L, KOLMOGOROV V, ROTHER C. A dual decomposition approach to feature correspondence[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2):259-271.
[5] 杨化超, 姚国标, 王永波. 基于SIFT的宽基线立体影像密集匹配[J]. 测绘学报, 2011, 40(5):537-543. YANG Huachao, YAO Guobiao, WANG Yongbo. Dense matching for wide base-line stereo images based on SIFT[J]. Acta Geodaetica et Cartographica Sinica, 2011, 40(5):537-543.
[6] SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]//Proceedings of 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York, NY:IEEE, 2006.
[7] FURUKAWA Y, PONCE J. Accurate, dense, and robust multi-view stereopsis[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, Minnesota:IEEE, 2007.
[8] KE Yan, SUKTHANKAR R. PCA-SIFT:a more distinctive representation for local image descriptors[C]//Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2004). Washington DC:IEEE, 2004.
[9] SCHARSTEIN D, SZELISKI R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47(1-3):7-42.
[10] ZABIH R, WOODFILL J. Non-parametric local transforms for computing visual correspondence[C]//Proceedings of European Conference on Computer Vision-ECCV'94(ECCV 1994). Berlin, Heidelberg:Springer, 1994.
[11] 王之卓. 摄影测量原理[M]. 北京:测绘出版社, 1990. WANG Zhizhuo. Principle of photogrammetry[M]. Beijing:Publishing House of Surveying and Mapping, 1990.
[12] TOLA E, LEPETIT V, FUA P. A fast local descriptor for dense matching[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, Alaska:IEEE, 2008.
[13] HOSNI A, BLEYER M, GELAUTZ M. Secrets of adaptive support weight techniques for local stereo matching[J]. Computer Vision and Image Understanding, 2013, 117(6):620-632.
[14] ZEGLAZI O, RZIZA M, AMINE A, et al. A hierarchical stereo matching algorithm based on adaptive support region aggregation method[J]. Pattern Recognition Letters, 2018, 112:205-211.
[15] WANG Jialiang, ZICKLER T. Local detection of stereo occlusion boundaries[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019). Long Beach, California:IEEE, 2019.
[16] KANADE T, OKUTOMI M. A stereo matching algorithm with an adaptive window:theory and experiment[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2002). Sacramento, CA:IEEE, 1991.
[17] ZHANG Ke, LU Jiangbo, LAFRUIT G. Cross-based local stereo matching using orthogonal integral images[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(7):1073-1079.
[18] YOON K J, KWEON I S. Adaptive support-weight approach for correspondence search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(4):650-656.
[19] YANG Qingxiong, TAN K H, AHUJA N. Real-time O(1) bilateral filtering[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, Florida:IEEE, 2009.
[20] WANG Liang, LIAO Miao, GONG Minglun, et al. High-quality real-time stereo using adaptive cost aggregation and dynamic programming[C]//Proceedings of the 3rd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06). Chapel Hill, NC:IEEE, 2006.
[21] HOSNI A, BLEYER M, GELAUTZ M, et al. Local stereo matching using geodesic support weights[C]//Proceedings of the 16th IEEE International Conference on Image Processing (ICIP). Cairo, Egypt:IEEE, 2009.
[22] HE Kaiming, SUN Jian, TANG Xiaoou. Guided image filtering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6):1397-1409.
[23] RHEMANN C, HOSNI A, BLEYER M, et al. Fast cost-volume filtering for visual correspondence and beyond[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI:IEEE, 2011.
[24] LUKEŽIC A, VOJIR T, ZAJC L C, et al. Discriminative correlation filter with channel and spatial Reliability[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, Hawaii:IEEE, 2017.
[25] GERRITS M, BEKAERT P. Local stereo matching with segmentation-based outlier rejection[C]//Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV'06). Quebec, Canada:IEEE, 2006.
[26] COMANICIU D, MEER P. Mean shift:a robust approach toward feature space analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(5):603-619.
[27] TOMBARI F, MATTOCCIA S, STEFANO L D. Segmentation-based adaptive support for accurate stereo correspondence[C]//Proceedings of the Pacific Rim Conference on Advances in Image and Video Technology. Berlin, Heidelberg:Springer, 2007.
[28] ACHANTA R, SHAJI A, SMITH K, et al. SLIC superpixels compared to state-of-the-art superpixel methods[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11):2274-2282.
[29] BLEYER M, ROTHER C, KOHLI P, et al. Object stereo-joint stereo matching and object segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI:IEEE, 2011.
[30] YAMAGUCHI K, MCALLESTER D, URTASUN R. Efficient joint segmentation, occlusion labeling, stereo and flow estimation[C]//Proceedings of the European Conference on Computer Vision (ECCV 2014). Cham:Springer, 2014.
[31] CHEN Wang, HOU Jiaquan, ZHANG Maojun, et al. Semantic stereo:integrating piecewise planar stereo with segmentation and classification[C]//Proceedings of the 4th IEEE International Conference on Information Science and Technology. Shenzhen, China:IEEE, 2014:200-204.
[32] LI Lincheng, ZHANG Shunli, XIN Yu, et al. PMSC:PatchMatch-based superpixel cut for accurate stereo matching[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(3):679-692.
[33] YANG Fan, LI Xin, CHENG Hong, et al. Object-aware dense semantic correspondence[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, Hawaii:IEEE, 2017.
[34] COLLINS R T. A space-sweep approach to true multi-image matching[C]//Proceedings of CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA:IEEE, 1996.
[35] GALLUP D, FRAHM J M, MORDOHAI P, et al. Real-time plane-sweeping stereo with multiple sweeping directions[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, Minnesota:IEEE, 2007.
[36] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch:a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics-TOG, 2009, 28(3):24-33.
[37] BLEYER M, RHEMANN C, ROTHER C. PatchMatch stereo-stereo matching with slanted support windows[C]//Proceedings of the British Machine Vision Conference. Dundee, Scotland:BMVA Press, 2011.
[38] HIRSCHMVLLER H, SCHARSTEIN D. Evaluation of stereo matching costs on images with radiometric differences[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(9):1582-1599.
[39] HIRSCHMVLLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2):328-341.
[40] BROX T, MALIK J. Large displacement optical flow:descriptor matching in variational motion estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(3):500-513.
[41] LUO Wenjie, SCHWING A G, URTASUN R. Efficient deep learning for stereo matching[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas:IEEE, 2016:5695-5703.
[42] ŽBONTAR J, LECUN Y. Stereo matching by training a convolutional neural network to compare image patches[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015). Boston, USA:IEEE, 2015.
[43] 刘生礼, 唐敏, 董金祥. 遗传模拟退火算法在约束求解中的应用[J]. 中国图像图形学报, 2003, 8(8):938-945. LIU Shengli, TANG Min, DONG Jinxiang. Geometric constraint satisfaction using genetic simulated annealing algorithm[J]. Journal of Image and Graphics, 2003, 8(8):938-945.
[44] BIRCHFIELD S, TOMASI C. Depth discontinuities by pixel-to-pixel stereo[J]. International Journal of Computer Vision, 1999, 35(3):269-293.
[45] TRAN S, DAVIS L S. 3D surface reconstruction using graph cuts with surface constraints[C]//Proceedings of the European Conference on Computer Vision (ECCV 2006). Graz, Austria:Springer, 2006.
[46] SINHA S N, MORDOHAI P, POLLEFEYS M. Multi-view stereo via graph cuts on the dual of an adaptive tetrahedral mesh[C]//Proceedings of the 11th IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil:IEEE, 2007.
[47] 饶旻骅. 基于置信度传播的立体匹配算法研究[D]. 武汉:华中科技大学, 2010. RAO Minhua. The study of stereo matching using belief propagation[D]. Wuhan:Huazhong University of Science and Technology, 2010.
[48] SUN Jian, ZHENG Nanning, SHUM H Y. Stereo matching using belief propagation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(7):787-800.
[49] ROTHERMEL M, WENZEL K, FRITSCH D, et al. SURE:photogrammetric surface reconstruction from imagery[C]//Proceedings of LC3D Workshop. Berlin, Germany:[s.n.], 2012.
[50] HABBECKE M, KOBBELT L. Iterative multi-view plane fitting[C]//Proceedings of the 11th Fall Workshop Vision, Modeling and Visualization. Aachen, Germany:[s.n.], 2006:73-80.
[51] SHAN Qi, CURLESS B, FURUKAWA Y, et al. Occluding contours for multi-view stereo[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014.
[52] SCHÖNBERGER J L, ZHENG Enliang, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Proceedings of European Conference on Computer Vision (ECCV 2016). Cham:Springer, 2016:501-518.
[53] AI Mingyao, HU Qingwu, LI Jiayuan, et al. A robust photogrammetric processing method of low-altitude UAV images[J]. Remote Sensing, 2015, 7(3):2302-2333.
[54] BALTSAVIAS E P. Digital ortho-images-a powerful tool for the extraction of spatial- and geo-information[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 1996, 51(2):63-77.
[55] SHAO Zhenfeng, YANG Nan, XIAO Xiongwu, et al. A multi-view dense point cloud generation algorithm based on low-altitude remote sensing images[J]. Remote Sensing, 2016, 8(5):381.
[56] YUAN Wei, CHEN Shiyu, ZHANG Yong, et al. An aerial-image dense matching approach based on optical flow field[J]. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2016, XLI-B3:543-548.
[57] YUAN Wei, YUAN Xiuxiao, XU Shu, et al. Dense image-matching via optical flow field estimation and fast-guided filter refinement[J]. Remote Sensing, 2019, 11(20):2410.
[58] ZHANG Feihu, WAH B W. Fundamental principles on learning new features for effective dense matching[J]. IEEE Transactions on Image Processing, 2018, 27(2):822-836.
[59] ZHANG Feihu, PRISACARIU V, YANG Ruigang, et al. GA-Net:guided aggregation net for end-to-end stereo matching[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019). Long Beach, CA, USA:IEEE, 2019.
[60] CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Sault Lake City:IEEE, 2018.
[61] MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV:IEEE, 2016.
[62] KIM K R, KIM C S. Adaptive smoothness constraints for efficient stereo matching using texture and edge information[C]//Proceedings of 2016 IEEE International Conference on Image Processing (ICIP). Phoenix, Arizona:IEEE, 2016.