Progress and future of image matching in low-altitude photogrammetry

doi:10.11947/j.AGCS.2019.20190466

Abstract

Abstract: Image matching is the process of obtaining corresponding points between two or more overlapping images by a specific algorithm. It is the critical step in the low-altitude photogrammetric data processing. The quality and efficiency of matching directly affect the subsequent data processing and the quality of mapping product generation. Therefore, image matching is one of the hot topics in the field of low-altitude photogrammetry and many relevant algorithms have been proposed. In this paper, the research status and prospect of image matching in low-altitude photogrammetry are described systematically. Firstly, the categories of image matching are summarized and can be generally divided into gray- and feature-based matching. We focus on feature-based image matching, e.g., point, line, and region-based features extraction and the relevant descriptors and similarity measures are described in detail. Besides, the latest image matching algorithms based on deep learning are listed, and the image matching methods involved in data fusion of various sensors on low-altitude platforms are mentioned.

Key words: image matching, low-altitude photogrammetry, feature extraction, deep learning

CLC Number:

P237

CHEN Xiaoyong, HE Haiqing, ZHOU Junchao, AN Puyang, CHEN Ting. Progress and future of image matching in low-altitude photogrammetry[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12): 1595-1603.

References

[1] 李德仁, 李明. 无人机遥感系统的研究进展与应用前景[J]. 武汉大学学报(信息科学版), 2014, 39(5):505-513, 540. LI Deren, LI Ming. Research advance and application prospect of unmanned aerial vehicle remote sensing system[J]. Geomatics and Information Science of Wuhan University, 2014, 39(5):505-513, 540.
[2] 何海清. 低空摄影测量航带重构及数据处理方法研究[D]. 武汉:武汉大学, 2013. HE Haiqing. The study of strips reconstruction oriented data processing methodology in low-altitude photogrammetry[D]. Wuhan:Wuhan University, 2013.
[3] MASRY S E. An automatic method for height profile determination[J]. The Photogrammetric Record, 1973, 7(42):728-730.
[4] ACKERMANN F. High precision digital image correlation[C]//Proceedings of the 39th Photogrammetric Week. Stuttgart:University of Stuttgart, 1983.
[5] 张祖勋, 张剑清. 数字摄影测量学[M]. 2版. 武汉:武汉大学出版社, 2012. ZHANG Zuxun, ZHANG Jianqing. Digital photogrammetry[M]. 2nd ed. Wuhan:Wuhan University Press, 2012.
[6] MORAVEC H P. Obstacle avoidance and navigation in the real world by a seeing robot rover[R]. Pittsburgh, Pennsylvania:Carnegie Mellon University, 1980.
[7] SMITH S M, BRADY J M. SUSAN:a new approach to low level image processing[J]. International Journal of Computer Vision, 1997, 23(1):45-78.
[8] HARRIS C, STEPHENS M. A combined comer and edge detector[C]//Proceedings of the 4th Alvey Vision Conference. Manchester, UK:[s.n.], 1988:147-152.
[9] TOMASI C, KANADE T. Detection and tracking of point features[R]. Pittsburgh:Carnegie Mellon University, 1991.
[10] MIKOLAJCZYK K, SCHMID C. Scale & affine invariant interest point detectors[J]. International Journal of Computer Vision, 2004, 60(1):63-86.
[11] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[12] WITKIN A. Scale-space filtering:a new approach to multi-scale description[C]//Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing. San Diego, CA:IEEE, 1984:150-153.
[13] ROSTEN E, DRUMMOND T. Machine learning for high-speed corner detection[C]//Proceedings of the 9th European Conference on Computer Vision. Graz, Austria:Springer, 2006:430-443.
[14] MAIR E, HAGER G D, BURSCHKA D, et al. Adaptive and generic corner detection based on the accelerated segment test[C]//Proceedings of the European Conference on Computer Vision. Heraklion, Crete, Greece:Springer, 2010:183-196.
[15] HARALICK R M. Ridges and valleys on digital images[J]. Computer Vision, Graphics, and Image Processing, 1983, 22(1):28-38.
[16] SOBEL I, FELDMAN G. A 3×3 isotropic gradient operator for image processing[R]. California:Stanford Artificial Intelligence Project, 1968.
[17] CANNY J. A computational approach to edge detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986, PAMI-8(6):679-698.
[18] ASGHARI M H, JALALI B. Edge detection in digital images using dispersive phase stretch transform[J]. International Journal of Biomedical Imaging, 2015, 2015:687819.
[19] MATAS J, CHUM O, URBAN M, et al. Robust wide-baseline stereo from maximally stable extremal regions[J]. Image and Vision Computing, 2004, 22(10):761-767.
[20] DENG Hongli, ZHANG Wei, MORTENSEN E, et al. Principal curvature-based region detector for object recognition[C]//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN:IEEE, 2007:1-8.
[21] SHI Jianbo, MALIK J. Normalized cuts and image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8):888-905.
[22] BELLAVIA F, COLOMBO C. Rethinking the sGLOH descriptor[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4):931-944.
[23] WAN Xue, LIU Jianguo, YAN Hongshi, et al. Illumination-invariant image matching for autonomous UAV localisation based on optical sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 119:198-213.
[24] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[25] BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF)[J]. Computer Vision and Image Understanding, 2008, 110(3):346-359.
[26] TOLA E, LEPETIT V, FUA P. DAISY:an efficient dense descriptor applied to wide-baseline stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(5):815-830.
[27] CHEN Jian, TIAN Jie, LEE N, et al. A partial intensity invariant feature descriptor for multimodal retinal image registration[J]. IEEE Transactions on Biomedical Engineering, 2010, 57(7):1707-1718.
[28] ZHANG Shiliang, TIAN Qi, LU Ke, et al. Edge-SIFT:discriminative binary descriptor for scalable partial-duplicate mobile search[J]. IEEE Transactions on Image Processing, 2013, 22(7):2889-2902.
[29] LIAO Kaiyang, LIU Guizhong, HUI Youshi. An improvement to the SIFT descriptor for image representation and matching[J]. Pattern Recognition Letters, 2013, 34(11):1211-1220.
[30] SEDAGHAT A, EBADI H. Remote sensing image matching based on adaptive binning SIFT descriptor[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(10):5283-5293.
[31] FAN Bin, WU Fuchao, HU Zhanyi. Rotationally invariant descriptors using intensity order pooling[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10):2031-2045.
[32] FAN Bin, WU Fuchao, HU Zhanyi. Aggregating gradient distributions into intensity orders:a novel local image descriptor[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO:IEEE, 2011:2377-2384.
[33] WANG Henhua, FAN Bin, WANG Gang, et al. Exploring local and overall ordinal information for robust feature description[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(11):2198-2211.
[34] YANG Yi, DUAN Fajie, MA Ling, et al. A robust method for constructing rotational invariant descriptors[J]. Signal Processing:Image Communication, 2018, 60:224-236.
[35] CALONDER M, LEPETIT V, OZUYSAL M, et al. BRIEF:computing a local binary descriptor very fast[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(7):1281-1298.
[36] ALAHI A, ORTIZ R, VANDERGHEYNST P. FREAK:fast retina keypoint[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2012:510-517.
[37] YANG Xin, CHENG K T T. Local difference binary for ultrafast and distinctive feature description[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(1):188-194.
[38] FAN Bin, KONG Qingqun, TRZCINSKI T, et al. Receptive fields selection for binary feature description[J]. IEEE Transactions on Image Processing, 2014, 23(6):2583-2595.
[39] HEIKKILÄ M, PIETIKÄNEN M, SCHMID C. Description of interest regions with local binary patterns[J]. Pattern Recognition, 2009, 42(3):425-436.
[40] WANG Han, YOON S M, HAN D K, et al. A feature descriptor based on the local patch clustering distribution for illumination-robust image matching[J]. Pattern Recognition Letters, 2017, 94:46-54.
[41] DUO Jingyun, CHEN Pengfeng, ZHAO Long. LCO:a robust and efficient local descriptor for image matching[J]. AEU-International Journal of Electronics and Communications, 2017, 72:234-242.
[42] SIMO-SERRA E, TORRAS C, MORENO-NOGUER F. DaLI:deformation and light invariant descriptor[J]. International Journal of Computer Vision, 2015, 115(2):136-154.
[43] YE Yuanxin, SHAN Jie, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2941-2958.
[44] YE Yuanxin, SHAN Jie, HAO Siyuan, et al. A local phase based invariant feature for remote sensing image matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 142:205-221.
[45] SEDAGHAT A, ALIZADEH NAEINI A. DEM orientation based on local feature correspondence with global DEMs[J]. GIScience & Remote Sensing, 2018, 55(1):110-129.
[46] SEDAGHAT A, EBADI H. Distinctive order based self-similarity descriptor for multi-sensor remote sensing image matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 108:62-71.
[47] CHEN Shuhan, LI Xiaorun, YANG Han, et al. Robust local feature descriptor for multisource remote sensing image registration[J]. Journal of Applied Remote Sensing, 2018, 12(1):015005.
[48] 肖雄武. 具备结构感知功能的倾斜摄影测量场景三维重建[J]. 测绘学报,2019,48(6):802. DOI:10.11947/j.AGCS.2019.20180319. XIAO Xiongwu. Oblique photogrammetry based scene 3D reconstruction with structure sensing functions[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):802. DOI:10.11947/j.AGCS.2019.20180319.
[49] 柯涛. 旋转多基线数字近景摄影测量[D]. 武汉:武汉大学, 2008. KE Tao. Rotating multi-baseline digital close-up photogrammetry[D]. Wuhan:Wuhan University, 2008.
[50] JACOBSEN K. Geometry of vertical and oblique image combinations[C]//Proceedings of the 28th EARSeL Symposium. Istanbul:[s.n.], 2009:16-23.
[51] 许彪. 基于航空影像的真正射影像制作关键技术研究[D]. 武汉:武汉大学, 2012. XU Biao. Research on key technology of true orthophoto generation based on aerial images[D]. Wuhan:Wuhan University, 2012.
[52] 邓宝松, 高宇, 魏迎梅. 图像匹配中具有仿射不变性特征的定量分析[J]. 信号处理, 2008, 24(2):227-230. DENG Baosong, GAO Yu, WEI Yingmei. A quantificational analysis of affine invariant features for image matching[J]. Signal Processing, 2008, 24(2):227-230.
[53] TUYTELAARS T, VAN GOOL L. Wide baseline stereo matching based on local, affinely invariant regions[C]//Proceedings of British Machine Vision Conference. London, UK:BMVC, 2000:412-422.
[54] MIKOLAJCZYK K, SCHMID C. An affine invariant interest point detector[C]//Proceedings of the 7th European Conference on Computer Vision. Copenhagen, Denmark:Springer, 2002:128-142.
[55] MATAS J, CHUM O, URBAN M, et al. Robust wide baseline stereo from maximally stable extremal regions[C]//Proceedings of British Machine Vision Conference. Cardiff, UK:BMVC, 2002:384-393.
[56] MIKOLAJCZYK K, TUYTELAARS T, SCHMID C, et al. A comparison of affine region detectors[J]. International Journal of Computer Vision, 2005, 65(1-2):43-72.
[57] KIMMEL R, ZHANG Cuiping, BRONSTEIN A M. Are MSER features really interesting?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(11):2316-2320.
[58] MOREl J M, YU Guoshen. ASIFT:a new framework for fully affine invariant image comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2):438-469.
[59] 程亮, 龚健雅, 宋小刚, 等. 面向宽基线立体影像匹配的高质量仿射不变特征提取方法[J]. 测绘学报, 2008, 37(1):77-82. DOI:10.3321/j.issn:1001-1595.2008.01.014. CHENG Liang, GONG Jianya, SONG Xiaogang, et al. A method for affine invariant feature extraction with high quality for wide baseline stereo image matching[J]. Acta Geodaetica et Cartographica Sinica, 2008, 37(1):77-82. DOI:10.3321/j.issn:1001-1595.2008.01.014.
[60] 张力, 张继贤. 基于多基线影像匹配的高分辨率遥感影像DEM的自动生成[J]. 武汉大学学报(信息科学版), 2008, 33(9):943-946. ZHANG Li, ZHANG Jixian. Automatic DEM generation from high-resolution satellite imagery based on multiple-baseline image matching[J]. Geomatics and Information Science of Wuhan University, 2008, 33(9):943-946.
[61] 杨化超, 姚国标, 王永波. 基于SIFT的宽基线立体影像密集匹配[J]. 测绘学报, 2011, 40(5):537-543. YANG Huachao, YAO Guobiao, WANG Yongbo. Dense matching for wide base-line stereo images based on SIFT[J]. Acta Geodaetica et Cartographica Sinica, 2011, 40(5):537-543.
[62] 姚国标, 邓喀中, 张力, 等. 融合互补仿射不变特征的倾斜立体影像高精度自动配准方法[J]. 测绘学报, 2013, 42(6):869-876, 883. YAO Guobiao, DENG Kazhong, ZHANG Li, et al. An automated registration method with high accuracy for oblique stereo images based on complementary affine invariant features[J]. Acta Geodaetica et Cartographica Sinica, 2013, 42(6):869-876, 883.
[63] 翁璐斌, 田原, 王彦情, 等. 复杂地形下基于三视约束的景象匹配方法[J]. 航空学报, 2008, 29(5):1288-1296. WENG Lubin, TIAN Yuan, WANG Yanqing, et al. Novel scene matching method for complex terrain based on three-view constraints[J]. Acta Aeronautica et Astronautica Sinica, 2008, 29(5):1288-1296.
[64] KÖSER K, KOCH R. Perspectively invariant normal features[C]//Proceedings of 2007 IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil:IEEE, 2007:1-8.
[65] ZAHARESCU A, BOYER E, VARANASI K, et al. Surface feature detection and description with applications to mesh matching[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009:373-380.
[66] GOSSOW D, WEIKERSDORFER D, BEETZ M. Distinctive texture features from perspective-invariant keypoints[C]//Proceedings of the 21st IEEE International Conference on Pattern Recognition. Tsukuba, Japan:IEEE, 2012:2764-2767.
[67] LIU Wei, WANG Yongtian, CHEN Jing, et al. A completely affine invariant image-matching method based on perspective projection[J]. Machine Vision and Applications, 2012, 23(2):231-242.
[68] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553):436-444.
[69] SCHMIDHUBER J. Deep learning in neural networks:an overview[J]. Neural Networks, 2015, 61:85-117.
[70] ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:1592-1599.
[71] SEKI A, POLLEFEYS M. SGM-Nets:semi-global matching with neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:6640-6649.
[72] MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:4040-4048.
[73] KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy:IEEE, 2017:66-75.
[74] CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT:IEEE, 2018:5410-5418.
[75] PANG Jiahao, SUN Wenxiu, REN J S J, et al. Cascade residual learning:a two-stage convolutional neural network for stereo matching[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops. Venice, Italy:IEEE, 2017:878-886.