Deep learning based dense matching for aerial remote sensing images

doi:10.11947/j.AGCS.2019.20180247

Abstract

Abstract: This work studied that the application of deep learning based stereo methods in aerial remote sensing images, including its performance evaluation, the comparison with classical methods and generalization ability estimation.Three convolution neural networks are applied, MC-CNN(matching cost convolutional neural network), GC-Net(geometry and context network) and DispNet(disparity estimation network), on aerial stereo image pairs. The results are compared with SGM (semi-global matching) and a commercial software SURE. Secondly, the generalization ability of the MC-CNN and GC-Net are evaluated with models pretrained on other datasets. Finally, fine tuning on a small number of target training data with pretrained models are compared to direct training. Three sets of aerial images and two open-source street data sets are used for test. Experiments show that:firstly, deep learning methods perform slightly better than traditional methods; secondly, both GC-Net and MC-CNN have demonstrated good generalization ability, and can get satisfactory 3PE (3-pixel-error) results on aerial images using a model pretrained on available stereo benchmarks; thirdly, when the training samples in target dataset are insufficient, the strategy of fine-tuning on a pretrained model can improve the effect of direct training.

Key words: stereo matching, dense matching, aerial images, convolutional neural network, deep learning

CLC Number:

P237

LIU Jin, JI Shunping. Deep learning based dense matching for aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9): 1141-1150.

References

[1] SCHARSTEIN D, SZELISKI R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47(1-3):7-42.
[2] KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of 2007 IEEE International Conference on Computer Vision. Venice, Italy:IEEE, 2017:66-75.
[3] BOYKOV Y Y, JOLLY M P. Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Vancouver, BC, Canada:IEEE, 2001:105-112.
[4] FURUKAWA Y, PONCE J. Accurate, dense, and robust multiview stereopsis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8):1362-1376.
[5] HIRSCHMULLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2):328-341.
[6] BLEYER M, RHEMANN C, ROTHER C. Patchmatch stereo-stereo matching with slanted support windows[C]//Proceedings of the British Machine Vision Conference. Dundee:BMVA Press, 2011:14.1-14.11.
[7] ROTHERMEL M, WENZEL K, FRITSCH D, et al. SURE:photogrammetric surface reconstruction from imagery[C]//Proceedings of 2012 LC3D Workshop. Berlin:[s.n.], 2012:29.
[8] GOODFELLOW I, BENGIO Y, COURVILLE A. Deep learning[M]. Cambridge:MIT Press, 2016.
[9] 郑卓, 方芳, 刘袁缘, 等. 高分辨率遥感影像场景的多尺度神经网络分类法[J]. 测绘学报, 2018, 47(5):620-630. DOI:10.11947/j.AGCS.2018.20170191. ZHENG Zhuo, FANG Fang, LIU Yuanyuan, et al. Joint multi-scale convolution neural network for scene classification of high resolution remote sensing imagery[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(5):620-630. DOI:10.11947/j.AGCS.2018.20170191.
[10] 龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640. GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):693-704. DOI:10.11947/j.AGCS.2018.20170640.
[11] 范大昭, 董杨, 张永生. 卫星影像匹配的深度卷积神经网络方法[J]. 测绘学报, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627. FAN Dazhao, DONG Yang, ZHANG Yongsheng. Satellite image matching method based on deep convolution neural network[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6):844-853. DOI:10.11947/j.AGCS.2018.20170627.
[12] ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:1592-1599.
[13] SEKI A, POLLEFEYS M. SGM-Nets:Semi-global matching with neural networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:6640-6649.
[14] MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:4040-4048.
[15] CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT:IEEE, 2018:5410-5418.
[16] PANG Jiahao, SUN Wenxiu, REN J S, et al. Cascade residual learning:A two-stage convolutional neural network for stereo matching[C]//Proceedings of 2017 IEEE International Conference on Computer Vision Workshops. Venice, Italy:IEEE, 2017:878-886.
[17] SHAKED A, WOLF L. Improved stereo matching with constant highway networks and reflective confidence learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI:IEEE, 2017:6901-6910.
[18] ZHONG Yiran, DAI Yuchao, LI Hongdong. Self-supervised learning for stereo matching with self-improving ability[J]. arXiv:1709.00930, 2017.
[19] ZHANG Ke, LU Jiangbo, LAFRUIT G. Cross-based local stereo matching using orthogonal integral images[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(7):1073-1079.
[20] DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet:Learning optical flow with convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015:2758-2766.
[21] PAN S J, YANG Qiang. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10):1345-1359.
[22] MENZE M, GEIGER A. Object scene flow for autonomous vehicles[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA:IEEE, 2015:3061-3070.
[23] MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV:IEEE, 2016:4040-4048.
[24] FUSIELLO A, TRUCCO E, VERRI A. A compact algorithm for rectification of stereo pairs[J]. Machine Vision and Applications, 2000, 12(1):16-22.
[25] LIANG Zhengfa, FENG Yiliu, GUO Yulan, et al. Learning for disparity estimation through feature constancy[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT:IEEE, 2018:2811-2820.
[26] OQUAB M, BOTTOU L, LAPTEV I, et al. Learning and transferring mid-level image representations using convolutional neural networks[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:1717-1724.