An intelligent 3D reconstruction framework via deep learning based multi-view image matching

doi:10.11947/j.AGCS.2025.20230306

Abstract

Abstract:

The real-scene 3D model reconstruction of the ground surface based on high-resolution stereo or multi-view images is a key research topic in photogrammetry and computer vision, with dense image matching being the core technologies. At present, the mainstream 3D reconstruction algorithms are still based on the manual-designed methods. Although deep learning-based dense matching algorithms have shown excellent performance in recent years, they have not yet been deployed in 3D reconstruction projects, and there are few reports on the deployment of 3D reconstruction frameworks or software based on deep learning or intelligent methods, both domestically and internationally. To promote the application of modern artificial intelligence methods in large-scale 3D surface reconstruction task, this article proposes a general intelligent framework for real-scene 3D reconstruction called Deep3D, with the core component being a deep learning dense matching network. This framework includes complete processes of aerial triangulation, optimal view selection, deep learning-based dense matching, depth map fusion, and 3D surface model reconstruction, aiming for urban-level real-scene 3D surface reconstruction from multi-view remote sensing images. This general framework integrates the processing of aerial and satellite images by incorporating the perspective model and the rational polynomial coefficient model into the network, as well as the processing of binocular, multi-view and oblique view images by using adaptive multi-view alignment and aggregation strategies. This paper compares the Deep3D framework, software and open source solutions on two sets of oblique aerial images, and confirms that the proposed Deep3D framework performs essentially on par with or slightly better than software, far better than existing open source frameworks. This article also discusses the performance on satellite multi-view images of different methods. This study provides an outlook and reference for the application of deep learning methods in the real-scene 3D reconstruction projects.

Key words: 3D reconstruction framework, deep learning, image dense matching, remote sensing images, real-scene 3D model

CLC Number:

P237

Shunping JI, Jin LIU, Jian GAO, Jianya GONG. An intelligent 3D reconstruction framework via deep learning based multi-view image matching[J]. Acta Geodaetica et Cartographica Sinica, 2025, 54(9): 1633-1646.

Figures/Tables 9

Fig. 1

Tab. 1

Tab. 2

Fig. 2

Tab. 3

Fig. 3

Tab. 4

Tab. 5

Fig. 4

References 47

[1]	SCHÖNBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 4104-4113.
[2]	SCHÖNBERGER J L, ZHENG Enliang, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Proceedings of 2016 Computer Vision-ECCV. Cham: Springer, 2016: 501-518.
[3]	MOULON P, MONASSE P, PERROT R, et al. OpenMVG: open multiple view geometry[C]//Proceedings of 2017 Reproducible Research in Pattern Recognition. Cham: Springer, 2017: 60-74.
[4]	OpenMVS: open multi-view stereo reconstruction library[CP]. [2023-07-24]. https://github.com/cdcseacave/openMVS.
[5]	GRIWODZ C, GASPARINI S, CALVET L, et al. AliceVision Meshroom: an open-source 3D reconstruction pipeline[C]//Proceedings of the 12th ACM Multimedia Systems Conference. Istanbul, Turkey: ACM press, 2021: 241-247.
[6]	RUPNIK E, DAAKIR M, PIERROT DESEILLIGNY M. MicMac-a free, open-source solution for photogrammetry[J]. Open Geospatial Data, Software and Standards, 2017, 2(1): 14.
[7]	HIRSCHMULLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341.
[8]	BLEYER M, RHEMANN C, ROTHER C. PatchMatch stereo - stereo matching with slanted support windows[C]//Proceedings of 2011 British Machine Vision Conference. Dundee: British Machine Vision Association, 2011: 14.
[9]	ROTHERMEL M, WENZEL K, FRITSCH D, et al. SURE: photogrammetric surface reconstruction from imagery[C]//Proceedings of 2012 LC3D Workshop. Berlin: [s.n.], 2012.
[10]	龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6): 693-704. DOI: . doi: 10.11947/j.AGCS.2018.20170640
	GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 693-704. DOI: . doi: 10.11947/j.AGCS.2018.20170640
[11]	LIU J, JI S, ZHANG C, et al. Evaluation of deep learning based stereo matching methods: from ground to aerial images[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, XLII-2: 593-597.
[12]	YAO Yao, LUO Zixin, LI Shiwei, et al. MVSNet: depth inference for unstructured multi-view stereo[C]//Proceedings of 2018 Computer Vision-ECCV. Cham: Springer, 2018: 785-801.
[13]	CHANG Jiaren, CHANG Peichun, CHEN Yongsheng. Attention-aware feature aggregation for real-time stereo matching on edge devices[C]//Proceedings of 2020 Computer Vision. Cham: Springer, 2021: 365-380.
[14]	ZHANG Xudong, HU Yutao, WANG Haochen, et al. Long-range attention network for multi-view stereo[C]//Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2021: 3781-3790.
[15]	DING Yikang, YUAN Wentao, ZHU Qingtian, et al. TransMVSNet: global context-aware multi-view stereo network with transformers[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 8575-8584.
[16]	ZHANG Jingyang, LI Shiwei, LUO Zixin, et al. Vis—MVSNet: visibility-aware multi-view stereo network[J]. International Journal of Computer Vision, 2023, 131(1): 199-214.
[17]	XU Qingshan, SU Wanjuan, QI Yuhang, et al. Learning inverse depth regression for pixel wise visibility-aware multi-view stereo networks[J]. International Journal of Computer Vision, 2022, 130(8): 2040-2059.
[18]	WEI Zizhuang, ZHU Qingtian, MIN Chen, et al. AA-RMVSNet: adaptive aggregation recurrent multi-view stereo network[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 6167-6176.
[19]	YAO Yao, LUO Zixin, LI Shiwei, et al. Recurrent MVSNet for high-resolution multi-view stereo depth inference[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5520-5529.
[20]	LIU Jin, JI Shunping. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6049-6058.
[21]	YAN Jianfeng, WEI Zizhuang, YI Hongwei, et al. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking[C]//Proceedings of 2020 Computer Vision. Cham: Springer, 2020: 674-689.
[22]	CHEN Poheng, YANG H C, CHEN Kuanwen, et al. MVSNet: learning depth-based attention pyramid features for multi-view stereo[J]. IEEE Transactions on Image Processing, 2020, 29: 7261-7273.
[23]	PENG Rui, WANG Rongjie, WANG Zhenyu, et al. Rethinking depth estimation for multi-view stereo: a unified representation[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 8635-8644.
[24]	GU Xiaodong, FAN Zhiwen, ZHU Siyu, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2492-2501.
[25]	CHENG Shuo, XU Zexiang, ZHU Shilin, et al. Deep stereo using adaptive thin volume representation with uncertainty awareness[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2521-2531.
[26]	YANG Jiayu, MAO Wei, ALVAREZ J M, et al. Cost volume pyramid based depth inference for multi-view stereo[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4876-4885.
[27]	DAI Yuchao, ZHU Zhidong, RAO Zhibo, et al. MVS2: deep unsupervised multi-view stereo with multi-view symmetry[C]//Proceedings of 2019 International Conference on 3D Vision. Quebec City: IEEE, 2019: 1-8.
[28]	HUANG Baichuan, YI Hongwei, HUANG Can, et al. M3VSNET: unsupervised multi-metric multi-view stereo network[C]//Proceedings of 2021 IEEE International Conference on Image Processing. Anchorage: IEEE, 2021: 3163-3167.
[29]	XU Hongbin, ZHOU Zhipeng, QIAO Yu, et al. Self-supervised multi-view stereo via effective co-segmentation and data-augmentation[C]//Proceedings of 2021 AAAI Conference on Artificial Intelligence. [s.l.]: PKS, 2021: 3030-3038.
[30]	刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9): 1141-1150. DOI: . doi: 10.11947/j.AGCS.2019.20180247
	LIU Jin, JI Shunping. Deep learning based dense matching for aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9): 1141-1150. DOI: . doi: 10.11947/j.AGCS.2019.20180247
[31]	上海人工智能实验室. 书生·天际[EB/OL]. [2023-07-24]. https://landmark.intern-ai.org.cn/.
	Shanghai AI Lab. LandMark[EB/OL]. [2023-07-24]. https://landmark.intern-ai.org.cn/.
[32]	CHEN Peimin, HUANG Huabing, LIU Jinying, et al. Leveraging Chinese GaoFen-7 imagery for high-resolution building height estimation in multiple cities[J]. Remote Sensing of Environment, 2023, 298: 113802.
[33]	FRASER C S, HANLEY H B. Bias compensation in rational functions for IKONOS satellite imagery[J]. Photogrammetric Engineering & Remote Sensing, 2003, 69(1): 53-57.
[34]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[C]//Proceedings of 2014 Computer Vision. Cham: Springer, 2014: 346-361.
[35]	GAO Jian, LIU Jin, JI Shunping. A general deep learning based framework for 3D reconstruction from multi-view stereo satellite images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 195: 446-461.
[36]	JANCOSEK M, PAJDLA T. Exploiting visibility information in surface reconstruction to preserve weakly supported surfaces[J]. International Scholarly Research Notices, 2014, 2014: 798595.
[37]	VU H H, LABATUT P, PONS J P, et al. High accuracy and visibility-consistent dense multiview stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 889-901.
[38]	WAECHTER M, MOEHRLE N, GOESELE M. Let there be color! large-scale texturing of 3D reconstructions[C]//Proceedings of 2014 Computer Vision. Cham: Springer, 2014: 836-850.
[39]	PÉREZ P, GANGNET M, BLAKE A. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318.
[40]	AANÆS H, JENSEN R R, VOGIATZIS G, et al. Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120(2): 153-168.
[41]	GAO Jian, LIU Jin, JI Shunping. Rational polynomial camera model warping for deep learning based satellite multi-view stereo matching[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 6128-6137.
[42]	BOSCH M, KURTZ Z, HAGSTROM S, et al. A multiple view stereo benchmark for satellite imagery[C]//Proceedings of 2016 IEEE Applied Imagery Pattern Recognition Workshop. Washington DC: IEEE, 2016: 1-9.
[43]	AGISOFT. Agisoft metashape[EB/OL]. [2023-07-24]. https://www.agisoft.com/.
[44]	ZHANG Kai, SNAVELY N, SUN Jin. Leveraging vision reconstruction pipelines for satellite imagery[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul: IEEE, 2019: 2139-2148.
[45]	FAN Bin, KONG Qingqun, WANG Xinchao, et al. A performance evaluation of local features for image-based 3D reconstruction[J]. IEEE Transactions on Image Processing, 2019, 28(10): 4774-4789.
[46]	MA Jiayi, JIANG Xingyu, FAN Aoxiang, et al. Image matching from handcrafted to deep features: a survey[J]. International Journal of Computer Vision, 2021, 129(1): 23-79.
[47]	ZHANG Jingyang, YAO Yao, QUAN Long. Learning signed distance field for multi-view surface reconstruction[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 6505-6514.

超参数类型	设置
深度采样平面数量	｛48，32，8｝
深度采样间隔	WHU-OMVS：｛（d_max－d_min）/48 m，0.2 m，0.1 m｝ DTU：｛4 mm，2 mm，1 mm｝
匹配视角数量	5
损失函数权重因子｛λ₀，λ₁，λ₂，λ₃｝	｛0.5，0.5，1，2｝
迭代次数	～25万
训练批次大小	1
优化器与学习率	RMSProp
学习率	0.001

软件/方案	PAG_{0.2 m}/（%）↑	PAG_{0.4 m}/（%）↑	PAG_{0.6 m}/（%）↑	MAE/m T=20 m↓	RMSE/m T=20 m↓	Runtime/min↓
ContextCapture	83.35	95.28	97.18	0.190	0.916	128
Metashape	88.34	95.41	97.21	0.170	0.972	193
SURE-Aerial	60.68	82.20	90.49	0.324	1.049	134
COLMAP	80.33	92.39	95.67	0.236	1.193	412
OpenMVS	83.26	94.09	96.53	0.202	0.998	299
Deep3D	86.75	95.47	97.60	0.166	0.803	187

软件/方案	PAG_{0.2 m}/（%）↑	PAG_{0.4 m}/（%）↑	PAG_{0.6 m}/（%）↑	MAE/m T=20 m↓	RMSE/m T=20 m↓	Runtime/min↓
ContextCapture	55.49	75.28	80.69	0.822	2.189	65.2
Metashape	63.27	76.42	80.55	0.811	2.153	133.2
SURE-Aerial	54.87	71.98	78.65	0.841	2.177	75.0
COLMAP	65.29	76.07	80.41	0.812	2.205	300.5
OpenMVS	63.57	75.38	80.22	0.830	2.243	204.5
Deep3D	66.98	76.83	81.01	0.818	2.250	100.9

训练策略	PAG_{0.2 m}/（%）↑	PAG_{0.4 m}/（%）↑	PAG_{0.6 m}/（%）↑	MAE/m T=20 m↓	RMSE/m T=20 m↓
DTU预训练模型	65.89	76.31	80.74	0.822	2.257
WHU-OMVS预训练模型	66.98	76.83	81.01	0.818	2.250
无监督训练模型	64.93	76.39	80.81	0.832	2.281
模拟数据训练模型	66.51	76.84	81.05	0.819	2.248

数据集	评价指标	Metashape（PhotoScan）	Adapted COLMAP	Deep3D
WHU-TLC（ZY3-02影像）	RMSE/m	13.047	4.714	3.654
	MAE/m	2.693	2.168	1.895
	PAG_{2.5 m}/（%）	56.59	58.78	64.82
	PAG_{7.5 m}/（%）	75.46	76.80	80.05
MVS3D（World View-3影像）	RMSE/m	3.464	8.397	3.242
	Median/m	0.495	0.371	0.397
	PAG_{1.0 m}/（%）	56.73	50.38	60.48