深度和法线联合估计的深度学习多视密集匹配方法

doi:10.11947/j.AGCS.2024.20230579

摘要/Abstract

摘要：

近年来，基于深度学习的多视密集匹配方法在三维重建任务上展现出了巨大的潜力，但是这些方法在恢复场景的几何结构细节方面仍显不足。在一些传统的多视密集匹配方法中，法线常被作为一种重要的几何约束来辅助推理精细的深度图，然而，这种蕴含了场景几何结构特征的法线信息在深度学习方法中却并未得到充分利用。本文针对多视密集匹配和三维场景重建任务，提出了一种基于深度学习的深度和法线联合估计方法。该方法采用多阶段金字塔的网络结构，从多视影像中同时推理深度和表面法线，并促进二者的联合优化。所提出的网络由特征提取模块、法线辅助的深度估计模块、深度辅助的法线估计模块及深度-法线联合优化模块组成。其中，深度估计模块通过整合表面法线信息构建具有几何感知的代价体，从而推理精细的深度图；法线估计模块利用深度约束构建局部代价体来推理精细的法线信息；联合优化模块则用于增强深度和法线估计结果之间的几何一致性。在WHU-OMVS数据集上的试验结果表明，本文方法在深度推理和表面法线推理任务上都表现出色，并显著优于现有方法。在两种不同数据集上的三维重建结果进一步表明，本文方法能够有效地恢复局部高曲率区域和全局平面区域的几何结构，有助于获得具有良好结构的高质量三维场景模型。

关键词: 深度估计, 法线估计, 多视密集匹配, 深度学习, 三维重建

Abstract:

In recent years, deep learning-based multi-view stereo matching methods have demonstrated significant potential in 3D reconstruction tasks. However, they still exhibit limitations in recovering fine geometric details of scenes. In some traditional multi-view stereo matching methods, surface normal often serves as a crucial geometric constraint to assist in finer depth inference. Nevertheless, the surface normal information, which encapsulates the geometric information of the scene, has not been fully utilized in modern learning-based methods. This paper introduces a deep learning-based joint depth and surface normal estimation method for multi-view dense matching and 3D scene reconstruction task. The proposed method employs a multi-stage pyramid structure to simultaneously infer depth and surface normal from multi-view images and promote their joint optimization. It consists of a feature extraction module, a normal-assisted depth estimation module, a depth-assisted normal estimation module, and a depth-normal joint optimization module. Specifically, the depth estimation module constructs a geometry-aware cost volume by integrating surface normal information for fine depth estimation. The normal estimation module utilizes depth constraints to build a local cost volume for inferring fine-grained normal maps. The joint optimization module further enhances the geometric consistency between depth and normal estimation. Experimental results on the WHU-OMVS dataset demonstrate that the proposed method performs exceptionally well in both depth and surface normal estimation, outperforming existing methods. Furthermore, the 3D reconstruction results on two different datasets indicate that the proposed method effectively recovers the geometric structures of both local high-curvature areas and global planar regions, contributing to well-structured and high-quality 3D scene models.

Key words: depth estimation, normal estimation, multi-view dense matching, deep learning, 3D reconstruction

中图分类号:

P237

刘瑾, 季顺平. 深度和法线联合估计的深度学习多视密集匹配方法[J]. 测绘学报, 2024, 53(12): 2391-2403.

Jin LIU, Shunping JI. Deep learning based multi-view dense matching with joint depth and surface normal estimation[J]. Acta Geodaetica et Cartographica Sinica, 2024, 53(12): 2391-2403.

图/表 15

图1

图2

图3

表1

图4

表2

图5

表3

图6

表4

图7

表5

图8

表6

表7

参考文献 31

[1]	董秀军, 邓博, 袁飞云, 等. 航空遥感在地质灾害领域的应用：现状与展望[J]. 武汉大学学报(信息科学版), 2023, 48(12): 1897-1913.
	DONG Xiujun, DENG Bo, YUAN Feiyun, et al. Application of aerial remote sensing in geological hazards: current situation and prospects[J]. Geomatics and Information Science of Wuhan University, 2023, 48(12): 1897-1913.
[2]	沙洪俊, 袁修孝. 双目影像密集匹配方法的回顾与展望[J]. 武汉大学学报(信息科学版), 2023, 48(11): 1813-1833.
	SHA Hongjun, YUAN Xiuxiao. State-of-the-art binocular image dense matching method[J]. Geomatics and Information Science of Wuhan University, 2023, 48(11): 1813-1833.
[3]	HONG Danfeng, ZHANG Bing, LI Hao, et al. Cross-city matters: a multimodal remote sensing benchmark dataset for cross-city semantic segmentation using high-resolution domain adaptation networks[J]. Remote Sensing of Environment, 2023, 299: 113856.
[4]	HONG Danfeng, ZHANG Bing, LI Xuyang, et al. SpectralGPT: spectral remote sensing foundation model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(8): 5227-5244.
[5]	KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]//Proceedings of 2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 4015-4026.
[6]	SUN Xian, WANG Peijin, LU Wanxuan, et al. RingMo: a remote sensing foundation model with masked image modeling[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 3194732.
[7]	刘瑾, 季顺平. 基于深度学习的航空遥感影像密集匹配[J]. 测绘学报, 2019, 48(9): 1141-1150. DOI:. doi: 10.11947/j.AGCS.2019.20180247
	LIU Jin, JI Shunping. Deep learning based dense matching for aerial remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(9): 1141-1150. DOI:. doi: 10.11947/j.AGCS.2019.20180247
[8]	JI Shunping, LIU Jin, LU Meng. CNN-based dense image matching for aerial remote sensing images[J]. Photogrammetric Engineering & Remote Sensing, 2019, 85(6): 415-424.
[9]	季顺平, 罗冲, 刘瑾. 基于深度学习的立体影像密集匹配方法综述[J]. 武汉大学学报(信息科学版), 2021, 46(2): 193-202.
	JI Shunping, LUO Chong, LIU Jin. A review of dense stereo image matching methods based on deep learning[J]. Geomatics and Information Science of Wuhan University, 2021, 46(2): 193-202.
[10]	龚健雅, 季顺平. 摄影测量与深度学习[J]. 测绘学报, 2018, 47(6): 693-704. DOI:. doi: 10.11947/j.AGCS.2018.20170640
	GONG Jianya, JI Shunping. Photogrammetry and deep learning[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 693-704. DOI:. doi: 10.11947/j.AGCS.2018.20170640
[11]	LAGA H, JOSPIN L V, BOUSSAID F, et al. A survey on deep learning techniques for stereo-based depth estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 1738-1764.
[12]	YAO Yao, LUO Zixin, LI Shiwei, et al. MVSNet: depth inference for unstructured multi-view stereo[C]//Proceedings of 2018 European Conference on Computer Vision. Munich: Springer, 2018: 767-783.
[13]	GU Xiaodong, FAN Zhiwen, ZHU Siyu, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2495-2504.
[14]	WEI Zizhuang, ZHU Qingtian, MIN Chen, et al. AA-RMVSNet: adaptive aggregation recurrent multi-view stereo network[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 6187-6196.
[15]	CHANG Jiaren, CHANG Peichun, CHEN Yongsheng. Attention-aware feature aggregation for real-time stereo matching on edge devices[C]//Proceedings of 2020 Asian Conference on Computer Vision. Kyoto: Springer, 2020: 365-380.
[16]	LIU Jin, JI Shunping. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6050-6059.
[17]	YAN Jianfeng, WEI Zizhuang, YI Hongwei, et al. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking[C]//Proceedings of 2020 European Conference on Computer Vision. Glasgow: Springer, 2020: 674-689.
[18]	YU Dawen, JI Shunping, LIU Jin, et al. Automatic 3D building reconstruction from multi-view aerial images with deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171: 155-170.
[19]	LIU Jin, GAO Jian, JI Shunping, et al. Deep learning based multi-view stereo matching and 3D scene reconstruction from oblique aerial images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 204: 42-60.
[20]	GAO Jian, LIU Jin, JI Shunping. Rational polynomial camera model warping for deep learning based satellite multi-view stereo matching[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 6128-6137.
[21]	GAO Jian, LIU Jin, JI Shunping. A general deep learning based framework for 3D reconstruction from multi-view stereo satellite images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 195: 446-461.
[22]	BLEYER M, RHEMANN C, ROTHER C. PatchMatch stereo-stereo matching with slanted support windows[C]//Proceedings of 2011 British Machine Vision Conference. Dundee: British Machine Vision Association, 2011: 1-11.
[23]	GALLIANI S, LASINGER K, SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 873-881.
[24]	SCHÖNBERGER J L, ZHENG Enliang, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Proceedings of 2016 European Conference on Computer Vision. Amsterdam: Springer Cham, 2016: 501-518.
[25]	XU Qingshan, TAO Wenbing. Multi-scale geometric consistency guided multi-view stereo[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5478-5487.
[26]	LONG Xiaoxiao, LIU Lingjie, THEOBALT C, et al. Occlusion-aware depth estimation with adaptive normal constraints[C]//Proceedings of 2020 European Conference on Computer Vision. Glasgow: Springer, 2020: 640-657.
[27]	KUSUPATI U, CHENG Shuo, CHEN Rui, et al. Normal assisted stereo depth estimation[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2186-2196.
[28]	ZHAO Wang, LIU Shaohui, WEI Yi, et al. A confidence-based iterative solver of depths and surface normals for deep multi-view stereo[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Visio. Montreal: IEEE, 2021: 6148-6157.
[29]	YANG Zhenheng, WANG Peng, XU Wei, et al. Unsupervised learning of geometry with edge-aware depth-normal consistency[EB/OL]. [2023-10-01]. https://arxiv.org/abs/1711.03665v1.
[30]	KNAPITSCH A, PARK J, ZHOU Qianyi, et al. Tanks and temples[J]. ACM Transactions on Graphics, 2017, 36(4): 1-13.
[31]	AANÆS H, JENSEN R R, VOGIATZIS G, et al. Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120(2): 153-168.

方法	MAE/m↓	PAG-D_{0.1 m}/（%）↑	PAG-D_{0.3 m}/（%）↑	PAG-D_{0.6 m}/（%）↑
COLMAP	0.279	48.40	75.60	88.00
NAS	0.289	31.82	71.58	90.32
Cas-MVSNet	0.157	60.14	92.00	96.87
DN-MVS（本文方法）	0.132	61.89	93.09	97.81

方法	mean/（°）↓	median/（°）↓	PAG-N_10°/（%）↑	PAG-N_20°/（%）↑	PAG-N_30°/（%）↑
COLMAP	20.06	11.54	46.27	68.41	79.28
NAS	16.84	14.20	30.03	70.29	88.61
Cas-MVSNet（D2N）	11.86	7.88	61.91	85.69	93.03
DN-MVS（本文方法）	8.23	5.70	72.20	91.86	97.19

方法	边缘区域			非边缘区域			完整区域
方法	PAG-S_{0.2 m}/（%）↑	PAG-S_{0.4 m}/（%）↑	PAG-S_{0.6 m}/（%）↑	PAG-S_{0.05 m}/（%）↑	PAG-S_{0.1 m}/（%）↑	PAG-S_{0.2 m}/（%）↑	PAG-S_{0.2 m}/（%）↑	PAG-S_{0.4 m}/（%）↑	PAG-S_{0.6 m}/（%）↑
COLMAP	50.08	71.55	78.01	47.70	65.80	82.74	80.32	92.38	95.67
Open MVS	61.96	78.37	83.35	42.66	64.71	84.94	83.24	94.07	96.53
Cas-MVSNet	50.61	71.30	81.27	44.68	69.90	87.57	84.84	94.40	97.02
DN-MVS	61.05	79.20	84.99	50.99	73.23	88.81	86.76	95.11	97.11
（本文方法）	（+10.44）	（+7.90）	（+3.72）	（+6.31）	（+3.33）	（+1.24）	（+1.92）	（+0.71）	（+0.09）

方法	平均距离/m↓			百分比（阈值<0.2 m）/（%）↑			百分比（阈值<0.4 m/（%））↑
方法	Acc.	Comp.	overall	Acc.	Comp.	F₁值	Acc.	Comp.	F₁值
COLMAP	0.509	0.367	0.438	46.14	62.91	53.24	66.10	81.96	73.18
Open MVS	0.410	0.306	0.358	47.57	72.97	57.59	75.03	84.61	79.53
Cas-MVSNet	0.410	0.256	0.333	51.41	80.13	62.63	74.82	87.10	80.49
DN-MVS	0.404	0.237	0.320	51.51	81.53	63.13	75.28	87.86	81.14

消融设置	深度推理			法线推理
消融设置	MAE/m↓	PAG-D_{0.3 m}/（%）↑	PAG-D_{0.6 m}/（%）↑	mean/（°）↓	PAG-N_20°/（%）↑	PAG-N_30°/（%）↑
试验1	0.157	92.00	96.87	11.86	85.69	93.03
试验2	0.152	91.55	97.16	10.14	87.63	95.73
试验3	0.142	92.60	97.63	9.39	88.58	96.32
试验4（本文方法）	0.132	93.09	97.81	8.23	91.86	97.19