影像一致性优化的视图选取策略

doi:10.11947/j.AGCS.2020.20190499

摘要/Abstract

摘要： 在三维重建中，网格优化通常用于解决密集点云构建的三角网格含有较多噪声且缺少细节的问题。现有的变分优化方法利用完整的影像数据对初始网格进行影像一致性优化，但在一定程度上忽视了影像信息的冗余以及视图的质量对网格优化的影响。对此，本文提出主视图选取与从视图选取策略，以提升网格优化的效率与质量。首先综合影像梯度幅值与轮廓检测，构建马尔科夫随机场，为每个三角面选取主视图；其次根据相应的观测条件为每个主视图选取从视图；然后计算主、从视图间归一化加权的影像一致性；最后利用梯度下降法最小化表面能量函数，实现网格优化。试验在定性和定量上证实了本文方法的有效性，表明本文方法能恢复更多精细细节，且优化的时间更短、精度更高。

关键词: 视图选取, 影像一致性, 变分优化, 三维重建

Abstract: In 3D reconstruction, mesh refinement is generally applied to deal with noise and lack of details in triangular mesh built from dense point cloud. The existing variational refinement methods optimize the photo-consistency of the initial mesh by utilizing all the image data, but ignore the redundancy of image information and the impact of view quality on mesh refinement to some extent. In this regard, this paper proposes the strategies of master view selection and slave view selection, so as to improve the efficiency and quality of mesh refinement. Firstly, Markov random field is constructed by combining image gradient magnitude and contour detection to select master view for each triangular facet, and then slave view is selected according to the corresponding observation condition for each master view. Afterwards, we calculate the norm-weighted photo-consistency between master view and slave view, and finally surface energy function is minimized by using gradient decent method to obtain the refined mesh. The experiments show that the proposed method can recover more fine-scale details, meanwhile shorten time and increase accuracy of refinement, which confirms the validity of the proposed method qualitatively and quantitatively.

Key words: view selection, photo-consistency, variational refinement, 3D reconstruction

中图分类号:

P237

朱艳, 颜青松, 曲英杰, 陈欣, 邓非. 影像一致性优化的视图选取策略[J]. 测绘学报, 2020, 49(11): 1463-1472.

ZHU Yan, YAN Qingsong, QU Yingjie, CHEN Xin, DENG Fei. View selection strategy for photo-consistency refinement[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(11): 1463-1472.

参考文献

[1] SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York, NY:IEEE, 2006:519-528.
[2] STRECHA C, VON HANSEN W, VAN GOOL L, et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK:IEEE, 2008:1-8.
[3] SCHÖPS T, SCHÖNBERGER J L, GALLIANI S, et al. A multi-view stereo benchmark with high-resolution images and multi-camera videos[C]//Proceedings 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2017:2538-2547.
[4] ZENG Gang, LHUILLIER M, QUAN Long. Recent methods for reconstructing surfaces from multiple images[C]//Proceedings of the 6th International Conference on Computer Algebra and Geometric Algebra with Applications. Xian, China:Springer-Verlag, 2004:429-447.
[5] TOLA E, STRECHA C, FUA P. Efficient large-scale multi-view stereo for ultra high-resolution image sets[J]. Machine Vision and Applications, 2012, 23(5):903-920.
[6] EBNER T, SCHREER O, FELDMANN I. Fully automated highly accurate 3D reconstruction from multiple views[C]//Proceedings of 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China:IEEE, 2017:2528-2532.
[7] 徐振亮. 轴角描述的车载序列街景影像空中三角测量与三维重建方法研究[J]. 测绘学报, 2015, 44(10):1178. DOI:10.11947/j.AGCS.2015.20150113. XU Zhenliang. Research on aerial triangulation angle/axis representation and 3d recon-struction for vehicle-borne street-level image sequence[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(10):1178. DOI:10.11947/j.AGCS.2015.20150113.
[8] 肖雄武. 具备结构感知功能的倾斜摄影测量场景三维重建[J]. 测绘学报, 2019, 48(6):802. DOI:10.11947/j.AGCS.2019.20180319. XIAO Xiongwu. Oblique photogrammetry based scene 3D reconstruction with structure sensing functions[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):802. DOI:10.11947/j.AGCS.2019.20180319.
[9] VU H H, LABATUT P, PONS J P, et al. High accuracy and visibility-consistent dense multiview stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5):889-901.
[10] KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]//Proceedings of the 4th Eurographics Symposium on Geometry Processing. Aire-la-Ville, Switzerland:SGP, 2006:61-70.
[11] KAZHDAN M, HOPPE H. Screened Poisson surface reconstruction[J]. ACM Transactions on Graphics, 2013, 32(3):61-70.
[12] LABATUT P, PONS J P, KERIVEN R. Robust and efficient surface reconstruction from range data[J]. Computer Graphics Forum, 2009, 28(8):2275-2290.
[13] FAUGERAS O, KERIVEN R. Variational principles, surface evolution, PDEs, level set methods, and the stereo problem[J]. IEEE Transactions on Image Processing, 1998, 7(3):336-344.
[14] YEZZI A, SOATTO S. Stereoscopic segmentation[J]. International Journal of Computer Vision, 2003, 53(1):31-43.
[15] SOLEM J E, OVERGAARD N C. A geometric formulation of gradient descent for variational problems with moving surfaces[C]//Proceedings of the 5th International Conference on Scale Space and PDE Methods in Computer Vision. Hofgeismar, Germany:Springer-Verlag, 2005:419-430.
[16] HIEP V H, KERIVEN R, LABATUT P, et al. Towards high-resolution large-scale multi-view stereo[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009:1430-1437.
[17] 张春森, 张萌萌, 郭丙轩. 影像信息驱动的三角网格模型优化方法[J]. 测绘学报, 2018, 47(7):959-967. DOI:10.11947/j.AGCS.2018.20170733. ZHANG Chunsen, ZHANG Mengmeng, GUO Bingxuan. Refinement of the 3D mesh model driven by the image information[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(7):959-967. DOI:10.11947/j.AGCS.2018.20170733.
[18] TYLECEK R, SARA R. Refinement of surface mesh for accurate multi-view reconstruction[J]. International Journal of Virtual Reality, 2010, 9(1):45-54.
[19] DELAUNOY A, POLLEFEYS M. Photometric bundle adjustment for dense multi-view 3D modeling[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:1486-1493.
[20] LI Zhaoxin, WANG Kuanquan, ZUO Wangmeng, et al. Detail-preserving and content-aware variational multi-view stereo reconstruction[J]. IEEE Transactions on Image Processing, 2016, 25(2):864-877.
[21] LI Shiwei, SIU Singyu, FANG Tian, et al. Efficient multi-view surface refinement with adaptive resolution control[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands:Springer-Verlag, 2016:349-364.
[22] BLÁHA M, ROTHERMEL M, OSWALD M R, et al. Semantically Informed Multiview Surface Refinement[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy:IEEE, 2017:3839-3847.
[23] ROMANONI A, FIORENTI D, MATTEUCCI M. Mesh-based 3D textured urban mapping[C]//Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, BC, Canada:IEEE, 2017:3460-3466.
[24] ROMANONI A, MATTEUCCI M. Mesh-based camera pairs selection and occlusion-aware masking for mesh refinement[J]. Pattern Recognition Letters, 2019, 125:364-372.
[25] SCHÖNBERGER J L, ZHENG Enliang, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands:Springer-Verlag, 2016:501-518.
[26] ZHENG Enliang, DUNN Enrique,JOJIC Vladimir, et al. Patch match based joint view selection and depthmap estimation[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:1510-1517.
[27] ZHU Siyu, FANG Tian, ZHANG Runze, et al. Multi-view geometry compression[C]//Proceedings of the 12th Asian Conference on Computer Vision. Singapore:Springer-Verlag, 2014:3-18.
[28] KOBBELT L, CAMPAGNA S, VORSATZ J, et al. Interactive multi-resolution modeling on arbitrary meshes[C]//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. New York, NY:ACM, 1998:105-114.
[29] PONS J P, KERIVEN R, FAUGERAS O. Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score[J]. International Journal of Computer Vision, 2007, 72(2):179-193.
[30] DELAUNOY A, PRADOS E. Gradient flows for optimizing triangular mesh-based surfaces:applications to 3D reconstruction problems dealing with visibility[J]. International Journal of Computer Vision, 2011, 95(2):100-123.
[31] BOYKOV Y, VEKSLER O, ZABIH R. Fast approximate energy minimization via graph cuts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11):1222-1239.
[32] WAECHTER M, MOEHRLE N, GOESELE M. Let there be color! large-scale texturing of 3D reconstructions[C]//Proceedings of the 13th European Conference on Computer Vision. Cham:Springer-Verlag, 2014:836-850.
[33] KOENDERINK J J. What does the occluding contour tell us about solid shape?[J]. Perception, 1984, 13(3):321-330.
[34] ZHANG Liangpei, ZHANG Yun, CHEN Zhenzhong, et al. Splitting and merging based multi-model fitting for point cloud segmentation[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):78-89.