View selection strategy for photo-consistency refinement

doi:10.11947/j.AGCS.2020.20190499

Abstract

Abstract: In 3D reconstruction, mesh refinement is generally applied to deal with noise and lack of details in triangular mesh built from dense point cloud. The existing variational refinement methods optimize the photo-consistency of the initial mesh by utilizing all the image data, but ignore the redundancy of image information and the impact of view quality on mesh refinement to some extent. In this regard, this paper proposes the strategies of master view selection and slave view selection, so as to improve the efficiency and quality of mesh refinement. Firstly, Markov random field is constructed by combining image gradient magnitude and contour detection to select master view for each triangular facet, and then slave view is selected according to the corresponding observation condition for each master view. Afterwards, we calculate the norm-weighted photo-consistency between master view and slave view, and finally surface energy function is minimized by using gradient decent method to obtain the refined mesh. The experiments show that the proposed method can recover more fine-scale details, meanwhile shorten time and increase accuracy of refinement, which confirms the validity of the proposed method qualitatively and quantitatively.

Key words: view selection, photo-consistency, variational refinement, 3D reconstruction

CLC Number:

P237

ZHU Yan, YAN Qingsong, QU Yingjie, CHEN Xin, DENG Fei. View selection strategy for photo-consistency refinement[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(11): 1463-1472.

References

[1] SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York, NY:IEEE, 2006:519-528.
[2] STRECHA C, VON HANSEN W, VAN GOOL L, et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK:IEEE, 2008:1-8.
[3] SCHÖPS T, SCHÖNBERGER J L, GALLIANI S, et al. A multi-view stereo benchmark with high-resolution images and multi-camera videos[C]//Proceedings 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI:IEEE, 2017:2538-2547.
[4] ZENG Gang, LHUILLIER M, QUAN Long. Recent methods for reconstructing surfaces from multiple images[C]//Proceedings of the 6th International Conference on Computer Algebra and Geometric Algebra with Applications. Xian, China:Springer-Verlag, 2004:429-447.
[5] TOLA E, STRECHA C, FUA P. Efficient large-scale multi-view stereo for ultra high-resolution image sets[J]. Machine Vision and Applications, 2012, 23(5):903-920.
[6] EBNER T, SCHREER O, FELDMANN I. Fully automated highly accurate 3D reconstruction from multiple views[C]//Proceedings of 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China:IEEE, 2017:2528-2532.
[7] 徐振亮. 轴角描述的车载序列街景影像空中三角测量与三维重建方法研究[J]. 测绘学报, 2015, 44(10):1178. DOI:10.11947/j.AGCS.2015.20150113. XU Zhenliang. Research on aerial triangulation angle/axis representation and 3d recon-struction for vehicle-borne street-level image sequence[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(10):1178. DOI:10.11947/j.AGCS.2015.20150113.
[8] 肖雄武. 具备结构感知功能的倾斜摄影测量场景三维重建[J]. 测绘学报, 2019, 48(6):802. DOI:10.11947/j.AGCS.2019.20180319. XIAO Xiongwu. Oblique photogrammetry based scene 3D reconstruction with structure sensing functions[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6):802. DOI:10.11947/j.AGCS.2019.20180319.
[9] VU H H, LABATUT P, PONS J P, et al. High accuracy and visibility-consistent dense multiview stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5):889-901.
[10] KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]//Proceedings of the 4th Eurographics Symposium on Geometry Processing. Aire-la-Ville, Switzerland:SGP, 2006:61-70.
[11] KAZHDAN M, HOPPE H. Screened Poisson surface reconstruction[J]. ACM Transactions on Graphics, 2013, 32(3):61-70.
[12] LABATUT P, PONS J P, KERIVEN R. Robust and efficient surface reconstruction from range data[J]. Computer Graphics Forum, 2009, 28(8):2275-2290.
[13] FAUGERAS O, KERIVEN R. Variational principles, surface evolution, PDEs, level set methods, and the stereo problem[J]. IEEE Transactions on Image Processing, 1998, 7(3):336-344.
[14] YEZZI A, SOATTO S. Stereoscopic segmentation[J]. International Journal of Computer Vision, 2003, 53(1):31-43.
[15] SOLEM J E, OVERGAARD N C. A geometric formulation of gradient descent for variational problems with moving surfaces[C]//Proceedings of the 5th International Conference on Scale Space and PDE Methods in Computer Vision. Hofgeismar, Germany:Springer-Verlag, 2005:419-430.
[16] HIEP V H, KERIVEN R, LABATUT P, et al. Towards high-resolution large-scale multi-view stereo[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009:1430-1437.
[17] 张春森, 张萌萌, 郭丙轩. 影像信息驱动的三角网格模型优化方法[J]. 测绘学报, 2018, 47(7):959-967. DOI:10.11947/j.AGCS.2018.20170733. ZHANG Chunsen, ZHANG Mengmeng, GUO Bingxuan. Refinement of the 3D mesh model driven by the image information[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(7):959-967. DOI:10.11947/j.AGCS.2018.20170733.
[18] TYLECEK R, SARA R. Refinement of surface mesh for accurate multi-view reconstruction[J]. International Journal of Virtual Reality, 2010, 9(1):45-54.
[19] DELAUNOY A, POLLEFEYS M. Photometric bundle adjustment for dense multi-view 3D modeling[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:1486-1493.
[20] LI Zhaoxin, WANG Kuanquan, ZUO Wangmeng, et al. Detail-preserving and content-aware variational multi-view stereo reconstruction[J]. IEEE Transactions on Image Processing, 2016, 25(2):864-877.
[21] LI Shiwei, SIU Singyu, FANG Tian, et al. Efficient multi-view surface refinement with adaptive resolution control[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands:Springer-Verlag, 2016:349-364.
[22] BLÁHA M, ROTHERMEL M, OSWALD M R, et al. Semantically Informed Multiview Surface Refinement[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy:IEEE, 2017:3839-3847.
[23] ROMANONI A, FIORENTI D, MATTEUCCI M. Mesh-based 3D textured urban mapping[C]//Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, BC, Canada:IEEE, 2017:3460-3466.
[24] ROMANONI A, MATTEUCCI M. Mesh-based camera pairs selection and occlusion-aware masking for mesh refinement[J]. Pattern Recognition Letters, 2019, 125:364-372.
[25] SCHÖNBERGER J L, ZHENG Enliang, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands:Springer-Verlag, 2016:501-518.
[26] ZHENG Enliang, DUNN Enrique,JOJIC Vladimir, et al. Patch match based joint view selection and depthmap estimation[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH:IEEE, 2014:1510-1517.
[27] ZHU Siyu, FANG Tian, ZHANG Runze, et al. Multi-view geometry compression[C]//Proceedings of the 12th Asian Conference on Computer Vision. Singapore:Springer-Verlag, 2014:3-18.
[28] KOBBELT L, CAMPAGNA S, VORSATZ J, et al. Interactive multi-resolution modeling on arbitrary meshes[C]//Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. New York, NY:ACM, 1998:105-114.
[29] PONS J P, KERIVEN R, FAUGERAS O. Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score[J]. International Journal of Computer Vision, 2007, 72(2):179-193.
[30] DELAUNOY A, PRADOS E. Gradient flows for optimizing triangular mesh-based surfaces:applications to 3D reconstruction problems dealing with visibility[J]. International Journal of Computer Vision, 2011, 95(2):100-123.
[31] BOYKOV Y, VEKSLER O, ZABIH R. Fast approximate energy minimization via graph cuts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11):1222-1239.
[32] WAECHTER M, MOEHRLE N, GOESELE M. Let there be color! large-scale texturing of 3D reconstructions[C]//Proceedings of the 13th European Conference on Computer Vision. Cham:Springer-Verlag, 2014:836-850.
[33] KOENDERINK J J. What does the occluding contour tell us about solid shape?[J]. Perception, 1984, 13(3):321-330.
[34] ZHANG Liangpei, ZHANG Yun, CHEN Zhenzhong, et al. Splitting and merging based multi-model fitting for point cloud segmentation[J]. Journal of Geodesy and Geoinformation Science, 2019, 2(2):78-89.