Multi-view to Novel View: Synthesizing Novel Views with Self-Learned Confidence

Abstract

We address the task of multi-view novel view synthesis, where we are interested in synthesizing a target image with an arbitrary camera pose from given source images. We propose an end-to-end trainable framework that learns to exploit multiple viewpoints to synthesize a novel view without any 3D supervision. Specifically, our model consists of a flow prediction module and a pixel generation module to directly leverage information presented in source views as well as hallucinate missing pixels from statistical priors. To merge the predictions produced by the two modules given multi-view source images, we introduce a self-learned confidence aggregation mechanism. We evaluate our model on images rendered from 3D object models as well as real and synthesized scenes. We demonstrate that our model is able to achieve state-of-the-art results as well as progressively improve its predictions when more source images are available.

Model Overview

We propose an end-to-end trainable framework that learns to exploit multiple viewpoints to synthesize a novel view without any 3D supervision. Specifically, our model consists of a flow prediction module (flow predictor) and a pixel generation module (recurrent pixel generator) to directly leverage information presented in source views as well as hallucinate missing pixels from statistical priors. To merge the predictions produced by the two modules given multi-view source images, we introduce a self-learned confidence aggregation mechanism. An illustration of the proposed framework is as follows.

Results

ShapeNet Cars

ShapeNet Chairs

Scenes: KITTI and Synthia

Paper

@inproceedings{sun2018multiview,
  title = {Multi-view to Novel View: Synthesizing Novel Views with Self-Learned Confidence},
  author = {Sun, Shao-Hua and Huh, Minyoung and Liao, Yuan-Hong and Zhang, Ning and Lim, Joseph J},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
}

Related Work

Check out some other work in novel view synthesis:

• [L₁] Multi-view 3D Models from Single Images with a Convolutional Network in CVPR 2016
• [Appearance Flow] View Synthesis by Appearance Flow in ECCV 2016
• [TVSN] Transformation-Grounded Image Generation Network for Novel 3D View Synthesis in CVPR 2017
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis in NIPS 2015
• Neural scene Representation and Rendering in Science 2018
• DeepStereo: Learning to Predict New Views From the World's Imagery in CVPR 2016
• Learning-Based View Synthesis for Light Field Cameras in SIGGRAPH Asia 2016