单张图像三维重建One-2-3-45: Any Single Image to 3D Mesh in 45Seconds without Per-Shape Optimization

Minghua Liu1∗Chao Xu2∗Haian Jin3,4∗Linghao Chen1,4∗Mukund Varma T5Zexiang Xu6Hao Su1。One-2-3-45: Any Single Image to 3D Mesh in 45

Seconds without Per-Shape Optimization，Project Website: http://one-2-3-45.com

Abstract

Single image 3D reconstruction is an important but challenging task that requires

extensive knowledge of our natural world. Many existing methods solve this

problem by optimizing a neural radiance field under the guidance of 2D diffusion

models but suffer from lengthy optimization time, 3D inconsistency results, and

poor geometry. In this work, we propose a novel method that takes a single image

of any object as input and generates a full 360-degree 3D textured mesh in a single

feed-forward pass. Given a single image, we first use a view-conditioned 2D

diffusion model, Zero123, to generate multi-view images for the input view, and

then aim to lift them up to 3D space. Since traditional reconstruction methods

struggle with inconsistent multi-view predictions, we build our 3D reconstruction

module upon an SDF-based generalizable neural surface reconstruction method

and propose several critical training strategies to enable the reconstruction of 360-

degree meshes. Without costly optimizations, our method reconstructs 3D shapes

in significantly less time than existing methods. Moreover, our method favors

better geometry, generates more 3D consistent results, and adheres more closely to

the input image. We evaluate our approach on both synthetic data and in-the-wild

images and demonstrate its superiority in terms of both mesh quality and runtime.

In addition, our approach can seamlessly support the text-to-3D task by integrating

with off-the-shelf text-to-image diffusion models.

1Introduction

Although these optimization-based methods have achieved impressive results on both text-to-3D [55,

25, 35]and image-to-3D tasks [43, 68], they face some common dilemmas: (a) time-consuming.Per-

shape optimization typically involves tens of thousands of iterations of full-image volume rendering

and prior model inferences, resulting in typically tens of minutes per shape. (b) memory intensive.

Since the full image is required for the 2D prior model, the volume rendering can be memory-intensive

when the image resolution goes up. (c) 3D inconsistent.Since the 2D prior model only sees a

single view at each iteration and tries to make every view look like the input, they often generate 3D

inconsistent shapes (e.g., with two faces, or the Janus problem [43, 55]). (d) poor geometry.Many

methods utilize the density field as the representation in volume rendering. It is common that they

produce good RGB renderings but extracting high-quality mesh tends to be difficult.

In this paper, instead of following the common optimization-based paradigm, we propose a novel

approach to utilize 2D prior models for 3D modeling. At the heart of our approach is the combi-

nation of a 2D diffusion model with a cost-volume-based 3D reconstruction technique, enabling

the reconstruction of a high-quality 360◦ textured mesh from a single image in a feed-forward pass

without per-scene optimization. Specifically, we leverage a recent 2D diffusion model, Zero123 [36],

which is fine-tuned on Stable Diffusion [64]to predict novel views of the input image given the

camera transformation. We utilize it to generate multi-view predictions of the input single image so that we can leverage multi-view 3D reconstruction techniques to obtain a 3D mesh. There are two challenges associated with reconstruction from synthesized multi-view predictions: (a) the inherent lack of perfect consistency within the multi-view predictions, which can lead to severe failures in optimization-based methods such as NeRF methods [48, 5]. (b) the camera pose of the input image is

required but unknown. To tackle them, we build our reconstruction module upon a cost volume-based

neural surface reconstruction approach, SparseNeuS [40], which is a variant of MVSNeRF [6].

Additionally, we introduce a series of essential training strategies that enable the reconstruction of

360-degree meshes from inherently inconsistent multi-view predictions. We also propose an elevation

estimation module that estimates the elevation of the input shape in Zero123’s canonical coordinate

system, which is used to compute the camera poses required by the reconstruction module.

By integrating the three modules of multi-view synthesis, elevation estimation, and 3D reconstruction,

our method can reconstruct 3D meshes of any object from a single image in a feed-forward manner.

Without costly optimizations, our method reconstructs 3D shapes in significantly less time, e.g., in just

45 seconds. Our method favors better geometry due to the use of SDF representations, and generates

more consistent 3D meshes, thanks to the camera-conditioned multi-view predictions. Moreover, our

reconstruction adheres more closely to the input image compared to existing methods. See Figure 1

for some of our example results. We evaluate our method on both synthetic data and real images and

demonstrate that our method outperforms existing methods in terms of both quality and efficiency.

3Method

Our overall pipeline is illustrated in Figure 2. In Section 3.1, we introduce a view-conditioned 2D

diffusion model, Zero123 [36], which is used to generate multi-view images. In Section 3.2, we

show that traditional NeRF-based and SDF-based methods fail to reconstruct high-quality meshes

from inconsistent multi-view predictions even given ground truth camera poses. Therefore, in

Section 3.3, we propose a cost volume-based neural surface reconstruction module that can be trained

to handle inconsistent multi-view predictions and reconstruct a 3D mesh in a single feed-forward pass.

Specifically, we build upon the SparseNeuS [40]and introduce several critical training strategies

to support 360◦ mesh reconstruction. Additionally, in Section 3.4, we demonstrate the necessity of

estimating the pose of the input view in Zero123’s canonical space for 3D reconstruction. While the

azimuth and radius can be arbitrarily specified, we propose a novel module that utilizes four nearby

views generated by Zero123 to estimate the elevation of the input view.

4Experiments