As the saying “a picture is worth a thousand words” suggests, adding images as a second modality to 3D production offers substantial advantages over text-only systems. Images mainly provide rich and detailed visual information that language can only partially describe or not at all. A picture, for example, may clearly and immediately express minor features such as textures, colors, and spatial connections, but a verbal description may need help to fully represent the same level of detail or use very lengthy explanations. Because the system can directly reference actual visual cues rather than interpreting written descriptions, which can vary widely in complexity and subjectivity, this visual specificity helps generate more accurate and detailed 3D models.
Additionally, users can explain their intended outcomes more simply and directly when using visuals, especially for people who find it difficult to express their visions in words. This multimodal method can serve a broader range of creative and practical applications, combining the contextual depth of text with the richness of visual data to provide a more reliable, user-friendly and effective 3D production process. While useful, using photographs as an alternative modality for developing 3D objects also presents several difficulties. Unlike text, images have many additional elements, such as color, texture, and spatial connections, making them more difficult to analyze and understand correctly using a single encoder like CLIP.
Additionally, considerable variation in light, shape, or object self-occlusion could result in view synthesis that could be more accurate and consistent, which can provide incomplete or confusing 3D models. Advanced and computationally demanding techniques are required to effectively decode visual information and ensure a consistent appearance across many views due to the complexity of image processing. Researchers have transformed images of 2D features into 3D models using various diffusion modeling methodologies, such as Zero123 and other recent efforts. A disadvantage of image-only systems is that, while the synthetic views look excellent, the reconstructed models sometimes need more geometric correction and intricate textures, especially when it comes to rear perspectives of the object. The main cause of this problem is the large geometric discrepancies between the produced or synthesized perspectives.
As a result, mismatched pixels are averaged into the final 3D model during reconstruction, resulting in blurred textures and rounded geometry. In essence, image-conditioned 3D generation is an optimization problem with more restrictive constraints compared to text-conditioned generation. Because a limited amount of 3D data is available, optimizing 3D models with accurate features becomes more difficult because the optimization process tends to deviate from the training distributions. For example, if the training data set contains a variety of horse styles, creating a horse from text descriptions alone can generate detailed models. However, the creation of textures in the novel view can easily differ from the taught layouts when an image specifies specific features, shapes, and textures of the fur.
To address these issues, ByteDance research team provides ImageDream in this work. The research team proposes a multi-level image controller that can be easily incorporated into the current architecture while considering canonical camera coordination across multiple object instances. In particular, according to canonical camera coordination, the produced image should represent the centered front view of the object while using the default camera settings (identity rotation and zero translation). This simplifies the process of translating differences in the input image into three dimensions. By providing hierarchical control, the multilevel controller streamlines the information transfer process by directing the diffusion model from the image input to each architectural block.
Compared to strictly text-driven models like MVDream, ImageDream excels at producing objects with the correct geometry from a given image, as seen in Fig. 1. This allows users to use well-developed image generation models to improve image-text alignment. In terms of geometry and texture quality, ImageDream outperforms current state-of-the-art zero-shot single image (SoTA) 3D model generators such as Magic123. ImageDream outperforms previous SoTA techniques, as demonstrated by its extensive evaluation in the experimental part, including quantitative evaluations and qualitative comparisons through user testing.
Review the Paper and Project. All credit for this research goes to the researchers of this project. Also, don't forget to join. our 33k+ ML SubReddit, 41k+ Facebook community, Discord Channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.
If you like our work, you'll love our newsletter.
Aneesh Tickoo is a consulting intern at MarktechPost. She is currently pursuing her bachelor's degree in Data Science and artificial intelligence at the Indian Institute of technology (IIT), Bhilai. She spends most of her time working on projects aimed at harnessing the power of machine learning. Her research interest is image processing and she is passionate about creating solutions around it. She loves connecting with people and collaborating on interesting projects.
<!– ai CONTENT END 2 –>
