It has never been easier to capture a realistic digital representation of a real-world 3D scene, thanks to the development of effective 3D neural reconstruction techniques. The steps are simple:
- Take multiple photos of a scene from multiple angles.
- Re-create the camera settings.
- Use the prepared images to enhance a neural radiation field.
They anticipate that because it is so easy to use, recorded 3D content will progressively replace manually generated components. While the pipelines for turning an actual scene into a 3D representation are fairly established and readily available, many of the additional tools needed to develop 3D assets, such as those needed to edit 3D scenes, are still in their infancy.
Traditionally, manually sculpting, extruding, and re-texturing an element required specialized tools and years of skill in modifying 3D models. This process is significantly more complicated since neural representations frequently need explicit surfaces. This reinforces the need for 3D editing methods built for the contemporary era of 3D rendering, especially methods that are as accessible as capture methods. To do this, the UC Berkeley researchers provide Instruct-NeRF2NeRF, a technique for modifying 3D NeRF scenarios that require written input instructions. His method is based on a 3D scene that has already been recorded and ensures that any adjustments made as a result are 3D consistent.
They can allow for a variety of changes, for example, using flexible and expressive language instructions such as “Give him a cowboy hat” or “Make him turn into Albert Einstein” given a 3D scene capture of a person like the one in the picture. Figure 1 (left). ). His method makes modifying 3D scenes simple and accessible to regular users. Although 3D generative models are available, more data sources must be needed to train them effectively. Therefore, instead of a 3D diffusion model, they use a 2D diffusion model to extract antecedents for shape and appearance. They specifically use the instruction-based 2D image editing capability offered by the newly developed image-conditioned diffusion model InstructPix2Pix.
Unfortunately, using this model on specific photos generated with reconstructed NeRF results in uneven changes for different angles. They develop a simple technique to address this comparable to current 3D rendering systems like DreamFusion. By toggling between altering the “dataset” of the NeRF input photos and updating the underlying 3D rendering to include the modified images, their underlying technique, which they call Iterative Dataset Updating (Iterative DU), is somewhat that they refer
They test their technique on a variety of NeRF scenes that have been collected, verifying their design decisions through comparisons with ablated versions of their methodology and naive implementations of scoring distillation (SDS) sampling loss suggested in DreamFusion. They qualitatively contrast their strategy with an ongoing text-based styling strategy. They show that various modifications can be made to people, objects, and expansive environments using their technology.
review the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 16k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.
Aneesh Tickoo is a consulting intern at MarktechPost. She is currently pursuing her bachelor’s degree in Information Science and Artificial Intelligence at the Indian Institute of Technology (IIT), Bhilai. She spends most of her time working on projects aimed at harnessing the power of machine learning. Her research interest is image processing and she is passionate about creating solutions around her. She loves connecting with people and collaborating on interesting projects.
