RecolorNeRF: A Novel Easy-to-Use Color Editing Approach to the Neural Radiance Field

Neural Radiance Fields (NeRF) are a powerful representation of 3D scenes, making it possible for them to one day replace photos and movies as a new type of media. Supporting the edition of this new representation is essential to achieve this goal. Recent posts on the subject have explored NeRF editing in terms of geometric deformation, appearance editing, and style transfer, among other things. Recoloring is appearance editing that often involves adjusting certain color tones in a scene to enhance or correct it. This process is crucial to the process of making movies. In the example in Fig. 1, the red car can be changed to a blue one in a photorealistic way using a color change edit.

Palette-based color editing (PCE), one of the methods of changing the color of an image currently in use, offers the smoothest means of interaction. PCE involves these three essential steps: 1) Removing a paddle. The first step is to choose a representative color group and create a palette based on the landscape. 2) Decomposition of layers. They specify a matching image layer with a uniform color value for each item on the palette. The main objective of this stage is to choose the best method to combine these layers to recreate the original image. Color edition. By changing the color of each layer, the scene can be naturally recolored based on the two processes above.

Tan and colleagues presented a convex hull simplification technique for vane extraction as one of the PCE SOTA approaches for imaging. The layer decomposition is then expressed as an optimization problem using the acquired palette. The assumption of the scarcity of the combined weights makes the problem manageable. In this study, they presented an entirely new technique called RecolorNeRF which, to their knowledge, is the first attempt to use a fully learnable palette for layer decomposition in photorealistic PCE for NeRF rendering. Although PosterNeRF has experimented with palette-based NeRF color changing, the results could be more realistic as color adjustment can only be enabled after posterization. As is well known, multiview images are frequently used to reconstruct the NeRF of a scene.

Therefore, another possible way to perform NeRF PCE is to extract palettes from the pixels in all input images, following the method of , and then perform layer decomposition and color editing for each rendered view of the pretrained NeRF. . Although easy to implement, this strategy suffers from three main problems: first, changing color in this way becomes a post-processing of the NeRF representation, leading to high computational costs. Second, because each view is processed independently, the results need more view consistency. Third, palette extraction is accomplished using a heuristic method, which can cause palette color to be less representative and layer decomposition not clean enough, interfering with color manipulation. . His main suggestion is to improve the palette, layer combination weights, and volumetric radiation fields in a single framework to address the issues mentioned above. They then employ an “over” composition to deal with complex scenarios such as final image formulation. Specifically, the alpha combination of a collection of ordered layers, each corresponding to an alpha weight, is used to represent each pixel. Then, for each layer, they build a volumetric alpha field that, like the radiation field, can also be represented by an MLP. Note that different levels employ various MLPs. Therefore, they need to jointly optimize the MLPs for the density field and the MLPs for the mix weights.

As is well known, each of the early PCE systems performed pallet removal individually. The first attempt to optimize the palette is what they have offered. Two novel designs are presented to help with the joint optimization problem: 1) An innovative convex hull regularization is proposed to allow a limited color palette to faithfully represent the entire scene. 2) Spreads are often used in mix weights to make the palette color more realistic. The scarcity constraint is given a unique order-aware weighting mechanism to improve the ability to simulate complicated situations. RecolorNeRF can create photorealistic images with adjustable color schemes and robustly deconstruct implicit rendering, based on experiments. To their knowledge, they are the first to consider co-optimizing the palette and alpha mix weights, for which a novel convex hull regulation was designed to be solvable. The entire RecolorNeRF framework is carefully designed, allowing color editing of the NeRF rendering to be done photorealistically using a fully learnable palette for layer decomposition. The code has not yet been published, but a video demo can be found on the project website.

review the Paper Y Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our reddit page, discord channel, Y electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.