MIT researchers present a new computer vision system that turns any shiny object into a kind of camera: allowing an observer to see around corners or past obstructions

Valuable and often hidden information about the immediate environment can be obtained from the reflection of an object. By repurposing them as cameras, previously inconceivable imaging feats can be performed, such as looking through walls or up into the sky. This is challenging because several factors influence reflections, including object geometry, material qualities, the 3D environment, and the observer’s point of view. By internally deconstructing the object’s geometry and glow from the specular glow reflected from it, humans can gain depth and semantic clues about occluded parts in the surroundings.

Computer vision researchers at MIT and Rice have developed a method for using reflections to produce images of the real environment. Using reflections, they transform shiny objects into “cameras”, giving the impression that the user is looking at the world through the “lenses” of commonplace items like a ceramic coffee mug or metal paperweight.

The method used by the researchers consists of transforming shiny objects of indeterminate geometry into radiant field chambers. The main idea is to use the surface of the object as a digital sensor to record the reflected light from the surrounding environment in two dimensions.

The researchers demonstrate that novel view synthesis, the rendering of novel views that are only directly visible to the bright object in the scene but not to the observer, is made possible by recovery of radiation fields from the environment. Also, we can imagine occluders created by nearby objects in the scene using the radiation field. The method developed by the researchers is taught from start to finish using many photographs of the object to simultaneously estimate its geometry, diffuse radiation, and the radiation field of its 5D environment.

The research aims to separate the object from its reflections so that the object can “see” the world as if it were a camera and record its surroundings. Computer vision has had problems with reflections for some time because they are a distorted 2D representation of a 3D scene whose shape is unknown.

The researchers model the surface of the object as a virtual sensor, collecting the 2D projection of the radiation field from the 5D environment around the object to create a 3D representation of the world as you see it. Most of the surrounding radiation field is obscured, except through reflections from the object. Beyond the field of view, novel view synthesis, or the representation of novel views that are only directly visible to the bright object in the scene but not to the observer, is possible through the use of ambient radiation fields, which also They allow for the estimation of the depth and radiation of the object to its surroundings.

In summary, the team did the following:

• They demonstrate how implicit surfaces can be transformed into virtual sensors with the ability to capture 3D images of their environments using only virtual cones.
• Together, they calculate the 5D ambient radiation field of the object and estimate its diffuse radiation.
• They demonstrate how to use the light field of the surrounding environment to generate new viewpoints invisible to the human eye.

This project aims to reconstruct the surrounding 5D radiation field from many photographs of a bright element whose shape and albedo are unknown. The glare from reflective surfaces reveals elements of the scene outside the frame of view. Specifically, the surface normals and curvature of the shiny object determine how the observer’s images are mapped in the real world.

Researchers may need more precise information about the shape of the object or the reflected reality, which contributes to the distortion. It is also possible that the color and texture of the shiny object is mixed with the reflections. Also, it’s not easy to discern depth in mirrored scenes, since reflections are two-dimensional projections of a three-dimensional environment.

The team of researchers overcame these obstacles. They begin by photographing the shiny object from various angles, capturing a variety of reflections. Orca (Objects such as Radiance-Field Cameras) is the acronym for their three-stage process.

Orca can record multi-view reflections by generating images of the object from various angles, which are then used to estimate the depth between the shiny object and other objects in the scene and the shape of the shiny object itself. ORCa’s 5D radiation field model captures more information about the strength and direction of light rays coming from and hitting each point in the image. Orca can make more accurate depth estimates using data from this 5D radiation field. Because the scene is displayed as a 5D radiation field rather than a 2D image, the user can see details that corners or other obstacles would otherwise obscure. The researchers explain that once ORCa has collected the 5D radiation field, the user can place a virtual camera anywhere in the area and generate the synthetic image that the camera would produce. The user can also alter the appearance of an item, say from ceramic to metal, or add virtual things to the scene.

By expanding the definition of the radiation field beyond the traditional direct line-of-sight radiation field, researchers can open up new avenues of investigation into the environment and the objects within it. Using projected virtual views and depth, the work can open up possibilities in the insertion of virtual elements and 3D perception, such as the extrapolation of information from outside the camera’s field of view.

review the Paper and project page. Don’t forget to join our 22k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club