Introducing Shape Tokens, a continuous 3D representation that is compact and easy to integrate into machine learning models. Shape tokens serve as conditioning vectors and represent shape information within a 3D flow matching model. This flow matching model is trained to approximate probability density functions corresponding to delta functions concentrated on the surfaces of 3D shapes. By incorporating Shape Tokens into various machine learning models, we can generate new shapes, convert images to 3D, align 3D shapes with text and images, and render shapes directly at variable user-specified resolutions. Additionally, Shape Tokens enable systematic analysis of geometric properties, including normals, density, and deformation fields. Across all tasks and experiments, using Shape Tokens demonstrates solid performance compared to existing baselines.
Figure 1: Our Shape Tokens representation can be easily used as input/output for machine learning models in various applications, including 3D single image (left), neural representation of normal maps (top right), and alignment 3D-CLIP (bottom right). The resulting models achieve strong performance compared to baselines for individual tasks.Video 1: The video shows our unique image to 3D point cloud results. Images are invisible objects in the Objaverse test suite. Each video first shows the input image and then the generated point cloud. (Credits)Video 2: From the same invisible input image, we generate multiple point clouds independently. (Credits)
Figure 2: Overview of our architecture. (Left) We model a 3D shape as a probability density function that is concentrated at the surface, forming a 3D delta function. (Right) Our tokenizer uses cross-attention to add information about the sampled point cloud in the form ST. The velocity estimator only uses cross-attention and MLP to maintain independence between points.
Figure 3: Reconstruction, densification and normal estimation of invisible point clouds in a GSO dataset. For each row, we are given a point cloud containing 16,384 points (xyz only), we calculate ST and iid shows the resulting p(x|s) for 262,144 points. Different columns represent the input and point clouds sampled from different viewpoints. Indicated by the label in parentheses, we color the input points according to their xyz coordinates and the sampled points according to the uvw coordinates of their initial noise and their estimated normal (last two columns). Note that we do not provide normal as input to the shape tokenizer.Video 4: We compute shape tokens on input point clouds (16,384 points) from invisible Google scanned objects. We then sampled 16 times as many points (262,144 points). The video shows the uvw to xyz trajectory of the flow matching sampling process, that is, the ODE trajectory. We color the points with their initial position in the noise space (uvw). (Credits)
Figure 4: The ODE integration path defines a mapping from xyz (data) to uvw (noise).Video 3: The video shows recent methods of converting a single image to 3D on Google scanned objects, which are not seen in all methods.
From left to right:
input image
Splash Image (CVPR 2024) – Trained in Objaverse
Point-e (2022) – Trained on several million proprietary 3D meshes.
Make-a-shape (ICML 2024) – trained on 18 datasets including Objaverse
Ours: trained in Objaverse
Please note that this video is not intended to compare individual methods: these models differ in their training data (e.g. Point-e was trained on proprietary 3D meshes) and mechanisms (e.g. Splatter-image is not a generative model, our method assumes a known camera model). We provide the results for the viewer's reference. (Credits)
Video 5: The page shows the results of neural representation in invisible point clouds. Using Shape Tokens we use a neural network to independently estimate the intersection point of each ray and its surface normal.
