Introducing PointConvFormer, a novel component for point cloud-based deep network architectures. Inspired by generalization theory, PointConvFormer combines ideas of point convolution, where filter weights are only based on relative position, and transformers that use feature-based attention. In PointConvFormer, the attention calculated from the feature difference between the neighborhood points is used to modify the convolutional weights at each point. Thus, we preserve the invariances of the point convolution, while attention helps to select relevant points in the neighborhood for the convolution. PointConvFormer is suitable for multiple tasks that require detail at the point level, such as segmentation and scene flow estimation tasks. We experimented on both tasks with multiple data sets, including ScanNet, SemanticKitti, FlyingThings3D, and KITTI. Our results show that PointConvFormer offers a better precision/speed trade-off than classical convolutions, regular transformers, and voxelized sparse convolution approaches. The visualizations show that PointConvFormer works similarly to convolution on planar areas, while the neighborhood selection effect is strongest at object boundaries, proving it has the best of both worlds. The code will be available.
