Sonata: Advancing Self-Supervised Learning for 3D Point Representations

June 4, 2025 · 5 min read

Key Takeaways

  • We developed a novel self-supervised learning (SSL) approach for point cloud representation learning.
  • Key to the strong performance is that we identify and show ways to overcome the “geometric shortcut”, also called “representation collapse”.
  • The strong point cloud representations learned by Sonata enable 0-shot correspondence, strong and efficient linear probing for semantic segmentation tripling performance from 21.8% to 72.5% on ScanNet, and state of the art performance across indoor and outdoor segmentation tasks when fully finetuned.
  • We open source our model, its weights and training code to facilitate open research.

We are excited to introduce Sonata, a groundbreaking approach to self-supervised learning (SSL) for point clouds. SSL is a powerful way of training machine learning models because it doesn’t require any ground truth labels. This enables a model to learn from a large variety and quantity of data during training. Most SSL frameworks are focused on learning from text, images or video. Sonata focuses on 3D pointcloud data from sources such as RGB-D sensors, Structure-from-Motion (SfM) point clouds like from Project Aria, and LIDAR scanners. Pointclouds can also be extracted from mono-depth or multi-view reconstructions systems based on Neural Radiance Fields, Gaussian Splats or end-to-end learned models like DUSt3R and VGGT to enable 3D reasoning directly from video.

3D pointclouds organize information in a fundamentally different way to images

Unlike images (left), point cloud color values are not organized in a regular 2D grid, but rather associated to x,y,z positions (top). The point coordinates alone (bottom), which are used as positional encodings to the model, contain valuable information about the scene, which can lead to representation collapse if not treated properly.

The principal difference between image and point cloud data is in the way the information (i.e. color values) is organized. In images, color values are organized in a consistent, structured 2D grid. For point clouds, there is no such regular uniform structure. Instead, the information is organized via the point coordinates themselves, whose local structure and statistics can vary widely depending on the 3D data source (e.g. RGB-D, SfM, LIDAR as described above). This means that a model that uses the point coordinates in its operations has access to additional information. We cannot easily mask out spatial information, as typical text/image/video SSL formulations would, since they are used as input positional encodings to the model. This leads to what we call the “geometric shortcut”, where the model can “cheat” by using low level statistics in the point cloud, rather than reasoning holistically about the entire scene using the color information as a good 3D representation should. As shown below, the “geometric shortcut” leads to poor representations in related work such as CSC and MSC that collapse to point height or surface normal information.

Sonata Self-Supervised Framework

The Sonata framework is built on a point-based self-distillation approach similar to that used in DinoV2, leveraging point clouds from 140k scenes. We introduce two innovative strategies to address the fundamental challenges of point clouds and the “geometric shortcut”:

  • Spatial Obscuration: By applying SSL losses at coarser spatial scales and disturbing the spatial information of masked features on points, Sonata reduces the model's dependence on easily accessible geometric cues and reduces the impact of the “geometric shortcut”.
  • Decoder-Free Architecture: Sonata moves beyond traditional U-Net structures used in point cloud SSL. This is similar to the patch-based output seen in 2D SSL techniques using Vision Transformers such as DinoV2. We focus SSL exclusively on the encoder and leave decoding to be trained on downstream tasks like semantic segmentation. To get full resolution output during pretraining (i.e. one feature for input point), we deterministically upcast the features from coarser point clouds in deeper layers of the encoder to the full resolution pointcloud and concatenate them. This approach allows for training more flexible and generalizable multi-scale representations.

Scalable 3D Point Cloud Machine Learning Architecture

Sonata uses a state-of-the-art 3D backbone called PointTransformerV3 (PTv3), which is capable of operating on large scenes. At the heart of this architecture is the transformer layer. Though it is common to use an image patch as the input for 2D images with architectures like ViT, it is less clear how to formulate tokens or 3D patches with unstructured 3D data. Thus we chose PTv3 as it uses an efficient serialization mechanism to build higher level tokens. This enables the model to scale to very large point clouds with thousands of points, while keeping the computation tractable.

Powerful Point Cloud Representation leads to Exceptional Parameter Efficiency & Data Efficiency

For the first time, the Sonata SSL framework enables learning point cloud representations strong enough to be zero-shot or linear probed for segmentations without finetuning the model. The ability to zero-shot probe or train just a tiny linear layer (with <0.2% of parameters of the original model) for segmenting a point cloud shows that the features output by the self-supervised pretrained model already contains information to allow distinguishing different classes of objects. While this is common practice in image and video SSL, in point cloud SSL representations learned by prior work lead to poor linear probing performance.

This is shown in the figure above for the PointContrast (PC) and Masked Scene Contrast (MSC) SSL approaches. As can seen, the Sonata pretrained model achieves a breakthrough 3.3x higher linear probing performance. This performance is close to the new full finetuning state-of-the-art of 79.4% set by the Sonata pretrained model.

State-of-the-Art Perception Performance With Full Finetuning

To unlock the full potential of the Sonata pretrained model we fully finetune all the model parameters. We find that the Sonata pointcloud model demonstrates exceptional performance across a wide range of 3D perception tasks. It achieves state-of-the-art performance on both indoor and outdoor semantic segmentation and instance segmentation tasks.

Open-Sourcing the high performance Sonata-pretrained 3D point cloud model

Sonata represents a significant leap forward in the field of 3D self-supervised learning. By identifying and addressing the geometric shortcut and introducing a flexible, efficient framework, it is an exceptionally robust 3D point representation. This work advances the state-of-the-art and sets the stage for future innovations in 3D perception and its applications. We release the training code and model weights on github and weights.

Paper Link https://arxiv.org/abs/2503.16429

Github Link https://github.com/facebookresearch/sonata

Weights Link https://huggingface.co/facebook/sonata

Page Link https://xywu.me/sonata/