SceneScript: an AI model and method to understand and describe 3D spaces

SceneScript is a method for representing and inferring scene geometry using an auto-regressive structured language model, and end-to-end learning.

What is it for?

SceneScript allows AR & AI devices to understand the geometry of physical spaces. Shown on footage captured by Aria glasses, SceneScript can take visual input and estimate scene elements, such as walls, doors, or windows. Shown on Meta Quest, scene elements predicted by SceneScript can be arbitrarily extended to include new architectural features, objects, and even object decompositions.

How does it work?

SceneScript jointly estimates room layouts and objects from visual data using end-to-end learning. End-to-end learning avoids the need for fragile ‘hard-coded’ rules.

SceneScript is provided visual information in the form of images or point cloud from an egocentric device.

SceneScript encodes the visual information into a latent representation, which describes the physical space.

SceneScript decodes the latent representation to a concise, parametric, and interpretable language, similar to CAD. A 3D interpreter can convert the language to a geometric representation of the physical space.

What makes it special?

  • Flexible and extensible model design

  • Trained in Simulation
    SceneScript is trained using Aria Synthetic Environments, a fully simulated dataset consisting of 100,000 unique interior environments, each simulated with the same camera characteristics of Project Aria. The Aria Synthetic Environments dataset was made available to academic researchers last year, along with a public research challenge, to accelerate open research in this area.

  • Adaptable to new scene representations
    Because SceneScript both represents and predicts scenes using pure language, the scene elements can be easily expanded by expanding the language used to describe the simulated data. Unlike traditional rule-based approaches for scene reconstruction, this means there is no need to train new 3D detectors for different objects or scene elements.

Enables LLMs to reason about physical spaces

SceneScript leverages the same method of next-token prediction as large-language models. This provides AI models the vocabulary needed to reason about physical spaces. This advancement could ultimately unlock the next-generation digital assistants, by providing the real-world context necessary to answer complex spatial queries.

Extending SceneScript to human-in-the-loop corrections

Since SceneScript shares the same next-token prediction as large-language models, it can leverage their prediction techniques to enhance its capabilities. SceneScript can be extended to correct individual elements based on a person’s input by infilling. This technique can now be applied to scene correction.

Starting with an initial layout predicted by our model in global prediction mode, our human-in-the-loop system allows a user to identify local errors within the layout. The user then invokes our model in local correction mode to refine these entities. The user repeats this process until they achieve the desired layout.

SceneScript model weights are now available for research. If you are a researcher in AI or ML research, access the SceneScript model weights here.

Frequently Asked Questions

  • How is privacy respected in the generation of SceneScript?
    Because SceneScript is trained in simulation, no real-world data was used for training the model. To ensure that SceneScript works as expected for real-world scenes, the model was validated in fully consented environments.

  • How big is the SceneScript model?
    The base pointcloud encoder and decoder each comprise approximately 35M parameters, resulting in a total of around 70M parameters.

  • How long does it take to train the model?
    The model is trained until convergence for about 200k iterations resulting in a total of ~3 days.

  • Is SceneScript used on Quest today?
    At time of release, SceneScript is not used on Quest, but is a research project from Reality Labs Research.

  • Can it handle outdoor scenes?
    The model has been trained exclusively on synthetic indoor scenes and hence an inference on outdoor scenes may result in unpredictable outputs.

  • What is the model structure?
    The model is coarsely segmented into an encoder and decoder. The encoder consists of a series of 3D sparse convolution blocks pooling a large point cloud to a small number of features. Subsequently, a transformer decoder autoregressively generates tokens by leveraging the encoder's features as context for cross-attention.

  • How long does the inference take?
    Using a vanilla, non-optimized transformer featured directly in PyTorch, decoding 256 tokens, equivalent to a medium-sized scene containing walls, doors, windows, and object bounding boxes, requires approximately 2-3 seconds.

  • Does SceneScript work on non-egocentric devices e.g. action cameras, mobile phones?
    No, not at the moment as the current model is trained using sequences that simulate what would be captured on Project Aria glasses. However, the model could be finetuned using a different camera model with different kinds of lenses.

  • Has the SceneScript model been made available to external researchers?
    Yes, the model weights for SceneScript were made available for academic researchers in September 2024.

Acknowledgements

Research authors:
Armen Avetisyan, Chris Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Edward Miller, Jakob Engel, Richard Newcombe, Vasileios Balntas.