Autonomous driving requires forecasting both geometry and semantics over time to effectively reason about future environment states. Existing vision-based occupancy forecasting methods focus on motion-related categories such as static and dynamic objects, while semantic information remains largely absent. Recent semantic occupancy forecasting approaches address this gap but rely on past occupancy predictions obtained from separate networks. This makes current methods sensitive to error accumulation and prevents learning spatio-temporal features directly from images. In this work, we present ForecastOcc, the first framework for vision-based semantic occupancy forecasting that jointly predicts future occupancy states and semantic categories. Our framework yields semantic occupancy forecasts for multiple horizons directly from past camera images, without relying on externally estimated maps. We evaluate ForecastOcc in two complementary settings: multi-view forecasting on the Occ3D-nuScenes dataset and monocular forecasting on SemanticKITTI, where we establish the first benchmark for this task. We introduce the first baselines by adapting two 2D forecasting modules within our framework. Importantly, we propose a novel architecture that incorporates a temporal cross-attention forecasting module, a 2D-to-3D view transformer, a 3D encoder for occupancy prediction, and a semantic occupancy head for voxel-level forecasts across multiple horizons. Extensive experiments on both datasets show that ForecastOcc consistently outperforms baselines, yielding semantically rich, future-aware predictions that capture scene dynamics and semantics critical for autonomous driving.

We propose a feature-centric framework for vision-based 3D semantic occupancy forecasting. Instead of directly predicting future occupancy in 3D space, our approach first anticipates how the scene will evolve in the image domain. Given a sequence of multi-view images from past and current time steps, an image encoder extracts spatial features, which are then processed by a forecasting module to synthesize future feature representations at multiple prediction horizons. These future-aware features are subsequently lifted into a 3D voxel grid using a standard Lift-Splat-Shoot view transformation and refined by a spatio-temporal occupancy encoder to produce voxel-level semantic predictions. This design allows our method to remain fully compatible with existing vision-to-3D pipelines while enabling accurate future scene understanding.

The key component of our framework is a transformer-based future state synthesizer that models temporal evolution directly in feature space. To provide explicit context, the input features are enriched with learnable embeddings that encode camera identity, temporal position, and feature scale. Starting from the current frame, a set of future state queries iteratively interacts with past observations through cross-attention, progressively constructing a representation of the anticipated scene. During training, the synthesized features are explicitly aligned with features extracted from actual future images using a feature-level alignment loss. By learning to predict future visual representations rather than task outputs directly, the model captures scene dynamics more effectively and enables robust long-horizon semantic occupancy forecasting.

Riya Mohan, Juana Valeria Hurtado, Rohit Mohan, Abhinav Valada

@article{mohan2025forecastocc,
  title={ForecastOcc: Vision-based Semantic Occupancy Forecasting},
  shorttile={ForecastOcc},
  author={Mohan, Riya, Hurtado, Juana Valeria and Mohan, Rohit and Valada, Abhinav},
  journal={arXiv preprint arXiv:2602.08006},
  year={2025}
}