Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory

Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation.

Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified parameter-free geometry-aware memory framework that integrates Patch Memory and Warped PRoPE. Patch Memory lifts historical observations into 3D patch representations for geometry-aware retrieval, while Warped PRoPE injects camera projection geometry into spatiotemporal rotary position embeddings, unifying long-term memory and camera control without modifying the backbone architecture. Second, we introduce a disentangled world representation that separately models static scene geometry and dynamic subjects through static Patch Memory and lightweight multi-view Reference Tokens, jointly preserving geometric consistency and subject identity throughout long-horizon generation. Third, we develop a long-horizon real-time distillation framework that progressively transfers a bidirectional diffusion model into a few-step causal generator through Flow Matching and curriculum-based Self-Rollout DMD, enabling minute-long real-time interactive generation.

Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and Internet videos, Matrix-Game 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.

We build on PRoPE, which turns camera geometry into a relative position code: each latent frame is associated with a full world-to-image projection matrix P = lift(K)·W. Unlike native PRoPE, which allocates a separate subspace for the camera, our layout tiles the camera projection across all head channels, on top of the full spatiotemporal RoPE including its frame axis. After the standard RoPE rotation, queries are multiplied by P and keys and values by P⁻¹, so a single softmax carries both the relative time t = i − j and the relative pose, adding no learnable parameters.

Mosaic Memory takes an intermediate granularity, using localized image patches as the unit of memory. For each target latent patch, the corresponding memory is retrieved directly from camera geometry — historical patches are back-projected into metric 3D from depth, intrinsics, and pose, then reprojected into the target view. Where several patches land on the same location, a z-buffer selection keeps only the surface closest to the target camera, while occluded, unreliable, or unobserved regions are left empty for the diffusion model to synthesize. Within the pose-aware sequence, each retrieved patch takes the RoPE timestamp of the target frame it supports and the spatial coordinate at which its content lands in the target view (Figure 2).

Pose-aware token sequence — **Figure 2:** The pose-aware token sequence. All token types share one timeline: each carries its own RoPE time and, for camera-bearing blocks, its own camera. Memory patches reuse the target camera and RoPE time on a parallel memory track, placed at projected spatial coordinates.

To preserve both scene-level stability and object-level consistency, the memory system decouples static and dynamic content. A motion-aware object filter retains only static regions in Mosaic Memory and masks moving ones before fusion, while the dynamic subject is represented by sequence-level reference tokens encoded from a few reference images and reinforced by a subject-region auxiliary loss — preserving identity while avoiding ghosting and duplication.

To turn the bidirectional video DiT into a real-time interactive world model, we reformulate generation as chunk-wise causal denoising and distill it in two stages. Causal adaptation first uses teacher-forced Perceptual Flow Matching — constraining the flow objective in a frozen perceptual feature space rather than regressing velocity in the VAE latent — to obtain a high-quality few-step causal initializer.

Self-Rollout DMD then removes teacher forcing and applies Distribution Matching Distillation on the student's own autoregressive trajectory, with patch memory and context retrieved online from the causally visible history. A condition curriculum progressively distills classifier-free guidance, camera control, and finally memory-conditioned generation, sharing only stable external conditions (initial memory, anchor, prompt, trajectory) between student and scorers.

Matrix-Game Series

Matrix-Game 3.0 — Real-time and streaming interactive world model with long-horizon memory
Matrix-Game 2.0 — Real-time, streaming interactive world model

Acknowledgements

We would like to express our gratitude to:

Diffusers for their excellent diffusion model framework
Wan2.2 for their strong base model
Self-Forcing for their excellent work on autoregressive generation
GameFactory for their idea of action control module
LightX2V for their efficient quantization and VAE distillation techniques

We are grateful to the broader research community for their open exploration and contributions to the field of interactive world generation.

Matrix-Game 3.5

Abstract

Method

01PRoPE Camera Control

02Memory Design · patch-level, static–dynamic decoupled

03Progressive Distillation

Quantitative Results

Qualitative Results

General Case

Specific Capabilities

Related Links

Citation