A Breakthrough World Foundation Model for Controllable Minecraft Environments
- Innovative Model Introduction: Matrix-Game is a cutting-edge interactive world foundation model with over 17 billion parameters, designed for controllable game world generation in Minecraft, leveraging a two-stage training pipeline for unparalleled environment understanding and action-driven video creation.
- Comprehensive Dataset and Benchmark: Supported by the expansive Matrix-Game-MC dataset with over 3,700 hours of gameplay footage and the GameWorld Score benchmark, this model sets a new standard for evaluating visual quality, controllability, and physical consistency in game world models.
- Superior Performance and Future Potential: Matrix-Game outperforms existing models like Oasis and MineWorld, excels in diverse scenarios with precise keyboard and mouse control, and paves the way for future advancements in long-term coherence and action space enrichment.
Welcome to the future of interactive gaming with Matrix-Game, a groundbreaking world foundation model that redefines how we create and interact with virtual environments. Specifically tailored for Minecraft-style open-ended game worlds, this model introduces a new era of controllable video generation where players and developers can dictate character actions and camera movements with astonishing precision. With over 17 billion parameters, Matrix-Game isn’t just a tool; it’s a revolution in game design, offering high visual quality and temporal coherence that keeps the virtual world feeling alive and responsive.
At the heart of Matrix-Game’s development is a robust two-stage training pipeline. The first stage involves large-scale unlabeled pretraining to deeply understand game environments, while the second focuses on action-labeled training for interactive video generation. To fuel this process, we’ve curated the Matrix-Game-MC dataset, a treasure trove of over 2,700 hours of unlabeled gameplay clips and more than 1,000 hours of meticulously labeled footage featuring fine-grained keyboard and mouse action annotations. This dataset empowers Matrix-Game to adopt an image-to-world generation paradigm, using a single reference image as the foundation for crafting entire worlds and generating videos that are both visually stunning and highly controllable.
One of the standout features of Matrix-Game is its ability to handle a wide array of user inputs. Whether it’s simple keyboard commands like moving forward, backward, left, or right, or more complex actions like jumping and attacking, the model translates these instructions into seamless, high-quality video outputs. Beyond keyboard control, Matrix-Game excels in fine-grained mouse control, allowing for precise camera viewpoint shifts across upward, downward, leftward, rightward, and even diagonal perspectives. This level of detail extends to compound and dynamically changing actions, ensuring that even the most intricate user instructions are executed with accuracy during a single video generation process.
What truly sets Matrix-Game apart is its performance across diverse Minecraft environments. The model demonstrates remarkable generalization capabilities, adapting to eight distinct scenarios with varying terrain and interaction dynamics. Whether navigating a dense forest or scaling a rugged mountain, Matrix-Game maintains consistency and realism. For long-duration video generation, it employs an autoregressive strategy that preserves local temporal consistency across segments, enabling coherent dynamics over extended time horizons. This means players can immerse themselves in lengthy gameplay sequences without jarring transitions or loss of quality.
To rigorously assess and compare Minecraft world models, we’ve introduced the GameWorld Score, a unified benchmark that evaluates performance across eight critical dimensions. This benchmark doesn’t just measure the perceptual quality of generated videos; it also scrutinizes temporal quality, action controllability, and adherence to physical rules. Through extensive experiments, Matrix-Game has consistently outperformed prior open-source models like Oasis and MineWorld across all metrics, showing particularly strong gains in controllability and physical consistency. Double-blind human evaluations further validate these results, with participants overwhelmingly recognizing Matrix-Game’s ability to produce perceptually realistic and precisely controllable videos in a variety of game scenarios.
Looking ahead, while Matrix-Game achieves impressive results, it’s not without its challenges. In visually rare or structurally complex environments, the model occasionally struggles with generalization and physical rule understanding, largely due to limited training coverage. Addressing these gaps will require expanded data collection and targeted scenario enrichment. We’re committed to continual training to refine these aspects. Additionally, we’ve identified key areas for future exploration, such as enhancing long-term temporal consistency. Maintaining coherence over extended video sequences is a persistent challenge, and we plan to innovate our model architecture by integrating longer motion contexts or memory-based mechanisms to tackle this issue.
Another exciting direction is action space enrichment. Currently, Matrix-Game supports six types of keyboard actions and a limited range of mouse control values. However, real-world Minecraft environments demand a richer, more nuanced interaction spectrum. Our goal is to expand the keyboard action repertoire and enable a broader, more continuous range of mouse control values, enhancing the precision and expressiveness of user interactions. By releasing both the model weights and the GameWorld Score benchmark toolkit to the community, we aim to inspire and accelerate future research in interactive world generation.
Matrix-Game isn’t just a technological advancement; it’s a gateway to endless creative possibilities in gaming. By blending cutting-edge AI with the boundless imagination of players, this model invites us to reimagine how virtual worlds are built and experienced. As we continue to push the boundaries of what’s possible, Matrix-Game stands as a testament to the power of innovation in crafting interactive, immersive, and truly dynamic game environments.