Overview
The release of Grok 4 and Grok 4 Heavy by XAI marks a significant leap in AI capabilities, as the models outperform both human experts and previous leading AI systems across multiple benchmarks. The focus is now shifting from standard testing to evaluating AI performance in real-world tasks, signaling a new era in AI development and application.
Grok 4 & Grok 4 Heavy Performance Highlights
- Grok 4 is outperforming graduate-level humans in nearly all academic disciplines.
- Grok 4 scored 40% on “Humanity’s Last Exam,” a 2,500-question test designed to be unsolvable by humans.
- Grok 4 Heavy, leveraging a multi-agent system, reached 50% on the same exam.
- Both models outperform the current runner-up (Gemini 2.5 Pro) by a wide margin, especially with tool usage.
- Grok 4 Heavy scored 100% on the challenging AMI math benchmark, effectively solving it.
- On the ARC AGI 2 generalization benchmark, Grok 4 nearly doubles the score of the next best model.
Benchmarks & Reality as the New Test
- Traditional academic and logic benchmarks are being quickly saturated by state-of-the-art AI.
- XAI plans to move beyond written tests, using real-world tasks as benchmarks for future evaluation.
- This approach involves reinforcement learning tied to physics-based, real-world results (e.g., rocket launch success, working inventions).
- Grok 4 excelled in the “Vending Bench” simulation, significantly outperforming humans and other AI by autonomously running a vending machine business.
Grok 4 Heavy: Architecture and Pricing
- Grok 4 Heavy operates as a multi-agent system, spawning parallel expert “agents” that collaborate to generate high-quality answers.
- The system is more expensive ($300/month) due to increased computational complexity.
New Features: Grok Voice
- Grok Voice is now twice as fast, offers more natural interactions, and can perform tasks such as singing or whispering for comfort.
Future Roadmap for XAI and AI Applications
- August: Release of a dedicated coding model.
- September: Launch of a multimodal agent integrating vision, voice, tool use, and possibly memory.
- October: Introduction of a video generation model intended to rival top industry offerings.
- Planned developments include AI building full video games and producing watchable TV and movies potentially within a year.
Industry Position and Competitive Analysis
- XAI is now seen as a top-three AI lab, after Google and OpenAI, and ahead of Anthropic, with Meta also advancing.
- The rapid progress and ambitious roadmap position XAI at the frontier of AI innovation.
Decisions
- Shift AI Evaluation to Real-World Tasks: Move from traditional benchmarks to reality-based testing for AI capability assessment.
Action Items
- August – XAI Team: Release the dedicated coding model.
- September – XAI Team: Launch multimodal agent with vision, voice, and tool use.
- October – XAI Team: Release advanced video generation model.
Key Dates / Deadlines
- Humanity’s Last Exam benchmarks continuously updated as models advance.
- Major roadmap releases scheduled for August, September, and October this year.