Sakana Labs AI Scientist Announcement

Overview

• Date of Announcement: Today
• Company: Sakana Labs, Tokyo-based AI startup
• Founders: Leon Jones and David Ha (former Google researchers)
• Funding: $30 million in seed funding
• Goal: Automate scientific research and open-ended discovery using AI.

AI Scientist Capabilities

• Functionality: Automates scientific research processes from ideation to paper writing.
• Key Features:
  • Brainstorms ideas and evaluates their novelty.
  • Edits codebase using automated code generation.
  • Runs experiments and gathers results (numerical data and visual summaries).
  • Crafts scientific reports and conducts peer reviews.

Process Steps

1. Idea Generation: Brainstorms ideas based on a starting template.
2. Experiment Execution: Executes proposed experiments, produces plots, and notes.
3. Writing Up: Generates a report in LaTeX style, finds relevant papers to cite.
4. Automated Peer Review: Evaluates generated papers for improvements and feedback.

Key Aspects

• Cost Effectiveness: Approximately $15 per paper, expected to decrease over time with increased efficiency.
• Democratization of Research: Potential to lower research costs to cents in the future.

Paper Quality

• Insights: Generated papers contained new ideas but overall quality was mixed.
  • Example Contributions:
    • Diffusion models for low-dimensional data showed significant improvements.
    • Novel dual expert denoising architecture for diffusion models.
• Quality Assessment: Comparable to early stage machine learning researchers.
  • Limitations: Lack of theoretical justification, limited experimental scope, occasional hallucinations, and inconsistent quality.

Future Implications

• Papers are more like preliminary proposals needing substantial human input for development.
• Model Used: Claude Sonet 3.5 showed the best results with some papers rated for acceptance at conferences.
• Anticipation for Improvement: Expectation that all frontier LLMs will continue to improve.

Prompt Engineering

• Examples of prompts used in research were outlined in the paper for those interested in exploring.

Limitations of AI Scientist

• Lack of vision capabilities (e.g., cannot fix visual issues or read plots).
• Issues with readable plots, optimal layouts, and comparing numerical data accurately.

Open Source Release

• Sakana Labs has open-sourced the AI scientist's code for public access and experimentation.

Conclusion

• The announcement marks a significant advancement in AI-driven scientific research, with potential for rapid development in the field.