Exploring AI in Collective Forecasting

Aug 29, 2024

Cognitive Revolution Podcast Episode

Host and Sponsorship

  • Hosts: Nathan Lens and Eric Torberg
  • Sponsored by Work OS, a platform providing enterprise features for B2B SaaS applications.

Guest: Dar Teron, CEO of Metaculus

  • Background: Interested in collective intelligence and decision-making enhancements through AI.
  • Previous work focused on federal agency regulatory feedback and AI alignment in sociotechnical contexts.

Forecasting and AI

  • Current State of Forecasting

    • Forecasting is underutilized in making better decisions despite potential benefits.
    • Challenges: Difficulty in scaling and ensuring forecasts are contextually relevant.
    • Metaculus is a leading forecasting platform not designed as a prediction market.
    • It aggregates forecasts using community predictions and individual track records.
  • AI in Forecasting

    • Study shows AI systems' forecasts are competitive with top human forecasters.
    • Importance of using AI for scalability and improving decision-making processes.

Metaculus and its Unique Approach

  • Platform Features

    • No monetary incentives; focuses on accuracy and collaborative forecasting.
    • Uses weighted averages for predictions; incentivizes sharing and transparency.
    • Offers financial incentives through hiring top forecasters for bespoke projects.
  • Vision for Forecasting

    • Move beyond accuracy to usefulness—how forecasts can inform real-world decisions.
    • Integration with conditional forecasting and policy development.
    • Open-sourcing Metaculus for broader experimentation and use.

AI Forecasting Benchmark Tournament

  • Goals and Structure
    • Year-long competition with 250-400 questions each quarter.
    • Encourages AI development in forecasting; $120k prize pool.
    • Participants create and maintain Bots, submitting forecasts daily.
    • Open for new ideas and fostering AI's role in decision-making.

AI and Human Collaboration

  • Potential of AI

    • AI systems can assist in creating interpretable world models.
    • Importance of balancing AI capabilities with human decision-making.
  • Challenges and Considerations

    • Ensuring AI models do not manipulate or obscure human understanding.
    • Addressing failure modes and emphasizing decentralization to prevent systemic risks.

Future Directions and Impact

  • Experimentation and Collaboration

    • Encouraging different institutions to use forecasting for strategic planning.
    • Focus on short-term proxies, resource allocation, and policy impacts.
  • Broader Vision

    • Integrating AI for better world models and enhanced civic engagement.
    • Emphasizing collective intelligence and avoiding consensus illusion.
    • Aiming for a transformative impact on societal decision-making.