Structured Outputs in the OpenAI API

Introduction

• Presenters: Atiyah Leti (API Design Lead) and Michelle Pocras (Tech Lead)
• Purpose: Discuss the new feature of structured outputs in the OpenAI API, launched in August.
• Relevance: A major improvement for developers working with large language models (LLMs).

Background

• 2020: Launch of GPT-3, useful for tasks like generating text for emails, blogs, and scripts.
• 2023: Launch of GPT-4, capable of advanced reasoning, complex instructions, and extracting information from documents.
• Challenge: Need for structured outputs (e.g., JSON) to connect LLMs to external systems and applications.

Why Structured Outputs?

• Problem: Models often output text surrounding JSON (not just JSON) which is not ideal.
• Solution Attempts: Function calling launched with JSON schema in June last year; JSON mode introduced in November for valid JSON outputs.
• Remaining Issues: Incorrect types and hallucinated parameters continue to be problematic.

Structured Outputs Feature

• Purpose: Ensures model-generated outputs match JSON schemas specified by developers.
• Modes:
  • Function Calling: Allows models to generate parameters for tool calls.
  • Response Format Parameter: For model responses directly to users.
• Enabling: Set strict to true to ensure outputs follow a given schema.

Use Cases and Examples

• Startup Example: AI glasses product using structured outputs for admin dashboard queries.
• Response Formats: Extract information from unstructured data, ensuring outputs follow a specified schema.
• Function Calling: Generate UI, and control application workflows reliably.

Demos

• Recruiting Tool (Convex):
  • Uses response formats to extract information from resumes.
  • Uses function calling to filter candidates and generate UI dynamically.
  • Multi-step workflows for scheduling interviews with reliable outputs.

Under the Hood

• Engineering Implementation:

  • Uses constrained decoding to ensure models follow developer-supplied schemas.
  • Components: LLM inference, token masking, JSON schema grammar support.

• Research Improvements:

  • Trained models to better understand and follow complex JSON schemas.
  • Overcame issues like the infinite newline problem seen in older models.

Design Decisions

• Additional Properties: Disallowed by default to prevent errors.
• Required Properties: All properties required by default to ensure completeness.
• Order of Properties: Maintained as specified for better context and conditioning.

Conclusion

• Impact: Structured Outputs solve critical issues for developers, unlocking AI application capabilities.
• Developer Success: Early adopters, like Shopify, have significantly improved application reliability.
• Mission Alignment: Structured outputs align with OpenAI's mission to build safe AGI, facilitating developer success globally.
• Gratitude: OpenAI expresses appreciation to developers for their innovation and contributions.