Data-Driven Product Development and Experimentation at Deliveroo

Introduction

• Speakers: Jonathan (Head of Product Analytics) and Jamie (Data Scientist on Rider Payments Team)
• Focus: Experimentation in data-driven decisions at Deliveroo

Background of the Food Delivery Space

• Traditional Delivery: Local pizzerias with phone orders
• Online Marketplaces: Menus on websites, but restaurants managed their deliveries
• Modern Delivery (Third Wave): Deliveroo’s innovations:
  • Delivery for well-known brands
  • Faster delivery times (some as short as 6 minutes)
  • Predictable delivery times
  • Immense growth: 200 cities, six investment rounds

Importance of Data at Deliveroo

• Core of Deliveroo’s operations
• Three main data science categories:
  1. Product Analytics: Represented by Jonathan and Jamie
  2. Algorithms: E.g., Dispatcher algorithm, covered by Anthony
  3. Business Intelligence: Developing reporting structure, metric relations
• Data-Driven Decisions: Company-wide focus on making decisions based on data

Experimentation in Product Development

• Purpose: Measure the effect of new features or changes
• Experiment Design:
  • Split user population into control and variant groups
  • Control group experiences no changes, variant group experiences the new feature
  • Measure specific design metrics
  • Compare results between control and variant to determine effect
  • Challenges include dealing with randomness, requiring large sample sizes

Experiment Challenges in Specific Contexts

• Network-level effects and aggregation (e.g., zone level in London)
• Difficulties with small sample sizes and high variance
• Example: Extending delivery radius for restaurants in a zone
• Stakeholders:
  • Restaurants: More orders
  • Customers: May experience delays
  • Riders: Mixed effects based on order distances and payments
  • Deliveroo: Potential need for more riders, balancing order volume and customer service
• Aggregation Level: Usually zone-level, sometimes city-level

Block Design Experimentation Method

• Addressing Small Sample Sizes:
  • Randomly assign zones into groups for alternating treatment conditions
  • Each zone experiences treatment (extended radius) and control on different days
  • Averaging results to reduce variance impacts
  • Use one-sample t-tests to evaluate significance
• General experimental design considerations:
  • Identify nuisance variables (e.g., zone density, seasonality)
  • Example: Extending delivery radius for Japanese restaurants

Implementation of Experimentation at Deliveroo

• Process:
  1. Design Phase: Choose experiment method, location, and metrics
  2. Simulation Phase: Monte Carlo simulations to test accuracy and sensitivity
  3. Implementation Phase: Run experiment, typically 4-6 weeks
  4. Analysis Phase: Measure effects, derive insights, make recommendations
• Objective: Guide business decisions with reliable data, especially for operational aspects with small sample sizes

Conclusion

• Data science is foundational at Deliveroo and will become even more crucial
• Deliveroo is actively hiring for data science roles
• Encouragement to speak with team members for more insights

Networking

• Opportunities to connect with Deliveroo team members present at the event