Improving Restaurant Rankings for Deliveroo

Jul 12, 2024

Improving Restaurant Rankings for Deliveroo

Introduction

  • Speaker: Johnny, Data Scientist at Deliveroo
  • Focus: Ranking algorithms for restaurant recommendations
  • Goal: Explain Deliveroo's approach to ranking, modeling strategies, tool usage, and lessons learned.

Overview of Deliveroo

  • Platform that connects consumers to local restaurants through an app.
  • Fleet of riders pick up and deliver orders.
  • Three-sided marketplace: consumers, restaurants, and riders.
  • Operates in over 300 cities and 14 countries.

Problem Statement

  • Aim: Present the most relevant restaurants to users to increase engagement and order volume.
  • Metrics: Order volume and session-level conversion (proportion of sessions that result in orders).

Approach to Ranking

Framing the Problem

  • Objective: Rank restaurants optimally based on relevance to the user.
  • Metrics: Order volume and session-level conversion.
  • Classification framework: Predicting the probability of the user ordering from a restaurant.

Target Variables and Features

  • Target variable: Whether a user ordered from a restaurant (0 or 1).
  • Features: Popularity, estimated time of arrival, restaurant rating, etc.
  • Start simple: Initial model was a heuristic, mixing popularity and ETA ranks.
  • Iteration: From simplistic heuristic models to logistic regression and neural networks.

Evaluation

  • Initial Evaluation: Use metrics like accuracy or precision.
  • Better Metric: Mean Reciprocal Rank (MRR) to measure overall list quality.
  • MRR Calculation: Average of reciprocal ranks of ordered positions across sessions.

Model and Infrastructure

Workflow

  1. Data Extraction: SQL queries to build training and test sets.
  2. Data Validation: Python scripts for data validation (nulls, duplicates, etc.).
  3. Model Training: Train various models and calculate mean reciprocal ranks.
  4. Deployment: CircleCI for continuous integration, Docker for containerization, AWS for serving models.

Productionization

  • Serialize model and integrate into production language (Go for Deliveroo).
  • Tensorflow chosen for model maturity, community support, and flexible API.
  • Future Considerations: Exploring Amazon SageMaker for better workflow.

Lessons Learned

  1. Check Consistency: Ensure training and production environments match.
  2. Log and Monitor Everything: Log all metrics to catch potential data issues early.
  3. Global Metrics Limitations: Global metrics may not reflect user experience; include detailed scatter plots and user experience checks.
  4. Evaluation Beyond MRR: Use other metrics like Precision at K; consider user behavior and feedback.
  5. Look at Predictions: Ensure that the predicted ranks make intuitive sense.

Conclusion

  • Continuous Improvement: Iterative process of testing and refining models.
  • Future Work: Better integration of features, more complex models, empirical user experience checks.
  • Opportunities: Expanding the use of algorithms beyond restaurant ranking.

Call to Action

  • Recruitment: Opportunities to join Deliveroo's team and work on innovative solutions.

[Applause & Q&A Session]