Efficiency of Facebook's Tectonic File System

Aug 24, 2024

Tectonic File System Efficiency

Overview

  • Tectonic is Facebook's exabyte-scale distributed file system.
  • Consolidates multiple large tenants from different service-specific systems into a generalized file system.
  • Aims for performance similar to specialized systems with lower operational overhead and better performance utilization.

Background

  • Facebook's previous storage systems included Haystack, F4, and their data warehouse using HDFS.
  • These systems were migrated into Tectonic.
  • Tectonic allows efficient resource sharing among cluster tenants.

Challenges Faced

  1. Scaling to Exabyte Scale:

    • Pre-existing solutions could not scale to Facebook's required size.
    • Needed to handle many different tenants and datasets.

  2. Tenant-Specific Optimization:

    • Different tenants have different load requirements (e.g., low latency for blob stores vs. high throughput for data warehouses).

  3. Performance Isolation Across Tenants:

    • Ensuring a tenant doesn't hog resources, affecting others' SLA.

High-Level Solutions

  1. Disaggregating Metadata:

    • Divides file system metadata into scalable layers (similar to Azure Data Lake).

  2. Rich Client-Driven Microservices Architecture:

    • Clients drive data manipulation, allowing tenant-specific optimizations.

  3. Traffic Groups for Performance Isolation:

    • Aggregates applications into groups with prioritized resource allocation.

Architecture

  • Client Library: Entry point for users.
  • Metadata Store:
    • Disaggregated into name, file, and block layers.
    • Uses ZippyDB for key-value storage.
  • Chunk Store:
    • Stores data as chunks, flat and oblivious to higher-level abstractions.
    • Uses XFS as the underlying file system.
  • Background Services: Handles tasks like rebalancing and garbage collection.

Multi-Tenancy and Access Control

  • Uses a fair scheduler to manage ephemeral and non-ephemeral resources.
  • Traffic Groups: Up to 50 per cluster, assigned a traffic class (bronze, silver, gold).
  • Access Control: Token-based authorization mechanism for efficient granular access.

Tenant-Specific Optimizations

  • Allows tenants control over configuration.
  • Examples of optimizations include hedge quorum writes and blob storage optimizations.

Production Trade-offs

  • Metadata Latency: Increased due to disaggregation and sharding.
  • Hash Partitioning: Creates issues with directory listing and utilization reporting.
  • Some services do not use Tectonic due to bootstrap requirements.

Related Works

  • HDFS & GFS: Single-node metadata architecture.
  • Azure Data Lake & HopsFS: Disaggregated and sharded metadata.
  • Colossus (Google): Client-driven design using Spanner.

Note: The content is based on a talk about Facebook's Tectonic file system and its various components and challenges.