Why is Kafka Fast? The Secret Behind Kafka's Performance

Introduction

• Kafka's speed is often highlighted in terms of high throughput.
• The term "fast" can refer to:
  • Latency
  • Throughput
• Kafka is optimized to handle a large volume of data efficiently, much like a large pipe moving liquid.

Key Design Decisions Contributing to Kafka's Performance

1. Reliance on Sequential I.O.

• Sequential I.O. vs Random I.O.
  • Sequential I.O.: Faster as it involves reading/writing blocks of data in order.
  • Random I.O.: Slower due to the need for the disk arm to move to different locations.
• Kafka uses an append-only log as its primary data structure, which supports sequential access.
• Performance Metrics:
  • On modern hardware, sequential writes reach hundreds of megabytes per second.
  • Random writes are significantly slower, measured in hundreds of kilobytes per second.
• Cost Advantage:
  • Use of hot disks (cheaper and larger capacity than SSDs).
  • Allows Kafka to retain messages for longer periods cost-effectively.

2. Focus on Efficiency with Zero Copy

• Kafka transfers data from network to disk and vice versa with minimal copies.
• Zero Copy Principle:
  • Reduces excess copying during data transfer.
  • Utilizes a system call called sendfile for direct data transfer from OS cache to network interface card buffer.
  • Eliminates intermediate copies and system calls, enhancing efficiency.
  • Uses Direct Memory Access (DMA) for direct data transfer without CPU involvement.

Conclusion

• Sequential I.O. and Zero Copy Principle are critical to Kafka's high performance.
• Kafka employs additional techniques to optimize performance on modern hardware.
• For more insights, refer to the speaker's books and newsletter.