Honeycomb: Enhancing Observability with AWS

Introduction

• Honeycomb helps software engineers debug systems with Big Data
• Focus on providing observability, a sociotechnical capability
• Enables answering any system question quickly (within 10 seconds)

Key Concepts of Observability

• Requires comprehensive data collection and efficient querying
• Connects and threads data together into user requests
• Analyzes good and bad customer experiences to resolve issues rapidly

Product Differentiation

• Recognized leader in APM and observability
• Offers scale and speed
• Value in quick and instant results

Technical Implementation

Data Collection and Analysis

• Collects high-quality telemetry data via AWS services (RDS, CloudWatch, etc.)
• Uses Amazon Distro for OpenTelemetry for real user requests
• Data is pre-processed, uploaded to S3, and analyzed with AWS Lambda

Growth and Scaling

• Growth from 200,000 trace spans/sec (3 years ago) to 2.5 million trace spans/sec now
• 10x increase in data queries and questions
• Achieved scalability with only 50 engineers

Architecture

• Combination of stateful and stateless services
• Services written in Go, Java, Node.js
• Migrated to AWS Graviton technologies (Graviton2, Graviton3)

Stateless Services

• Managed by Amazon EKS
• EC2 C6G and C7G instances powered by Graviton2 and Graviton3
• Observed improvements in latency and throughput (10% with Graviton2, further 10-20% with Graviton3)
• Save money using EC2 spot instances

Data Storage

• Columnar data storage on EC2 M6gd instances
  • 2/3 reduction in tail latency when switching from I3 to M6gd
  • Use of S3 for data tiering
• Kafka streaming ingest on EC2 IM4gn instances powered by Nitro SSD

Query Retrieval

• Uses AWS Lambda for querying millions of files from S3
  • Tens of thousands of parallel workers for fast query results within 10 seconds
  • Combined Lambda and Graviton for about 40% improvement in price performance

Economic and Environmental Benefits

• 60% cost savings compared to 5th generation instances
• 60% reduction in emissions with Graviton2 and Graviton3

Conclusion

• Honeycomb leverages AWS to scale and save costs effectively
• Enables fast querying and great performance for users
• Interested users can learn more through additional resources provided