Kafka Metrics Monitoring

Introduction

• Importance of Kafka metrics monitoring
• Huge variety of Kafka metrics can be overwhelming

Key Metrics to Monitor

1. Number of Active Controllers

   • Should always be 1.
   • One broker acts as the active controller.
   • Manages leader election, assignments, etc.
   • Zero indicates cluster not working.
   • More than one implies a significant bug.

2. Number of Under Replicated Partitions (URP)

   • Should always be 0.
   • Indicates partitions are lagging in replication.
   • High write loads and slow replication can cause under-replication.
   • Be cautious; may indicate overloaded brokers or network issues.

3. Number of Offline Partitions

   • Should be 0.
   • Offline partitions mean the topic is partially down.

Importance of Monitoring and Alerts

• Monitor these metrics actively and set alerts.
• Alerts help in identifying issues quickly (e.g., via email notifications).
• More metrics monitored can simplify troubleshooting.

Documentation Reference

• Kafka documentation contains extensive metrics information.
• Key section: 6.6 Monitoring.
  • Explains how Kafka uses Yammer metrics exposed by JMX.
• Metrics include error rates, request rates, bytes in/out, etc.
• Set a normal value for metrics to know when something’s wrong.

Examples of Metrics

• Under Replicated Partitions:
  • Use Prometheus to check metrics at server and partition levels.
  • Two metrics available: one at server level and one at partition level.

Additional Resources

• Explore Kafka monitoring documentation for comprehensive metrics list.
• Breakdown of metrics:
  • Broker Metrics:
    • Under replicated partitions, offline partition counts, active controller count, etc.
  • Consider both producer and consumer metrics as well.

Conclusion

• Next lecture will cover adding these metrics to Grafana.
• Explore documentation further to identify additional metrics relevant to your needs.