Creating Grafana Managed Alerts in Grafana 9

Navigation and Initial Setup

• Navigate to the Alerting Tab: Click the bell icon on the left-hand side.
• No Existing Alert Rules: Initially, no alert rules are created.
• Create New Alert Rule: Click the blue 'New Alert Rule' button.

Choosing Rule Type

• Types of Rules:
  • Grafana Managed Alert
  • Mimir or Loki Alert
  • Mimir or Loki Recording Rule
• Example Focus: Grafana Managed Alerts => Keep this option checked.

Example Scenario: HTTP 500 Internal Server Errors

• Objective: Create an alert for HTTP 500 Internal Server Errors.
• Data Source: Use Prometheus collecting metrics from the HTTP server.

Setting Up the Query

• Metric: http_request_total
• Filter: Status code = 500
• Use Rate Function: Since it's a counter, use rate function to observe the rate of 500 errors.
• Reduce to Single Value:
  • Options: Classic Condition or Reduce and Math Expression
  • Classic Condition: Simple functions (average, min, max, sum)
  • Reduce and Math Expression: More complex conditions, useful for multiple queries.
• Condition: If >= 1, just use Reduce Expression.

Configuring Alert Timing

• Evaluation Interval: Every 1 minute
• Alert Firing Duration: If condition persists for 5 minutes

Naming and Organizing the Alert

• Rule Name: Short but descriptive (e.g., HTTP Request Failures)
• Folder: Use or create a folder (e.g., HTTP Monitoring)
• Rule Group: Equivalent to Prometheus Rule Groups (use same name as folder)

Saving and Verifying the Alert

• Save Rule: After configuring, save the rule.
• Verification:
  • Alert Status: Initially 'OK'
  • Pending Status: Changes to 'Pending' if errors are detected (waits 5 minutes).
  • Firing Status: Changes to 'Firing' if errors persist for 5 minutes.
  • Return to OK: If condition resolves, returns to 'OK'.

[Music in Background]