Automatic Incident Grouping

To reduce alert fatigue and help you focus on the root cause of a data problem, Sifflet automatically groups related monitor failures into a single incident. This page explains the logic behind this powerful feature.

How It Works

When a monitor in Sifflet fails, it doesn't automatically create a new incident. Instead, Sifflet performs the following steps:

1. Look for Recent Incidents: Sifflet searches for any open incidents that have had a new failure within the last 7 days.
2. Analyze Relationships: It then analyzes the new monitor failure to see if it's related to any of these recent incidents based on two key factors:
   • Grouping Rules: A set of logic that determines which types of monitors are likely related (detailed below).
   • Data Lineage: Whether the monitors are on the same data asset or on assets that are connected upstream or downstream.
3. AI Validation: Sifflet uses an AI model to validate the potential connection, analyzing the context of both the new failure and the existing incident to confirm if they seem related.
4. Group or Create:
   • If a strong relationship is found, the new monitor failure is added to the existing incident.
   • If no related incident is found, a new incident is created.

Grouping Rules

Sifflet uses the following logic to determine if monitor failures are related. Grouping is heavily dependent on whether the monitors share the same table/column or are connected via table lineage or column lineage.

Data Profile Monitors

• Freshness & Volume: Freshness monitors can be grouped with other Freshness or Volume monitor failures on the same table or connected tables.
• Schema Change: Schema Change monitor failures can be grouped with each other. They can also be grouped with Technical Error monitors on downstream assets.
• Duplicates: Volume and Line Duplicates monitor failures can be grouped together.

Field Profiling Monitors

These monitors are typically grouped with failures of the same type on the same column or on columns connected by lineage.

• Field Duplicates / Uniqueness
• Field Nulls
• Format Validation (e.g., is email, is UUID, is Regex)
• Value Range / Value List

Metrics Monitors

• Metrics: Metrics monitors (e.g., MIN, MAX, AVG) can be grouped with other Metrics monitor failures on the same column or connected columns.
• A SUM metric failure can also be grouped with Volume or Freshness failures on the same table.
• Distribution monitor failures can be grouped with Metrics failures.

Custom & SQL Monitors

• Monitors based on custom SQL or conditions are generally grouped with other failures from custom monitors, especially when they share table lineage.

AI-Generated Incident Descriptions

When incidents are automatically grouped or a new monitor is linked to an existing incident, Sifflet uses AI to generate a clear, human-readable description of the incident. This description summarizes the failures and provides context, helping you quickly understand the issue.