Automatic Incident Grouping

When a data quality monitor fails, Sifflet minimizes alert fatigue and helps you focus on the root cause of a data problem. Instead of creating a new, isolated incident for every single failure, Sifflet analyzes your data lineage and existing open incidents. If Sifflet finds a relevant incident, it groups the new failure into it.

This page explains the logic behind this feature.

How It Works

When a monitor in Sifflet fails (the "New Failure"), Sifflet performs a multi-step analysis to determine if it should be grouped with an existing issue or if a new incident should be created.

1. Look for Recent Incidents: Sifflet searches the most recent open incidents that have had a new failure within the last 7 days.
2. Analyze Relationships: It then analyzes the new monitor failure to see if it is related to any of these recent incidents based on two key factors:
   • Semantic Match: It checks if the monitor template of the active incident is a compatible match for the monitor template of the new failure, based on their positional relationship (see Grouping Rules below). Not all failures in a lineage are related; for example, a Freshness delay often causes a Volume issue, but a Schema Change is likely unrelated to an Email Format error.
   • Data Lineage: Whether the monitors are on the same asset, or on assets connected via upstream, downstream, or sibling lineage, within a maximum lineage depth of 3 hops.
3. AI Validation (Relevance Check): Sifflet uses an AI model to validate the potential connection, analyzing the context of both the new failure and the existing incident to confirm if they are truly related.
4. Group or Create:
   • If Sifflet finds a strong relationship, it adds the new monitor failure to the existing incident.
   • If Sifflet finds no related incident, it creates a new incident.

Notifications when a failure is grouped

When a failure is grouped into an incident that already exists, an incident updated notification is sent to the incident's destinations instead of a new incident notification. See the Notification catalog.

Grouping Rules

Sifflet uses the following logic to determine if monitor failures are related. Grouping is heavily dependent on the monitor template of the new failure and its positional relationship to the monitors already on the existing incident:

• Same Asset — the new failure is on the exact same asset as an existing incident's monitor.
• Upstream Asset — upstream failures that could have caused this incident. Infrastructure failures (Volume, Freshness) are valid causes of almost any downstream failure. For SQL, only infrastructure upstream applies — a business rule failing upstream does not cause your business rule to fail.
• Downstream Asset — downstream failures that are likely effects of this incident. This is more conservative than upstream: business and format monitors downstream are almost always independent.
• Sibling Asset — sibling failures that are likely co-symptoms of the same root cause. Sifflet considers two assets siblings if they share a parent asset of the same type (Dataset), or if they are both root assets (i.e. neither has a parent asset). Infrastructure monitor types have the strongest sibling correlation; format validators and business logic monitors have near-zero sibling correlation.

A new monitor failure is grouped into an existing incident only if the existing incident contains a monitor of one of the compatible templates listed for the corresponding relationship (same asset, upstream, downstream, or sibling), within a maximum lineage depth of 3 hops.

AI-Generated Incident Descriptions

When incidents are automatically grouped or a new monitor is linked to an existing incident, Sifflet uses AI to generate a clear, human-readable description of the incident. This description summarizes the failures and provides context, helping you quickly understand the issue.