Auto-Coverage

Setting up a complex monitor can take a few minutes. Sifflet's Auto-Coverage feature uses machine learning (ML) to generate a tailored set of monitors on multiple tables in a few clicks.

Within the Catalog, use Sifflet's search engine to narrow down the tables to which you want to apply Auto-Coverage. Select the tables and click the Auto-Coverage button.

Use the toggle to include or exclude any suggested data quality monitoring logic. There are currently 5 templates:

• Volume if temporal: If a field has a time format, Auto-Coverage creates a rule to verify that the volume of newly ingested rows follows the expectations based on past behavior (ML-based rule).
• Freshness (Update Time Gap): Creates a rule that fails if the duration since the last update deviates from historical norms (ML-based rule). This monitor is only supported for Snowflake, BigQuery, Databricks, Oracle, and MySQL tables (leveraging metadata information).
• Schema change: Creates a rule to detect any schema change.
• Freshness if temporal: If a field has a time format, Auto-Coverage creates a rule to verify that the data has been updated at the expected time.
• Not null if field is id: If a field has an ID format, Auto-Coverage creates a rule to verify that the field does not contain any null values.

Once you click Scan, Sifflet automatically identifies the correct fields to monitor and suggests a list of rules to create.

You can use the toggle on the left to select the rules you want to create, and use the dropdown lists on the right to define the severity of each rule. Sifflet automatically detects rules that already exist.

📘

Tips

Freshness (Update Time Gap) provides more accurate information if you schedule it to run @hourly. This monitor reads table metadata instead of scanning your data, so its query cost is minimal.

To create custom rules, see Monitor Setup.

To finish configuring your rule, see Time Parameters.