Auto-Coverage

Setting up a complex monitor can take up to a few minutes - which can be very time-consuming. Here's where Sifflet's Auto-coverage feature can save the day - it allows you to leverage the power of ML to generate a tailor-made set of monitors on multiple tables in a matter of just a few clicks.

Within the Catalog, use Sifflet's search engine to narrow down a selection of tables on which Auto-Coverage needs to be applied. Select the tables and click the Auto-Coverage button.

Use the toggle to include/exclude any suggested data quality monitoring logic. There are currently 5 templates:

• Volume if temporal: If a field has a time format, it will create a rule to verify that the volume of new ingested rows follows the expectations based on past behavior (ML-based rule).
• Freshness (Update time gap): It will create a rule that will fail if the duration since the last update deviates from historical norms (ML-based rule). This monitor is only supported for Snowflake and BigQuery tables (leveraging metadata information).
• Schema change: It will create a rule to detect any schema change
• Freshness if temporal: If a field has a time format, it will create a rule to verify that the data has been updated on the expected time
• Not null if field is id: If a field has an ID format, it will create a rule to verify that the field does not contain any null values

Once you click Scan, Sifflet will automatically identify the correct fields to monitor and suggest a list of rules to create.

You can use the toggle on the left to select the rules you want to be created, and use the dropdown lists on the right to define the severity of each rule. Sifflet will automatically detect those tests that have already been created.

📘

Tips

Freshness (update time gap) provides more accurate information if you schedule it to run @hourly. This monitor leverages Snowflake and BigQuery metadata tables and consequently as no associated costs.

To discover the way of creating customized rules, go to One-rule setup.

To understand how to finish configuring your rule, go to Time parameters.