Define and Deploy Your Data Monitoring Strategy

When you start deploying Sifflet on your data infrastructure, you need to think about which monitoring strategy to start with.

The answer always depends on your specific data organization and use cases, but this page gives you a few guidelines to get started.

1. Connect Your Sources to Sifflet

Before you start defining monitors on your data, you must first connect Sifflet to your data sources.

For monitoring, the most important sources are your data warehouses. Refer to the documentation page for your data warehouse to connect it to Sifflet (for example, Snowflake, BigQuery, Databricks or Redshift).

2. Define Your Monitoring Strategy

When you're getting started with data observability, there are two possible approaches.

If you're currently experiencing data incidents and want to start reducing their frequency without drowning in noise you can't address, use the strategy Start with the data products.

If you currently don't have a lot of data incidents and want immediate visibility on the entirety of your data stack, use the strategy Define freshness, volume and schema change monitors on everything.

a. Start with the Data Products

This strategy focuses on reducing the number of data incidents while keeping the alerting noise under control.

The idea is to start with a few monitors on critical assets, and once these monitors no longer report any issue, extend monitoring to another set of assets.

To deploy it:

Identify critical assets. Pick a few data assets (tables, dashboards, etc.) that are important to your business. These are the tables or dashboards that teams outside your data engineering teams use directly. The Data Catalog helps you find them.
Group them into data products. In Sifflet, create data products to encompass these assets.
Add a few critical monitors. For each table in these data products, set up one or two critical monitors, typically freshness and volume, with dynamic thresholds. Don't go overboard here: you'll add more monitors once these initial monitors stay green for a while.
Enable data product notifications. Configure data product notifications or notification rules scoped to your data products. Now, you have visibility when you have data issues on these assets.
Fix the underlying issues. Improve your data pipeline until you no longer regularly get alerts on these data products.
- To help investigate the root cause of issues, use lineage to find the upstream tables of these assets and add monitors on them. Again, freshness and volume monitors are a good start.
Expand the scope. Then, identify a few more data products and repeat the process.

b. Define Freshness, Volume and Schema Change Monitors on Everything

If you want a complete view of the state of your data quality, you can also start by defining common monitors on all your assets.

These common monitors are:

Freshness: Checks that your data is up to date.
Volume: Checks that the number of rows in the table matches your expectations.
Schema change: Detects changes to a table's schema, which can break its consumers.

You have several options to define all these monitors:

Automatic Monitoring (admins only): Monitor entire schemas or databases, including tables added in the future.
Auto-Coverage: Monitor a selection of tables from the Data Catalog.
Data-Quality-as-Code: Define monitors in code.

This is the noisiest strategy: with monitors on every table, you'll get failures on assets that nobody depends on. To keep it manageable, don't send notifications for these monitors at first. Instead, review their results in Sifflet, and only enable notifications (for example, with notification rules) on the assets that matter to your business.

Combine Both Strategies

The two strategies aren't mutually exclusive. You can define common monitors on everything to get visibility over your whole data stack, but only route notifications for your data products and their upstream tables. You get complete coverage, while your team only receives relevant alerts.

3. Additional Guidance

Use Dynamic Thresholds When You Have Many Monitors

When defining freshness and volume monitors at scale, consider dynamic thresholds over static ones.

Your data doesn't behave the same way every day: a table may receive more rows on weekdays than on weekends, grow steadily over time, or spike at the end of each month. A static threshold can't account for this. If it's too tight, it alerts on normal variations and creates noise; if it's too loose, it misses real incidents. And tuning static thresholds by hand for hundreds of tables isn't realistic.

Machine learning models trained on the history of each monitored metric compute dynamic thresholds. They learn the normal behavior of each table, including trends and seasonality, and only alert when the data deviates from it. This makes them a good fit when you deploy monitors at scale and don't know in advance what "normal" looks like for each table.

Keep static thresholds for known business rules, for example when a value must stay within a given range. See Monitor Setup for the available modes.

If a dynamic monitor is too noisy, you can tune it:

Sensitivity: Adjust it to only alert on significant deviations.
Feedback loop: Flag false positives so the model learns from them.

Mute Noisy Monitors

If your data team receives too many alerts, they'll start to ignore them. Fewer notifications that your team acts on are better than many notifications that it ignores.

If a monitor alerts too frequently, or if your team usually ignores its notifications, mute it until the underlying cause is fixed. In Sifflet, you can pause a monitor's notifications until its next status change: this way, you're notified again as soon as the situation evolves, and you won't forget to turn notifications back on.

Monitors that alert frequently can still carry useful information. Instead of deleting them, adjust their notification configuration, for example by excluding them from a notification rule or by removing their notification channels.

Configure Data Product Incident Notifications to Alert Business Teams

Business teams are not interested in the failure of individual monitors. However, they are interested in knowing when a given data product breaks.

Sifflet supports notifying a different team when an incident occurs on a data product:

Route your technical alerts (monitors, incidents) to your data teams.
Route data product alerts to both your business teams and data teams.