In Sifflet, a monitor checks that the data in one or more tables matches a data-quality criterion. For example, a monitor can check that a table receives new rows every day. When the data does not match the criterion, the monitor fails and Sifflet can send a notification.

You can run a monitor manually or on a schedule, such as every day at 8am.

When you create a monitor, you choose:

A template: what the monitor checks. For example, the table receives new rows every day.
A threshold: when the monitor fails. For example, fewer than 1,000 new rows.
A scan mode: which rows the monitor reads at each run. For example, only yesterday's rows.
Notifications: who gets notified, and where. For example, the #data-engineering Slack channel.

On this page, all examples use an orders table that receives new orders from an e-commerce website every day.

Monitor Templates

A template defines what the monitor checks. When you create a monitor, you select a template, then the table (and, for some templates, the columns) to check.

For example, to check that the orders table receives new rows every day, select the Freshness template and apply it to orders.

Other templates check the values in a column, for example empty values, duplicates, or badly formatted emails. The SQL templates let you write your own check. The three most common templates work on any table:

Template	What it checks	Example on the `orders` table
Volume	The number of new rows, for example per day.	The table usually receives about 10,000 new rows per day. One day, it receives only 2,000 because one of the sales websites stopped sending its orders.
Freshness	Whether new rows arrive when expected.	The table usually receives new rows every morning before 6am. One day, no new rows arrive because the pipeline that loads them is stuck.
Schema Change	Whether the columns of the table change: added, removed, renamed, or with a new type.	An engineer renames the `customer_id` column to `client_id`. The dashboards that use `customer_id` stop working.

To see all templates, go to All Monitor Templates. For help choosing one, see the Monitor Template Selection Guidelines.

Thresholds

The threshold defines when the monitor fails. There are three types of thresholds:

Static: You set fixed limits. For example, the monitor fails if orders receives fewer than 1,000 new rows in a day.
Relative: You set the maximum change from one day (or hour, week, and so on) to the next. For example, the monitor fails if the number of new rows in orders goes up or down by more than 20% compared to the day before.
Dynamic: Sifflet learns from the past values of your data what is normal, and fails the monitor when a value is unusual. For example, if orders always receives more rows on weekends, Sifflet learns this. The monitor does not fail on a busy Saturday, but it does fail on a Saturday with very few orders.

Choose a static threshold when you know the exact rule, for example, a price can never be negative. Choose a dynamic threshold when you do not know what the limits should be, or when you monitor many tables and cannot set limits for each one.

Some templates do not need a threshold. For example, a Schema Change monitor fails as soon as a column changes.

For more information, see Thresholds.

Scan Modes

The scan mode decides which rows the monitor reads at each run:

Full dataset scan: The monitor reads the whole table every time. For example, a products table lists all the products you sell and has no date column: the monitor checks all products at each run.
Incremental scan: The monitor reads only the new rows since its last run, using a date column. For example, a daily monitor on orders reads only yesterday's orders, using the created_at column. The monitor computes one value per period, such as the number of orders per day, and keeps the history of these values.

For more information, see Scan Modes.

Notifications

When a monitor fails, Sifflet can send a notification to Slack, Microsoft Teams, or email, or create a ticket in a tool like Jira. First, connect these tools in Alert Destinations.

You can set up notifications in three ways:

On a monitor: Choose where one monitor sends its notifications. For example, when the Freshness monitor on orders fails, it posts a message in the #data-engineering Slack channel. See Notifications.
On incidents: Create a rule that covers many monitors at once. When one of these monitors fails, Sifflet opens an incident to track the problem and notifies the people you chose. For example, all Freshness and Volume monitors on the Sales team's tables notify the #finance-alerts Slack channel. See Notification Rules.
On a data product: A data product groups the tables behind a business use case, such as a sales dashboard. You can get a notification when its health changes, or receive a regular summary. For example, the owners of the "Sales Reporting" data product get a notification when it goes from Healthy to At Risk because the monitor on orders failed.

Monitor Status

Each monitor shows a status:

Passing: The monitor found no problems, or someone reviewed all the problems it found.
Failing: The monitor found at least one problem that nobody has reviewed yet.
Needs Attention: The last run did not work, for example because Sifflet does not have permission to read the table.
Not Evaluated: The monitor has never run.

Each run of a monitor also has a status:

🟢 Success: The run found no new problems.
🔴 Failed: The run found a new problem.
🟡 Needs Attention: The run did not work because of a problem on your side, such as missing permissions.
🟠 Technical Error: The run did not work because of a problem on Sifflet's side.

Next Steps

Create your first monitor.
Create monitors at scale to cover many tables at once.