Monitor Overview

In Sifflet, a Monitor is a check that evaluates the quality of one or more data assets.

Monitors can be set both on a table or a field level in order to control specific Data Quality criteria. They can be run manually or automatically based on a Schedule, and they can alert users in case of a breach (see: Notifications Setup).

Here is the list of Sifflet Monitor Templates:

Monitor templateCategoryApplicationDescription
Volume Table-level HealthTableCounts newly ingested rows and alerts on anomalous behaviour
Row-level DuplicatesTable-level HealthTableComputes the duplication rate [%] on a row-level and compares it to the expected value based on past behavior.
FreshnessTable-level HealthTableVerifies whether new rows have been ingested into your table following the expected pattern.
Freshness (Update time gap)Table-level HealthTableThe monitor fails when the duration since the last update deviates from historical norms.
Schema ChangeTable-level HealthTableDetects any change to the schema: new field(s), removed field(s), existing field(s) with updated types or names.
Metrics MetricsFields: NumericThe monitor detects changes in an aggregated metric of a field (i.e. Sum)
Custom Metrics MetricsTableThe rule fails if the time series returned by the query behave differently than they did in the past
Correlated Metrics MetricsFields: NumericThe monitor fails if defined metrics diverge significantly from each other.
Distribution ChangeField profilingFields: AllThe monitor fails if the distribution of a given field has changed abnormally compared to a former given run.
Duplicates Field profilingFields: AllThe monitor detects anomalies regarding the count of duplicates for a column or a set of columns
UniqueField profilingFields: AllA simplified version of the duplicates monitor that fails if a column or set of columns is not unique
Nulls Field profilingFields: AllThe monitor detects anomalies regarding the count of nulls/empties in a column or a set of columns
Value List ValidationField profilingFields: StringThe monitor fails if the chosen field has values that are not present in the given list.
Value RangeField ProfilingFields: NumericThe monitor fails if the chosen field has any values outside of a given range.
Referential IntegrityField profilingFields: AllThe monitor fails if values in one table are not present in the other table
Is an emailFormat validationFields: StringThe monitor fails if the chosen field contains at least one row that does not have an email format.
Is a phone numberFormat validationFields: StringThe monitor fails if the chosen field contains at least one row that does not have a phone number format.
I.e. checking for 6 to 16 digits and will accept the field to contain the following characters +, -, (, )
Is UUIDFormat validationFields: StringThe monitor fails if the chosen field contains at least one row that does not have a UUID format.
Matches regexFormat validationFields: StringThe rule fails if the selected field contains at least one row that does not match the format specified by the given regular expression.
SQLCustomTableAdvanced template to write custom monitors based on business specifics. The SQL query must describe a quality breach on one or more tables within the same data source.
Conditional monitorsCustomTablesAs for SQL, this template allows to write custom monitors based on business use cases. With conditional statements, no SQL syntax is needed. The rule fails if values are found inside the filtering criteria set by conditional rules