Templates Library

Monitors can be set both on a table or a field level in order to control specific Data Quality criteria. They can be run manually or automatically based on a Schedule, and they can alert users in case of a breach (see: Notifications Setup).

Below, you'll find a Library of predefined Sifflet Monitor Templates.

Type	Monitor	Application	Description
Table-level Health	Volume (dynamic)	Table	Counts newly ingested rows and compares them to expectations based on past behavior.
Table-level Health	Volume (static)	Table	Counts newly ingested rows and compares them to expectations based on a set threshold (absolute or relative).
Table-level Health	Row-level Duplicates	Table	Computes the duplication rate [%] on a row-level and compares it to the expected value based on past behavior.
Table-level Health	Freshness	Table	Verifies whether new rows have been ingested into your table following the expected pattern.
Table-level Health	Schema Change	Table	Detects any change to the schema: new field(s), removed field(s), existing field(s) with updated types or names.
Metrics	Standard Deviation (static thresholds)	Fields: Numeric	The monitor fails if a standard deviation of the field is outside of a given range.
Metrics	Unique Values Count (static thresholds)	Fields: All	The monitor fails if a number of unique values is outside of a given range.
Metrics	Average (static thresholds)	Fields: Numeric	The monitor fails if an average value of a given field is outside of a given range.
Metrics	Sum (static thresholds)	Fields: Numeric	The monitor fails if a sum of the field is outside of a given range.
Metrics	Values (static thresholds)	Fields: Numeric	The monitor fails if the chosen field has any values outside of a given range.
Metrics	Quantile (static thresholds)	Fields: Numeric	The monitor fails if a quantile of the field is outside of a given range.
Metrics	Variance (static thresholds)	Fields: Numeric	The monitor fails if a variance of the field is outside of a given range.
Smart Metrics	Metrics (dynamic thresholds)	Fields: Numeric	The monitor fails if the selected statistical transformation of the field behaves differently than it did in the past.
Smart Metrics	Custom Metrics (dynamic thresholds)	Table	The rule fails if the time series returned by the query behave differently than they did in the past
Smart Metrics	Correlated Metrics	Fields: Numeric	The monitor fails if defined metrics diverge significantly from each other.
Field profiling	Distribution Change	Fields: All	The monitor fails if the distribution of a given field has changed abnormally compared to a former given run.
Field profiling	Duplicates Count (static thresholds)	Fields: All	The monitor fails if the count of duplicate field values is superior to a given threshold.
Field profiling	Duplicates Count (dynamic thresholds)	Fields: All	The monitor fails if the count of duplicate field values is abnormal compared to expectations based on past behavior.
Field profiling	Duplicates Percentage (static thresholds)	Fields: All	The monitor fails if chosen field duplication rate [%] is superior to a given threshold.
Field profiling	Duplicates Percentage (dynamic thresholds)	Fields: All	The monitor fails if the % of duplicate values of the field is abnormal compared to expectations based on past behavior.
Field profiling	Low Cardinality	Fields: All	The monitor fails if: - The chosen field has several different values above a given threshold; - The different values of the field changed since the previous run (e.g. from ['A', 'B'] to ['A', 'C', 'D']).
Field profiling	Not after date	Fields: Timestamps, Dates	The monitor fails if the table has rows after a given date.
Field profiling	Not before date	Fields: Timestamps, Dates	The monitor fails if the table has rows before a given date.
Field profiling	Not in the list	Fields: String	The monitor fails if the chosen field has values that are not present in the given list.
Field profiling	Null Count (static thresholds)	Fields: All	The monitor fails if the chosen field has values that are empty/null.
Field profiling	Null Percentage (static thresholds)	Fields: All	The monitor fails if the null value rate of the selected field is superior to a given threshold.
Field profiling	Null Count (dynamic thresholds)	Fields: All	The monitor fails if the count of null values of the field is abnormal compared to expectations based on past behavior.
Field profiling	Null Percentage (dynamic thresholds)	Fields: All	The monitor fails if the % of null values of the field is abnormal compared to expectations based on past behavior.
Field profiling	Unique	Fields: All	The monitor fails if the chosen field has duplications.
Format validation	Is an email	Fields: String	The monitor fails if the chosen field contains at least one row that does not have an email format.
Format validation	Is a phone number	Fields: String	The monitor fails if the chosen field contains at least one row that does not have a phone number format. I.e. checking for 6 to 16 digits and will accept the field to contain the following characters +, -, (, )
Format validation	Is UUID	Fields: String	The monitor fails if the chosen field contains at least one row that does not have a UUID format.
Format validation	Matches regex	Fields: String	The rule fails if the selected field contains at least one row that does not match the format specified by the given regular expression.
Custom	SQL	Table	Advanced template to write custom monitors based on business specifics. The SQL query must describe a quality breach on one or more tables within the same data source.
Custom	Conditional monitors	Tables	As for SQL, this template allows to write custom monitors based on business use cases. With conditional statements, no SQL syntax is needed. The rule fails if values are found inside the filtering criteria set by conditional rules