Templates Library

A Monitor can be set on a table or a field level to check specific data quality criteria. Monitors can be scheduled to run based on a flexible scheduler and can alert the user in case of a breach on Slack or by email.

Find below the available Monitors with a brief description of what they do.

TypeMonitorApplicationDescription
MetadataCompleteness (ML-Based)TableCounts newly ingested rows and compares them to expectations based on past behavior.
MetadataDuplicates (ML-Based)TableComputes the duplication rate (%) on row-level and compares it to the expected value based on past behavior.
MetadataFreshnessTableVerifies whether new rows have been ingested into your table following the expected pattern.
MetadataSchema ChangeTableDetects any change to the schema: new field(s), removed field(s), existing field(s) with updated types or names.
MetricsAverage (static thresholds)Fields: NumericThe monitor fails if an average value of a given field is outside of a given range.
MetricsValues count (static thresholds)Fields: AllThe monitor fails if a number of unique values is outside of a given range.
MetricsQuantile (static thresholds)Fields: NumericThe monitor fails if a quantile of the field is outside of a given range.
MetricsValues (static thresholds)Fields: NumericThe monitor fails if the chosen field has any values outside of a given range.
MetricsStandard Deviation (static thresholds)Fields: NumericThe monitor fails if a standard deviation of the field is outside of a given range.
MetricsSum (static thresholds)Fields: NumericThe monitor fails if a sum of the field is outside of a given range.
MetricsVariance (static thresholds)Fields: NumericThe monitor fails if a variance of the field is outside of a given range.
Smart MetricsInterlinked Metrics Fields: NumericThe monitor fails if defined metrics diverge significantly from each other.
Smart MetricsMetrics (dynamic thresholds)Fields: NumericThe monitor fails if the selected statistical transformation of the field behaves differently than it did in the past.
Smart MetricsMetrics Custom (dynamic thresholds)TableThe rule fails if the time series returned by the query behave differently than they did in the past
Field profilingDistribution ChangeFields: AllThe monitor fails if the distribution of a given field has changed abnormally compared to a former given run.
Field profilingDuplicates in % (static thresholds)Fields: AllThe monitor fails if chosen field duplication rate [%] is superior to a given threshold.
Field profilingDuplicates in # (dynamic thresholds)Fields: AllThe monitor fails if the count of duplicate field values is abnormal compared to expectations based on past behavior.
Field profilingDuplicates in % (dynamic thresholds)Fields: AllThe monitor fails if the % of duplicate values of the field is abnormal compared to expectations based on past behavior.
Field profilingLow CardinalityFields: AllThe monitor fails if:
- The chosen field has several different values above a given threshold;
- The different values of the field changed since the previous run
Ex: from ['dog', 'cat'] to ['dog', 'rabbit', 'turtle'].
Field profilingNot after dateFields: Timestamps, DatesThe monitor fails if the table has rows after a given date.
Field profilingNot before dateFields: Timestamps, DatesThe monitor fails if the table has rows before a given date.
Field profilingNot in the listFields: StringThe monitor fails if the chosen field has values that are not present in the given list.
Field profilingNull in # (static thresholds)Fields: AllThe monitor fails if the chosen field has values that are empty/null.
Field profilingNull in % (static thresholds)Fields: AllThe monitor fails if the null value rate of the selected field is superior to a given threshold.
Field profilingNull in # (dynamic thresholds)Fields: AllThe monitor fails if the count of null values of the field is abnormal compared to expectations based on past behavior.
Field profilingNull in % (dynamic thresholds)Fields: AllThe monitor fails if the % of null values of the field is abnormal compared to expectations based on past behavior.
Field profilingUniqueFields: AllThe monitor fails if the chosen field has duplications.
Format validationIs an emailFields: StringThe monitor fails if the chosen field contains at least one row that does not have an email format.
Format validationIs a phone numberFields: StringThe monitor fails if the chosen field contains at least one row that does not have a phone number format.
I.e. checking for 6 to 16 digits and will accept the field to contain the following characters +, -, (, )
Format validationIs UUIDFields: StringThe monitor fails if the chosen field contains at least one row that does not have a UUID format.
Format validationMatches regexFields: StringThe rule fails if the selected field contains at least one row that does not match the format specified by the given regular expression.
CustomSQLTableAdvanced template to write custom monitors based on business specifics. The SQL query must describe a quality breach on one or more tables within the same data source.
CustomConditional monitorsTablesAs for SQL, this template allows to write custom monitors based on business use cases. With conditional statements, no SQL syntax is needed. The rule fails if values are found inside the filtering criteria set by conditional rules