Overview

The Freshness monitor tracks whether data is updated regularly. By specifying a monitoring frequency, it verifies whether data has been ingested within the designated time intervals.

📘
Metadata Monitoring
Metadata is information about data, including its structure and the transformations applied to it. Metadata monitoring helps you identify and address issues related to data integration and data transformations.
As data volumes and complexity grow, metadata monitoring helps you keep your data reliable.

How To

Modes

The Sifflet Freshness monitor template offers two modes of operation.

Static Mode

In Static Mode, the monitor expects data to be ingested within each time slot. If no data is found within a particular time interval, Sifflet creates an incident to notify you.

Dynamic Mode

In Dynamic Mode, the monitor automatically detects the ingestion pattern of data based on past behavior. It detects an anomaly only if data is expected to have arrived in the last time slot but has not. In this mode, Sifflet trains a machine learning algorithm to detect and predict data ingestion patterns.

Example

Data is expected to arrive twice a week, every Monday and Thursday. A Freshness monitor with the following parameters (Mode: Dynamic, Run Schedule: weekly, Time-based Data Aggregation: daily) fails only if data arrives on a day other than Monday or Thursday, or does not arrive on one of those days.

Graph

The Freshness monitoring graph shows the freshness of the ingested data so you can track data updates. It provides a visual representation of data arrival. The graph uses binary values, 0 and 1, to represent the data status as follows:

0: No updates in the table since the previous time slot.
1: New rows ingested during the last time slot, indicating recent data updates.

Freshness Monitor, Static Mode

Freshness Monitor, Dynamic Mode

Time-based Data Aggregation

Independently of the monitor run schedule, the Time-based Data Aggregation parameter defines the frequency at which data points are created.