Overview

The Sifflet Volume monitor is a table-level metadata monitor. It detects changes in the volume of newly ingested data. Significant changes in data volume may indicate data duplication, data loss, or data corruption. By identifying and addressing these issues early on, you ensure data accuracy and enable better analysis.

📘

Metadata Monitoring

Metadata is information about data, including its structure and the transformations applied to it. Metadata monitoring helps you identify and address issues related to data integration and data transformations.

As data volumes and complexity grow, metadata monitoring helps you keep your data reliable.

How To

How It Works

The Volume monitor compares the actual volume of data ingested in a dataset per time interval (an hour, a day) with the expected volume of ingestion. Machine learning models compute expectations based on the historical behavior of data.

Example

A company aggregates all the orders from its different selling platforms in a table called "Orders". Monitoring the daily volume of data being ingested in that table can detect:

• Missing data coming from one of the sources (technical issue)
• Fewer orders made on one selling platform (business issue)

👍

Tips

Volume monitors are often used with a "Group by" statement to identify volume drift at a more precise granularity. Example: group by geography and a type of product for an international retail company.