Overview

In data monitoring and analysis, Time Parameters ensure that the data you consider is relevant to the specific goals of the analysis. Adjusting them helps you focus on the most pertinent data, whether you are looking for real-time insights or longer-term trends.

Modes

Monitors have two Time Parameter modes:

Full data scan: Each monitor run will scan the entire table.
Incremental scan: Each monitor run will only scan a portion of the data.

Time Parameters in Incremental Mode

There are two main Time Parameters to consider while configuring Sifflet monitors in incremental mode:

Time Aggregation: The frequency of the points checked for anomalies, e.g., daily, hourly, weekly.
Time Offset: The delay in the data in the table.

📘
Dataset- vs Monitor-Level Parameters
Note that Time Offset parameters can also be set at Dataset Level. If you define them at the dataset level, Sifflet uses their values as defaults for all newly created monitors that accept those parameters. See this table for details.

Time Aggregation

The Time Aggregation describes intervals between points that are checked for anomalies.

This is the first parameter you should consider when establishing a monitor based on time. When you select daily time aggregation, Sifflet checks one data point per day. Time aggregation also affects the maximum training window for ML monitors.

Read more about Time-based Data Aggregation

📘
Schedule vs Time Aggregation
A monitor's Run Schedule does not need to be equal to its Time Aggregation parameter. For example, you can choose to run a monitor once a week (Schedule: @weekly), with a daily data point creation frequency (Time-based Data Aggregation: daily).
Read more about Time-based Data Aggregation.

📘
Optimize your run frequency
Adapt your run frequency to the refresh frequency of your data. If your data is updated in a batch at 2am every day, running your monitors every hour is unnecessary.

Time Offset

By default, Sifflet runs monitors on today's date. In some cases, pipelines are configured so that they update with a delay: e.g. a sales table updated every morning with the data for T-2 days. To take this into account, you can change the reference date by using an OFFSET parameter.

The offset typically represents the data delays. If the table contains only orders from 2 days ago, setting an offset of 2 days ensures that the monitor does not alert on empty or partial days.

🚧
Time Offset with other Time Parameters
Adding an offset does not alter the duration of the time aggregation or the rolling aggregation window; it only shifts it into the past by the given offset.

Advanced Parameters

Rolling Aggregation

Rolling Aggregation lets you set a custom time interval for every point that Sifflet checks. When rolling aggregation is OFF, the time interval matches the Time Aggregation. This means that typically, a daily point represents one day's worth of data.

However, there are scenarios where you might want a daily point that represents something other than a day, such as a daily point representing the rolling sum over the last 7 days.

📘
Rolling Aggregation on Static Monitors
Rolling Aggregation is not available for static monitors. Static monitors do not differentiate between a data point and a run and do not have a time aggregation parameter, so each run of the monitor represents one data point.
They do, however, have a time window parameter that acts in the same way as the rolling aggregation parameter and specifies the time window being checked.

Lookback Period

By default, Sifflet monitors run incrementally and only query new data.

However, there are scenarios where past data is susceptible to change and you want today's run of the monitor to also check previous days (or other time aggregation).

This is where the Lookback period comes in. Use it to specify a time window in which Sifflet rechecks previous data points for changes. Points that were previously within range but whose value now appears as an anomaly are included in the run's result.

👍
Anomalies in the lookback window that have correct values are automatically qualified as fixed

Time Parameters as Code

Snapshot Mode

parameters:
  kind: <MonitorKind>
  threshold:
    sensitivity: Low
  //no timewindow

If you do not set a time window parameter, the monitor scans the entire table on each run.

Incremental Mode

timeWindow: object that represents the time window configuration, when absent, the monitor acts in Snapshot Mode; when present, it acts in Incremental Mode.

timeWindow.field: the time field used to incrementally query data.

timeWindow.frequency: the time aggregation configuration. ISO 8601 format. P1D: Daily. P1W: Weekly. P1M: Monthly. PT1H: Hourly. PT30M: Every 30 minutes. PT20M: Every 20 minutes. PT15M: Every 15 minutes.

timeWindow.firstRun: the amount of data to query in the first run. ISO 8601 format

timeWindow.offset: the offset that represents the delay between the data and the present. Typically, it matches the granularity of the aggregation: days for daily, hours for hourly, etc. ISO 8601 format

parameters:
  kind: <MonitorKind>
  timeWindow:
    field: auto
    firstRun: P365D
    offset: P1D
    frequency: P1D

Incremental config for: An automatically selected time field. 1 point per day, with an offset of 1 to only track completed days. 365 days queried in the first run to build the graph and the prediction model.

timeWindow.rollingTimeWindow. An ISO 8601 format representation of the rolling time window that each point represents. P7D will make each point a rolling aggregation over the last 7 days for each point.

parameters:
  kind: <MonitorKind>
  timeWindow:
    field: order_date
    firstRun: P365D
    offset: P1D
    frequency: P1D
    rollingTimeWindow: P7D

Incremental config for: An order_date time field. 1 point per day that represents an aggregation over the previous 7 days from that point's date. With an offset of 1 to only start from completed days. 365 days queried in the first run to build the graph and the prediction model.

timeWindow.deltaQuerying. An ISO 8601 format representation of the lookback period of the monitor. P7D checks the last 7 daily data points on each run, rather than checking only a single data point per day.

parameters:
  kind: <MonitorKind>
  timeWindow:
    field: order_date
    firstRun: P365D
    offset: P1D
    frequency: P1D
    deltaQuerying: P7D