Time parameters

There are 3 main time parameters to consider while configuring your monitors:

• Run Frequency
• Time Window
• Time Offset

Run Frequency

The Run Frequency describes intervals at which your monitor will be run. It can be defined either with wording shortcuts (@daily, @hourly) or a cron expression. More information on scheduling may be foundhere.

📘

Optimize your run frequency

You should adapt your frequency according to the refresh frequency of your data. If your data is updated by batch at 2am every day, running your monitors every hour would be suboptimal.

Time Window

Time Window is the time interval that will be analyzed every time the monitor is run. Sifflet will scan only new data ingested during a set time window. Unless your tables are frequently loaded entirely, setting a time window is recommended in order to optimize your resources.

Example

Let's say you want to monitor a table that is updated daily. Setting a Time Window on that table will require:

• A specific data field that represents the time dimension (for instance, the creation time of the data entry)
• The historical time window range that you want to consider for the monitors scan (one day, one month, one week, for example)

Some templates require a Time Window whereas it is optional for others. You will be asked to select the field representing the time dimension, the unit of time and the numerical amount of that unit of time.

📘

Optimize your Time Window setup

It's recommended to define your TIme Window based on the ingestion frequency of the data you're monitoring. For example, if a table is updated daily, a Run Frequency of 1 day in combination with a Time Window of 1 day would be recommended.

Time Offset

By default, Sifflet runs monitors on today's date. In some cases, your pipelines are configured so that it updates with a time delay: for instance, a sales table updated every morning with the data for T-2 days. To take this into account, Sifflet allows you to change the reference date by using an offset.

Parameters at dataset level

Time Window and Offset can also be generally configured at a dataset level. When doing so, every Monitor created on that asset will, by default, have those parameters. To do so, you need to select an asset in the Catalog and go to Overview. If Sifflet detects at least one temporal field (i.e. date), you will have a possibility to configure a Time Window and an Offset.