Time window

Unless your tables are frequently loaded entirely, setting a time window is recommended to optimize your resources. Sifflet will scan only the new data ingested during the time window that has been set up.

For instance, let's say you want to monitor a table that is updated daily. Setting a time window on this table will require:

• A specific field that represents the time dimension (for instance, the creation time of the data entry);
• The historical time window range that you want to consider for the rules scan (one day, one month, one week, for example)

Set up the Time Window at rule level

Please follow these steps:

• Go to "Monitoring" -> "New"
• After selecting your datasource and table, you will have the option to choose a Time Window

Set up the Time Window at table level

Please follow these steps:

• Go to "Catalog"
• Select your table
• Press the "Set Time Window" button

• Choose the right time field and time window range

Note: Since you can also set up the Time Window for each rule independently or at table level, a rule will consider in priority the former. If there isn't any defined, it will consider the latter.

📘

We recommend setting the time window based on the ingestion schedule of the table. If the table is being updated daily, for example, then a time window of one day will help check the new data ingested every day the rule run.