For Loops
For Loops allow for the creation of many monitors at the same time, in minimal code.
Start your object with:
for each dataset T:
For loops take two parameters: datasets
and monitors
Specifying Datasets
The datasets
parameter allows you to specify which datasets the for loop should create monitors for.
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.TABLE
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.OTHERTABLE
monitors:
- kind: Monitor
...
Wildcards
You can use wildcards to specify many assets at once. Simply use *
in your includes
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
monitors:
- kind: Monitor
...
The above example creates monitors for all the tables in the SCHEMA and OTHERSCHEMA schemas.
Accepted Types: Tables, Views, External Tables
When using wildcards you can specify accepted dataset types
acceptTypes
:
All
: Tables, Views and External TablesTable
: Physical Tables onlyView
: ViewsExternalTable
: External Table
One Type:
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
acceptedTypes: Table
monitors:
- kind: Monitor
...
Multiple Types
for each dataset:
datasets:
acceptedTypes:
- Table
- External
- ...
include:
- uri: bigquery:sifflet-demo-project.test_only.*
Excludes
When Using wildcards you may want to exclude tables. Use the exclude
key in the datasets
definition
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
exclude:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
monitors:
- kind: Monitor
...
Dynamic Monitor names
When writing for Loops you can dynamically include the dataset name in the name or description of the monitor
${T.name}
: Name of the dataset
${T.uri}
: Uri of the dataset
for each dataset T:
datasets:
include:
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.ORDERS
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.CUSTOMERS
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.STG_CUSTOMERS
monitors:
- kind: Monitor
name: "[${T.name}]Format monitor on CUSTOMER_ID"
extends:
- customer_id_check
An Example
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: bigquery:*.*.*ORDERS
exclude:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
monitors:
- kind: Monitor
version: 1
schedule: 34 5 * * *
incident:
severity: Low
parameters:
kind: SchemaChange
friendlyId: MySchemaChange
name: My Monitor ${T.name}.
- kind: Monitor
...
Updated 15 days ago