For Loops
For Loops allow for the creation of many monitors at the same time, in minimal code.
Start your object with:
for each dataset T:
For loops take two parameters: datasets
and monitors
Specifying Datasets
The datasets
parameter allows you to specify which datasets the for loop should create monitors for.
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.TABLE
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.OTHERTABLE
monitors:
- kind: Monitor
...
Wildcards
You can use wildcards to specify many assets at once. Simply use *
in your includes
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
monitors:
- kind: Monitor
...
The above example creates monitors for all the tables in the SCHEMA and OTHERSCHEMA schemas.
Excludes
When Using wildcards you may want to exclude tables. Use the exclude
key in the datasets
definition
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
exclude:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
monitors:
- kind: Monitor
...
Dynamic Monitor names
When writing for Loops you can dynamically include the dataset name in the name or description of the monitor
${T.name}
: Name of the dataset
${T.uri}
: Uri of the dataset
for each dataset T:
datasets:
include:
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.ORDERS
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.CUSTOMERS
- uri: snowflake://orgname-accountname/DEMO.SE_ENV.STG_CUSTOMERS
monitors:
- kind: Monitor
name: "[${T.name}]Format monitor on CUSTOMER_ID"
extends:
- customer_id_check
An Example
for each dataset T:
datasets:
include:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
- uri: bigquery:*.*.*ORDERS
exclude:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
monitors:
- kind: Monitor
version: 1
schedule: 34 5 * * *
incident:
severity: Low
parameters:
kind: SchemaChange
friendlyId: MySchemaChange
name: My Monitor ${T.name}.
- kind: Monitor
...
Updated about 1 month ago