For Loops

For Loops allow for the creation of many monitors at the same time, in minimal code.

Start your object with:
for each dataset T:

For loops take two parameters: datasets and monitors

Specifying Datasets

The datasets parameter allows you to specify which datasets the for loop should create monitors for.

for each dataset T:
  datasets:
   include:
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.TABLE
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.OTHERTABLE
  monitors:
  - kind: Monitor
    ...

Wildcards

You can use wildcards to specify many assets at once. Simply use * in your includes

for each dataset T:
 datasets:
   include:
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
  monitors:
  - kind: Monitor
    ...

The above example creates monitors for all the tables in the SCHEMA and OTHERSCHEMA schemas.

Accepted Types: Tables, Views, External Tables

When using wildcards you can specify accepted dataset types

acceptTypes:

All: Tables, Views and External Tables
Table: Physical Tables only
View : Views
ExternalTable: External Table

One Type:

for each dataset T:
 datasets:
   include:
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
   - uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
   acceptedTypes: Table
  monitors:
  - kind: Monitor
    ...

Multiple Types

for each dataset:
  datasets:
	  acceptedTypes:
		  - Table
		  - External
		  - ...
	  include:
		  - uri: bigquery:sifflet-demo-project.test_only.*

Excludes

When Using wildcards you may want to exclude tables. Use the exclude key in the datasets definition

for each dataset T:
  datasets:
    include:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.OTHERSCHEMA.*
    exclude:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
  monitors:
  - kind: Monitor
    ...

Dynamic Monitor names

When writing for Loops you can dynamically include the dataset name in the name or description of the monitor

${T.name} : Name of the dataset
${T.uri}: Uri of the dataset

for each dataset T:
  datasets:
   include:
     - uri: snowflake://orgname-accountname/DEMO.SE_ENV.ORDERS
     - uri: snowflake://orgname-accountname/DEMO.SE_ENV.CUSTOMERS
     - uri: snowflake://orgname-accountname/DEMO.SE_ENV.STG_CUSTOMERS
  monitors:
   - kind: Monitor
     name: "[${T.name}]Format monitor on CUSTOMER_ID" 
     extends: 
      - customer_id_check

An Example

for each dataset T:
  datasets:
    include:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
    - uri: bigquery:*.*.*ORDERS
    exclude:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE
  monitors:
  - kind: Monitor
    version: 1
    schedule: 34 5 * * *
    incident:
    severity: Low
    parameters:
      kind: SchemaChange
    friendlyId: MySchemaChange
    name: My Monitor  ${T.name}.
  - kind: Monitor
  ...

Updated 2 months ago