Airflow Custom Operators

To integrate Airflow with Sifflet (to retrieve its metadata), you can refer to this page.

Additionally, you can leverage the custom Airflow operators listed below to interact programmatically with Sifflet from within your Airflow DAGs.

For details on how to install the Sifflet package for Airflow, see this page.

The available operators are:

• SiffletRunRuleOperator: Trigger one or several Sifflet monitors
• SiffletDbtIngestOperator: Send the dbt artifacts

SiffletRunRuleOperator

The operator takes as parameters:

• task_id: unique identifier of the task
• rule_ids: list of one or several monitor IDs. For details on how to find these IDs, see this page.
• error_on_rule_fail (optional): default value is true. If the monitor fails, the task fails as well. You can set the parameter to false for less critical monitors.

sifflet_rule = SiffletRunRuleOperator(
  task_id="sifflet_rule",
  rule_ids=["<rule_id_1>","<rule_id_2>"],
)

SiffletDbtIngestOperator

The operator takes as parameters:

• task_id: unique identifier of the task
• input_folder: directory of the dbt artifacts
• target: same value as in the dbt Core source configuration, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• project_name: same value as in the dbt Core source configuration, the name of your dbt project (the "name" in your dbt_project.yml file)

sifflet_dbt_ingest = SiffletDbtIngestOperator(
  task_id="sifflet_dbt_ingest",
  input_folder="<DBT_PROJ_DIR>",
  target="<TARGET>",
  project_name="<DBT_PROJECT_NAME>",
)