Airflow custom operators

To integrate Airflow with Sifflet (to retrieve its metadata), you can refer to this page.

Additionally, you can leverage the custom Airflow operators listed below to interact programmatically with Sifflet from within your Airflow DAGs.

For more details on how to install the Sifflet package for Airflow, please refer to this page

Please find below all the available operators:

• SiffletRunRuleOperator: Trigger one or several Sifflet monitors
• SiffletDbtIngestOperator: Send the dbt artifacts

SiffletRunRuleOperator

The operator takes as parameters:

• task_id: unique identifier of the task
• rule_ids: list of one or several rule IDs. For more details on how to find the rule_ids, please refer to this page
• error_on_rule_fail (optional): default value is true. If the rule fails, the task will fail as well. You can choose to set the parameter to false for less critical rules.

sifflet_rule = SiffletRunRuleOperator(
  task_id="sifflet_rule",
  rule_ids=["<rule_id_1>","<rule_id_2>"],
)

SiffletDbtIngestOperator

The operator takes as parameters:

• task_id: unique identifier of the task
• input_folder: directory of the dbt artifacts
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• project_name: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)

sifflet_rule = SiffletDbtIngestOperator(
  task_id="sifflet_dbt_ingest",
  input_folder="<DBT_PROJ_DIR>",
  target="<TARGET>",
  project_name="<DBT_PROJECT_NAME>",
)