Flow Stopper

As your data moves through your pipelines, check its quality at every step and spot issues proactively before the data moves further down the pipeline.
For this purpose, use Sifflet's Flow Stopper.
You can implement it in a few steps:

Define the quality rules for the data to be monitored
Adapt your data pipeline: your orchestrator can trigger these rules after each transformation and check how reliable the data is. If an anomaly is detected, the orchestrator pauses the pipeline, which stops bad data from propagating.

You can find some examples on how to leverage Sifflet's Flow Stopper in Airflow here

Step 1: Define the Quality Rules

You can find all available data quality rules here.
Once done, the next part shows how to configure your data pipeline accordingly.

Step 2: Configure Your Data Pipeline

There are several options to insert Sifflet's Flow Stopper in your orchestrator:

If you use Airflow as your orchestrator, you can leverage the SiffletRunRuleOperator operator
For any other orchestrator, you can directly use Sifflet's CLI

Airflow SiffletRunRuleOperator Operator

Airflow Connection Setup

To allow your Airflow instance to connect to Sifflet, follow these steps:

Add the following package to your requirements.txt: airflow-provider-sifflet
Once the package is installed, add the Sifflet connection in Airflow. You can find more details on Airflow's docs:
- In the Airflow user interface, you can configure a Connection for Sifflet in Admin > Connections > Add a new record.
- In the "Add Connection" screen, you can fill the following parameters:
  - Connection Id: sifflet_default
  - Connection Type: Sifflet
  - Sifflet Token: You can find more information on how to generate it here
  - One of these two parameters is required, depending on your type of Sifflet deployment:
    - SaaS deployments: Sifflet Tenant, if you access Sifflet with https://abcdef.siffletdata.com, then your tenant would be abcdef
    - Self-hosted deployments: Sifflet Backend URL, full URL to the Sifflet backend on your deployment, for instance: https://sifflet-backend.mycompany.com

You can then use Sifflet's custom operator in your DAGs.

SiffletRunRuleOperator Operator Parameters

The operator takes as parameters:

task_id: unique identifier of the task
rule_ids: list of one or several rule IDs. For more details on how to find the rule IDs, see the Finding the rule ID callout below.
error_on_rule_fail (optional): default value is true. If the rule fails, the task will fail as well. You can choose to set the parameter to false for less critical rules.

sifflet_rule = SiffletRunRuleOperator(
  task_id="sifflet_rule",
  rule_ids=["<rule_id_1>","<rule_id_2>"],
)

📘
Finding the rule ID
You have several means to find the ID of your rule:

using the CLI sifflet rules list

on Sifflet's UI, from the URL on the page while on your rule. For instance, if your URL is https://<tenant_name>.siffletdata.com/monitoring/rule/e77ed92f-f248-418a-a4c6-a3de1e3aacf7/overview?name=%20Completeness%20Rule, then the rule_id would be e77ed92f-f248-418a-a4c6-a3de1e3aacf7

Using the CLI

For any other orchestrator, you can directly use Sifflet's CLI to trigger rules programmatically.
To set up Sifflet's CLI, see Sifflet CLI — Command-Line Interface.
You can then run the command sifflet rules run to trigger and test rules.
For more details, see the CLI reference.

# Run one or many rules
sifflet rules run --id <rule_id>
sifflet rules run --id <rule_id_1> --id <rule_id_2> --id <rule_id_3>

📘
Finding the rule ID
You have several means to find the ID of your rule:

using the CLI sifflet rules list

on Sifflet's UI, from the URL on the page while on your rule. For instance, if your URL is https://<tenant_name>.siffletdata.com/monitoring/rule/e77ed92f-f248-418a-a4c6-a3de1e3aacf7/overview?name=%20Completeness%20Rule, then the rule_id would be e77ed92f-f248-418a-a4c6-a3de1e3aacf7