Airflow ᴮᴱᵀᴬ
This page covers Sifflet's Airflow integration, which allows you to see Airflow metadata in Sifflet. To trigger Sifflet operations from within your Airflow DAGs, you can find all of Sifflet's custom operators and their purposes here.
You can leverage our full data stack approach to obtain a complete view of your data pipelines, from your orchestrator to the data warehouse and your BI tool.
By integrating Airflow with Sifflet, you will have at a glance:
- Various metadata about your DAGs, such as "Last/Next Execution Date" and "Last Updated Date"
- The latest status of your DAG runs. It allows you to see directly whether a failure happened recently.
![Screenshot 2022-05-20 at 14.47.05.png 1668](https://files.readme.io/019c23e-Screenshot_2022-05-20_at_14.47.05.png)
"example_bash_operator"'s last run was successful, while "example_branch_datetime_operator_2"'s last run failed and needs attention
To integrate Airflow with Sifflet, these are the steps to follow:
- Create a dedicated read-only user
- Connect to Sifflet
Supported Airflow versions
We currently support any self-hosted Airflow instance - version 2.0.0+ - and are working on cloud-managed instances (MWAA on AWS, etc.)
1. Create a read-only user
You can create a dedicated Sifflet user with a "Viewer" role.
Please choose a "User Name" (for instance, "sifflet_user") and a secure password. Store them carefully as you will need them when configuring the connection in Sifflet later.
![Screenshot 2022-05-20 at 15.02.28.png 1320](https://files.readme.io/807aa8f-Screenshot_2022-05-20_at_15.02.28.png)
Sample configuration for a Sifflet user in Airflow
2. Connect to Sifflet
Add an Airflow secret
To create the Airflow secret, follow the below steps:
- In "Integration" --> tab "Secrets", create a new secret.
- In the "Secret" area, copy-paste the below text and replace it with the correct username and password previously created in step 1:
{
"user": "<username>",
"password": "<password>"
}
Create a new Airflow integration
To connect Airflow with Sifflet, you will need three items:
- Connection details:
- Host: You can add the entire URL. For instance, if your URL is
http://xxxxx.yy
, your Host value would behttp://xxxxx.yy
. - Port: The port used to interact with Airflow's REST API. By default, this is 8080.
- Host: You can add the entire URL. For instance, if your URL is
- Secret: corresponds to the username and password you previously chose.
- Frequency: determines how often the information is refreshed.
![Screenshot 2022-05-24 at 11.03.54.png 1832](https://files.readme.io/f01bf11-Screenshot_2022-05-24_at_11.03.54.png)
The different details that you need to provide when configuring the integration
You can also refer to this page on adding a data source in Sifflet.
Updated 26 days ago