Amazon MWAA

This page explains how to connect Sifflet to Amazon Managed Workflows for Apache Airflow (MWAA). For an overview of the Airflow integration and its other deployments, see Apache Airflow; for what Sifflet imports, see Airflow Collected Data.

Permissions Required

Sifflet has read-only access to your MWAA environment. It assumes an AWS IAM role in your account, which you allow to log in to the environment with the Airflow Viewer role. Sifflet then calls the Airflow REST API (/api/v1) to list DAGs and read their latest runs.

The Airflow Viewer role only reads DAGs and DAG runs. airflow:InvokeRestApi doesn't grant more Airflow permissions: it forwards the same read-only calls through the AWS API, with the Viewer role of the resource ARN.

Requirements

Required Roles for Setup

• In AWS, IAM permissions to create policies and roles in the account of the MWAA environment.
• In Sifflet, the Admin role, to read the Sifflet AWS role ARN in Settings > Integration Preferences and to create the source. See Access Control.

Supported Versions and Editions

• MWAA environments running Airflow 2.
• MWAA environments running Airflow 3 aren't supported: Sifflet uses the Airflow REST API v1, which Airflow 3 removed.
• Private web servers require Airflow 2.4.3 or later, the MWAA requirement for InvokeRestApi. Some environments must be updated before they accept InvokeRestApi calls: see the MWAA documentation.

Network Access

• Default: with public network access (the Public network web server access mode), Sifflet calls the AWS MWAA API and the Airflow web server from the internet.
• Private web server: with private network access (the Private network web server access mode), Sifflet calls the Airflow REST API through the MWAA InvokeRestApi AWS API instead. This requires the optional airflow:InvokeRestApi permission. This feature is in private preview: contact Sifflet support to enable it.

Other Prerequisites

• The name and the AWS region of your MWAA environment.

Connect Sifflet to Amazon MWAA

Step 1: Create an IAM Role for Sifflet

Create the Policy

1. In the AWS console, go to IAM > Policies and click Create policy.

2. Select the JSON editor and paste the following policy. Replace <REGION>, <ACCOUNT_ID>, and <ENVIRONMENT_NAME> with the region, the AWS account ID, and the name of your MWAA environment.

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "airflow:CreateWebLoginToken"
         ],
         "Resource": [
           "arn:aws:airflow:<REGION>:<ACCOUNT_ID>:role/<ENVIRONMENT_NAME>/Viewer"
         ]
       }
     ]
   }

3. Optional (private web servers): to let Sifflet call the Airflow API through InvokeRestApi, use this policy instead:

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "airflow:CreateWebLoginToken",
           "airflow:InvokeRestApi"
         ],
         "Resource": [
           "arn:aws:airflow:<REGION>:<ACCOUNT_ID>:role/<ENVIRONMENT_NAME>/Viewer"
         ]
       }
     ]
   }

4. Click Next, name the policy (for example sifflet_mwaa_policy), and click Create policy.

Create the Role

1. In Sifflet, go to Settings > Integration Preferences and copy the Sifflet AWS role ARN.

2. In the AWS console, go to IAM > Roles and click Create role.

3. Select Custom trust policy and paste the following trust policy. Replace <SIFFLET_AWS_ROLE_ARN> with the ARN copied above.

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "<SIFFLET_AWS_ROLE_ARN>"
           ]
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

4. Click Next, select the policy created above (sifflet_mwaa_policy), and click Next.

5. Name the role (for example sifflet_mwaa_role) and click Create role.

6. Open the role and copy its ARN, for example arn:aws:iam::123456789012:role/sifflet_mwaa_role.

Keep the role ARN for step 2.

Step 2: Add the Amazon MWAA Source in Sifflet

MWAA sources don't use a Sifflet credential: Sifflet assumes the IAM role you created.

Create the Source

Go to Integrations > Sources, click New source, select Amazon MWAA, and fill in the form:

Sifflet imports all the DAGs of the environment: there is no scope to select.

Test and Save

1. Click Test Connection. Sifflet assumes the role, logs in to the environment, and checks that the Airflow API is healthy and returns DAGs.
   • When Sifflet uses InvokeRestApi, the result says Sifflet reaches the Airflow API through the MWAA InvokeRestApi API, which does not require network access to the web server.
   • When InvokeRestApi is enabled for your Sifflet instance but the role isn't allowed to call it, or the environment doesn't support it, the test succeeds with a warning: Sifflet connects to the web server directly, which only works with public network access.
2. Save the source. You can follow the status and logs of its refreshes in the source details: see Integrations Management. Your DAGs appear in the Data Catalog once the first refresh succeeds.

To link your DAGs to the tables they update, see Linking Airflow DAGs to Data Assets.

Troubleshooting

Sifflet Isn't Allowed to Access the Environment

• Symptom:

  Text

  The IAM role used by Sifflet is not allowed to access the MWAA environment <ENVIRONMENT_NAME>. Please grant it the airflow:CreateWebLoginToken and airflow:InvokeRestApi permissions on an Airflow role of the environment (for instance arn:aws:airflow:<region>:<account ID>:role/<ENVIRONMENT_NAME>/Viewer). AWS error: No Airflow role granted in IAM.

• Cause: the policy attached to the role doesn't allow airflow:CreateWebLoginToken on the Viewer role of this environment, or its resource ARN has the wrong region, account ID, or environment name.

• Fix:

  1. Check the policy and its resource ARN: see Step 1.
  2. Check that the policy is attached to the role entered in the source.

The Environment Isn't Found

• Symptom:

  Text

  The MWAA environment <ENVIRONMENT_NAME> was not found in the <REGION> region. Please check the environment name and region.

• Cause: the environment name or the region of the source doesn't match an MWAA environment in the account of the role.

• Fix: correct the Environment and AWS region fields of the source: see Step 2.

Sifflet Can't Connect to the Web Server

• Symptom:

  Text

  Sifflet cannot connect to the Airflow web server of your MWAA environment (<WEB_SERVER_HOSTNAME>): the connection timed out. This usually means that the web server uses private network access.

• Cause: the environment uses private network access, and Sifflet doesn't call the Airflow API through InvokeRestApi: the role isn't allowed to call it, or the feature isn't enabled for your Sifflet instance.

• Fix:

  1. Add airflow:InvokeRestApi to the policy of the role: see Step 1.
  2. If the error persists, contact Sifflet support to enable InvokeRestApi for your Sifflet instance.