Athena

Sifflet monitors the health and quality of the data you query with Amazon Athena, from table updates down to individual field values and business metrics. When an issue occurs, lineage built from your Athena query history and table definitions shows its root cause, from the S3 locations behind your tables, and the downstream tables and QuickSight dashboards it affects. For the full list of imported objects, see Athena Collected Data.

Permissions Required

Sifflet assumes an AWS IAM role in your AWS account to access Athena, AWS Glue, and Amazon S3. The role grants read-only access to your data, plus write access to the S3 location where Athena stores query results.

Write access: Athena writes the results of every query to an S3 output location. Sifflet needs s3:PutObject on this location only, not on the buckets that store your data.

Cost and load impact: Sifflet runs its metadata, preview, and monitor queries in the workgroup you configure, so they're billed per amount of data scanned, like your other Athena queries. Use a dedicated workgroup for Sifflet to track this cost and set data usage limits. Query results accumulate in the S3 output location: you can add an S3 lifecycle rule to delete them after a few days.

Requirements

Required Roles for Setup

• In AWS: the rights to create IAM policies and roles in the AWS account that hosts Athena (for example the IAMFullAccess managed policy).
• In Sifflet: a role that can create sources, such as Admin or System Editor. See Access Control.

Supported Versions and Editions

• Supported: Amazon Athena SQL queries on the AWS Glue Data Catalog (AwsDataCatalog). The permissions on this page cover this catalog only.

Network Access

• Default: Sifflet calls the public AWS endpoints of Athena, AWS Glue, and Amazon S3 with the IAM role you create.

Other Prerequisites

• An Athena workgroup for Sifflet to run its queries in (for example primary, or a dedicated workgroup).
• An S3 location where Athena stores query results (for example s3://my-athena-results/sifflet/). If the workgroup overrides client-side settings, Athena uses the workgroup's output location instead: grant the S3 permissions on that location.

Connect Sifflet to Athena

Step 1: Create an IAM Role in AWS

Sifflet assumes an IAM role in your account. You create a policy with the permissions above, then a role that Sifflet's own AWS role can assume.

Create the Policy

1. In the AWS console, go to IAM > Policies and click Create policy.

2. Select the JSON editor and paste the following policy. Replace the placeholders:

   • <AWS_REGION> and <AWS_ACCOUNT_ID>: the region and ID of the AWS account that hosts Athena (for example eu-west-1 and 111122223333).
   • <WORKGROUP>: the Athena workgroup Sifflet uses (for example primary).
   • <DATA_BUCKET>: a bucket that stores the data of your tables. Repeat the two resource lines for each bucket.
   • <RESULTS_BUCKET> and <RESULTS_PREFIX>: the bucket and prefix of the S3 output location (for example my-athena-results and sifflet).
   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AthenaQueries",
         "Effect": "Allow",
         "Action": [
           "athena:StartQueryExecution",
           "athena:GetQueryExecution",
           "athena:GetQueryResults",
           "athena:GetQueryResultsStream",
           "athena:CreatePreparedStatement",
           "athena:GetPreparedStatement",
           "athena:DeletePreparedStatement"
         ],
         "Resource": "arn:aws:athena:<AWS_REGION>:<AWS_ACCOUNT_ID>:workgroup/<WORKGROUP>"
       },
       {
         "Sid": "AthenaListDataCatalogs",
         "Effect": "Allow",
         "Action": "athena:ListDataCatalogs",
         "Resource": "*"
       },
       {
         "Sid": "AthenaMetadata",
         "Effect": "Allow",
         "Action": [
           "athena:GetDatabase",
           "athena:ListDatabases",
           "athena:GetTableMetadata",
           "athena:ListTableMetadata"
         ],
         "Resource": "arn:aws:athena:<AWS_REGION>:<AWS_ACCOUNT_ID>:datacatalog/*"
       },
       {
         "Sid": "GlueMetadata",
         "Effect": "Allow",
         "Action": [
           "glue:GetDatabase",
           "glue:GetDatabases",
           "glue:GetTable",
           "glue:GetTables",
           "glue:GetPartition",
           "glue:GetPartitions",
           "glue:BatchGetPartition"
         ],
         "Resource": [
           "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
           "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/*",
           "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/*/*"
         ]
       },
       {
         "Sid": "ReadData",
         "Effect": "Allow",
         "Action": [
           "s3:GetBucketLocation",
           "s3:ListBucket",
           "s3:GetObject"
         ],
         "Resource": [
           "arn:aws:s3:::<DATA_BUCKET>",
           "arn:aws:s3:::<DATA_BUCKET>/*"
         ]
       },
       {
         "Sid": "QueryResultsBucket",
         "Effect": "Allow",
         "Action": [
           "s3:GetBucketLocation",
           "s3:ListBucket"
         ],
         "Resource": "arn:aws:s3:::<RESULTS_BUCKET>"
       },
       {
         "Sid": "QueryResultsObjects",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject",
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::<RESULTS_BUCKET>/<RESULTS_PREFIX>/*"
       }
     ]
   }

3. Name the policy (for example sifflet_athena_read_policy) and click Create policy.

To let Sifflet build lineage from your query history (optional), create a second policy, for example sifflet_athena_lineage_policy, with the following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AthenaQueryHistory",
      "Effect": "Allow",
      "Action": [
        "athena:ListQueryExecutions",
        "athena:BatchGetQueryExecution"
      ],
      "Resource": "arn:aws:athena:<AWS_REGION>:<AWS_ACCOUNT_ID>:workgroup/<WORKGROUP>"
    }
  ]
}

Create the Role

1. In Sifflet, go to Settings > Integration Preferences and copy the AWS role ARN of your Sifflet instance.

2. In the AWS console, go to IAM > Roles and click Create role.

3. Select Custom trust policy and paste the following trust policy. Replace <SIFFLET_AWS_ROLE_ARN> with the ARN you copied.

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<SIFFLET_AWS_ROLE_ARN>"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

4. Attach the policy you created (sifflet_athena_read_policy), and the lineage policy if you created it.

5. Name the role (for example sifflet_athena_read_role) and click Create role.

For more details, see the AWS documentation on custom trust policies.

Keep the ARN of the role for step 2, for example arn:aws:iam::111122223333:role/sifflet_athena_read_role.

Step 2: Add the Athena Source in Sifflet

Athena sources don't use a Sifflet credential: Sifflet assumes the IAM role you created.

1. Create the source: in Integrations > Sources, click New source, choose Athena, and fill in the following fields:

   Field	Description	Example
   Source name	The name of the source in Sifflet	Athena Production
   AWS region	The AWS region of Athena	eu-west-1
   Athena data source	The name of the Athena data catalog	AwsDataCatalog (default)
   S3 output location	The S3 location where Athena stores query results, starting with s3://	s3://my-athena-results/sifflet/
   Workgroup	The Athena workgroup Sifflet runs its queries in	primary (default)
   AWS role ARN	The ARN of the role you created in step 1	arn:aws:iam::111122223333:role/sifflet_athena_read_role
   VPC url (optional)	The hostname of an Athena interface VPC endpoint, to use instead of the public Athena endpoint. Leave empty to use the public endpoint.

2. Select the scope: click Load assets, then select the Athena databases Sifflet catalogs. Enable Automatically include new schemas so that Sifflet also catalogs databases created later. Create a single source per data catalog: see Integrations Management for the one-source-per-environment model and refresh scheduling.

3. Test and save: click Test Connection, then save the source. Once the first metadata refresh completes, your tables and views appear in the Data Catalog. You can follow the status and logs of each refresh in the source details.

Troubleshooting

Test Connection or Asset Loading Fails

Symptom: Test Connection fails, or Load assets returns an error.

Cause: Sifflet can't assume the IAM role, or the role misses a permission on Athena, AWS Glue, or Amazon S3.

Fix:

1. Check that the trust policy of the role contains the AWS role ARN shown in Settings > Integration Preferences, as described in step 1.
2. Check that the role's policy matches the policy of step 1, with the right region, account ID, workgroup, and buckets, and that the S3 output location of the source matches the bucket and prefix of the policy.
3. If the issue persists, contact Sifflet support and provide the role's permission policy and trust policy.