Redshift

Sifflet monitors the health and quality of your Amazon Redshift data, from table updates down to individual field values and business metrics. Sifflet catalogs your Redshift tables and views and, on provisioned clusters, builds lineage from your query history: when an issue occurs, you see its root cause upstream and the downstream tables and dashboards it affects.

For the full list of assets and metadata Sifflet imports, see Redshift Collected Data.

Permissions Required

Sifflet uses read-only access. It connects to your provisioned cluster or Serverless workgroup as a dedicated database user, authenticated with a username and password (recommended), or, on Redshift Serverless only, with an AWS IAM role (private beta).

SYSLOG ACCESS UNRESTRICTED gives the Sifflet user visibility on the text of every query run on the cluster, which can contain literal values such as personal data. Sifflet uses these queries only to compute lineage. Without this privilege, lineage only covers the queries run by the Sifflet user, so it is mostly empty.

Cost and load: Sifflet runs metadata and lineage queries during each refresh, and monitor queries on each monitor run, on your cluster or workgroup. On a provisioned cluster, they use cluster capacity; on Redshift Serverless, they consume Redshift Processing Units (RPUs) that you pay for. To isolate Sifflet's load on a provisioned cluster, assign the Sifflet user to a dedicated workload management (WLM) queue.

Requirements

Required Roles for Setup

• In Redshift: a superuser, to create the Sifflet user, grant SYSLOG ACCESS UNRESTRICTED, and grant access to your schemas.
• In AWS, for IAM authentication only: the right to create IAM policies and roles.
• In Sifflet: the Admin role to create the credential, and the Admin or System Editor role to create the source. See Access Control.

Supported Versions and Editions

• Redshift provisioned clusters: supported, with lineage.
• Redshift Serverless: supported without lineage. Redshift Serverless doesn't expose SVL_STATEMENTTEXT, so Sifflet doesn't build lineage from your query history. Use the dbt integration or declarative lineage to add lineage.
• Authentication: username and password on both, or IAM authentication on Redshift Serverless only (private beta: ask your Sifflet customer success representative to enable it).
• Host names: with username and password authentication, the host must be an AWS endpoint ending in .redshift.amazonaws.com, .redshift-serverless.amazonaws.com, or .vpce.amazonaws.com. Custom DNS names and load balancer host names are only supported with IAM authentication.

Network Access

• Default: Sifflet connects to your cluster or workgroup over the internet, on its port (5439 by default), so it must be publicly accessible.
• IP allowlisting: restrict inbound traffic with the security group of your cluster or workgroup (AWS documentation), and allow the IP addresses of your Sifflet instance, listed in Settings > Integration Preferences.

Connect Sifflet to Redshift

Step 1: Create a Database User in Redshift

Use a database user with a password: it works on provisioned clusters and Redshift Serverless, and is generally available. To use an IAM role on Redshift Serverless instead, see Alternative: IAM Authentication below.

1. Connect to your cluster or workgroup as a superuser.

2. Create the Sifflet user. Choose an identifiable name, for example sifflet_user, and a strong password:

   SQL

   CREATE USER <USERNAME> PASSWORD '<PASSWORD>';

3. Grant read access to each schema to monitor. Run this block in each database that contains schemas to monitor, once per schema:

   SQL

   -- Repeat for each schema to monitor
   GRANT USAGE ON SCHEMA <SCHEMA_NAME> TO <USERNAME>;
   GRANT SELECT ON ALL TABLES IN SCHEMA <SCHEMA_NAME> TO <USERNAME>;
   ALTER DEFAULT PRIVILEGES IN SCHEMA <SCHEMA_NAME> GRANT SELECT ON TABLES TO <USERNAME>;

   ALTER DEFAULT PRIVILEGES only applies to the tables that the user running it creates later. If other users create tables in the schema, run it for each of them with ALTER DEFAULT PRIVILEGES FOR USER <TABLE_OWNER> IN SCHEMA <SCHEMA_NAME> GRANT SELECT ON TABLES TO <USERNAME>;.

4. Optional (lineage, provisioned clusters only): allow Sifflet to read the query history of all users:

   SQL

   -- Optional: lineage from query history
   ALTER USER <USERNAME> SYSLOG ACCESS UNRESTRICTED;
   GRANT SELECT ON TABLE SVL_STATEMENTTEXT TO <USERNAME>;

5. Optional (future features): grant read access to system views that Sifflet doesn't use today, so that you don't need new grants when it starts using them:

   SQL

   -- Optional: future features, not used today
   GRANT SELECT ON SVV_ALL_SCHEMAS TO <USERNAME>;
   GRANT SELECT ON SVV_TABLE_INFO TO <USERNAME>;

Keep the username and password for step 2.

Alternative: IAM Authentication (Redshift Serverless)

🚧

Private beta

IAM authentication is in private beta, and only available for Redshift Serverless. Ask your Sifflet customer success representative to enable it.

With IAM authentication, Sifflet assumes an IAM role in your AWS account, and Redshift Serverless gives it temporary database credentials. Use it when you manage access to your data with AWS Lake Formation, or when you can't store a database password in Sifflet.

1. In the AWS console, go to IAM > Policies, click Create policy, and create a policy that allows redshift-serverless:GetCredentials on your workgroup:

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "redshift-serverless:GetCredentials",
         "Resource": "arn:aws:redshift-serverless:<REGION>:<ACCOUNT_ID>:workgroup/<WORKGROUP_ID>"
       }
     ]
   }

   If you manage permissions with AWS Lake Formation, also grant this role read access to the data to monitor in Lake Formation.

2. Go to IAM > Roles, click Create role, select Custom trust policy, and paste the following policy. Replace <SIFFLET_AWS_ROLE_ARN> with the AWS Role ARN shown in Sifflet in Settings > Integration Preferences:

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "<SIFFLET_AWS_ROLE_ARN>"
           ]
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

3. Attach the policy created above, and name the role, for example sifflet_redshift_read_role.

4. Note the role ARN, for example arn:aws:iam::123456789012:role/sifflet_redshift_read_role. You need it in step 2.

5. Grant the database privileges. Redshift creates the database user IAMR:<ROLE_NAME> the first time Sifflet connects with the role, and you can only grant privileges to it after that. So create the source first (step 2) and click Test, then run the grants of item 3 of step 1, with "IAMR:<ROLE_NAME>" (double quotes included) as the username, and click Test again. If you manage permissions with Lake Formation instead, skip this item.

If the role lacks access to a schema, the connection still succeeds, but Sifflet finds no tables in it.

Step 2: Add the Redshift Source in Sifflet

1. Create the credential (username and password authentication only): go to Integrations > Credentials, click New credential, and paste the username and password of the Sifflet user:

   JSON

   {
     "user": "<USERNAME>",
     "password": "<PASSWORD>"
   }

2. Create the source: go to Integrations > Sources, click New source, select Redshift, and fill in the form. You find the endpoint of a provisioned cluster in the cluster's General information in the Redshift console, and the endpoint of a Serverless workgroup in the workgroup's General information. Remove the port and database from the endpoint:

   Field	Description	Example
   Source name	The name of the source in Sifflet	Redshift Production
   Host	The endpoint host name, without port or database	my-cluster.abc123xyz789.eu-west-1.redshift.amazonaws.com
   Port	The port of the cluster or workgroup	5439
   Enforce encrypted connection (SSL)	Connect with SSL. Sifflet accepts certificates from all certificate authorities. Required for IAM authentication	Checked
   Default database	The database Sifflet connects to first	dev
   Authentication method	User / password, or IAM for IAM authentication. Only shown when IAM authentication is enabled on your Sifflet instance	User / password
   Credential	User / password only: the credential created above	redshift-credential
   IAM Role ARN	IAM only: the ARN of the role created in step 1	arn:aws:iam::123456789012:role/sifflet_redshift_read_role
   Workgroup name	IAM only, optional: required when the host isn't a standard Serverless endpoint (custom DNS name, load balancer)	my-workgroup
   AWS region	IAM only, optional: required when the host isn't a standard Serverless endpoint	eu-west-1

3. Select the scope: click Load assets. Sifflet lists the databases and schemas the user can access. Select the databases or schemas to include in the Data Catalog, and enable Automatically include new schemas so that Sifflet also catalogs the schemas created later. Create one source per cluster or workgroup: see Integrations Management for the source model and refresh scheduling.

4. Test and save: click Test. Sifflet checks that it can connect, that the schemas exist, and whether it can read SVL_STATEMENTTEXT for lineage (a warning if not). Then save the source. Sifflet then refreshes the source: follow the status and logs of the refresh in the source details, from Integrations > Sources. Your Redshift assets appear in the Data Catalog once the refresh completes.

Troubleshooting

For connection, authentication, and permission errors, see Redshift Troubleshooting.