dbt Cloud

This page explains how to connect dbt Cloud to Sifflet: Sifflet fetches the artifacts of your latest job runs from the dbt Cloud API. For an overview of the dbt integrations, see dbt, and for the list of imported assets and metadata, see dbt Collected Data.

Permissions Required

Sifflet uses a dbt Cloud service token with read-only access to call the dbt Cloud Administrative API (v2). Sifflet never triggers, edits, or cancels jobs.

Sifflet doesn't run any query on your data platform.

Requirements

Required Roles for Setup

• In dbt Cloud: an account admin, to create the service token.
• In Sifflet: the Admin system role to create the credential, and the Admin or System Editor system role to create the source. See Access Control.

Supported Versions and Editions

• dbt Cloud (dbt platform), in any region or deployment that exposes the Administrative API v2, including multi-tenant regions and cell-based access URLs.
• Plans that support service tokens.
• The project's dbt adapter (adapter_type in manifest.json) must correspond to a data platform supported by Sifflet.

Network Access

Sifflet connects to the dbt Cloud API over the internet.

• IP allowlisting: if you restrict access to your dbt Cloud account with IP restrictions, allow the IP addresses of your Sifflet instance, listed in Settings > Integration Preferences.

Other Prerequisites

• Connect the data platform where dbt builds your models (Snowflake, BigQuery, Databricks, and so on) as a Sifflet source, so that Sifflet can link your dbt models to the monitored tables and views.

• Build your dbt Cloud jobs so that Sifflet finds complete artifacts. At each ingestion, Sifflet reads a single run per project:

  • It lists the most recent finished runs of the project (successful or failed; canceled runs are ignored), across all jobs and environments of the project.
  • It takes the most recent of these runs that has both a manifest.json and a run_results.json artifact. If none has both, it takes the most recent run that has a manifest.json, without run results.
  • dbt Cloud returns the artifacts of the last step of the run.

  This leads to the following constraints:

  Constraint	Why
  Use a single dbt Cloud environment per project	Sifflet doesn't filter runs by environment: runs of another environment (for example, a development environment) would replace your production metadata.
  Make dbt build the last step of every job	The run_results.json of a dbt build step contains both model and test results. With dbt run last, Sifflet gets no test results; with dbt test last, it gets no model run status.
  Clear the Generate docs on run option of your jobs	This option adds a dbt docs generate step at the end of the job, whose artifacts contain no model or test results. You can keep a dbt docs generate step in the job, as long as it isn't the last step.
  Schedule the Sifflet source to run right after your jobs end	Sifflet only reads the most recent run at ingestion time: the results of earlier runs since the previous ingestion are not imported. You may need to move the schedules of some jobs.

  Alternative: dedicated Sifflet job. If you can't change your existing jobs, create a job that runs dbt docs generate then dbt test (in this order, with dbt test last and Generate docs on run cleared), and schedule the Sifflet source to run right after it, with no other job of the project finishing in between. Sifflet then imports your test results, but not the status of your model runs.

  If neither option fits, use the dbt Core integration instead: upload the artifacts of each run to Sifflet from your own pipeline.

Connect Sifflet to dbt Cloud

Step 1: Create a Service Token in dbt Cloud

1. Log in to dbt Cloud with an account admin user.
2. Go to Account settings > Service tokens (in recent versions, Account settings > API tokens > Service tokens), and click New token (or Create service token).
3. Enter a name, for example sifflet.
4. Add the Read-Only permission set, and select All projects or each project that Sifflet monitors.
5. Save, then copy the token: dbt Cloud shows it only once. Store it in a safe location.

Also note the following values, used in step 2:

• Account ID: in the URL of any dbt Cloud page, after /accounts/. For https://cloud.getdbt.com/#/accounts/12345/settings/projects/67890/, the account ID is 12345.
• Access URL of your account: in Account settings, or the domain of your dbt Cloud URL (for example https://cloud.getdbt.com, https://emea.dbt.com, or https://ab123.us1.dbt.com). See Regions & IP addresses.

Step 2: Add the dbt Cloud Source in Sifflet

Create the Credential

In Integrations > Credentials, create a new credential and paste the service token as its value, as is (no JSON, no Token prefix):

<DBT_CLOUD_SERVICE_TOKEN>

Create the Source

1. Go to Integrations > Sources > New source and choose dbt Cloud.

2. Fill in the fields:

   Field	Description	Example
   Source name	Name of the source in Sifflet	dbt Cloud
   Base URL	Access URL of your dbt Cloud account followed by /api/v2. Default: https://cloud.getdbt.com/api/v2 (North America multi-tenant)	https://emea.dbt.com/api/v2
   Account id	Your dbt Cloud account ID	12345
   Credentials	The credential created above	dbt-cloud-token
   Create incident when monitor fails	When enabled (default), a failing dbt test creates an incident	Enabled

3. Under Asset to include in the Data Catalog, click Load assets and select the projects to monitor. Enable Automatically include new projects to monitor projects created later in dbt Cloud without editing the source.

4. In Set frequency, align the schedule with the end of your dbt Cloud jobs (see Other Prerequisites). For example, if your production job starts at 3:15 UTC and takes at most 45 minutes, schedule the source with the cron expression 0 4 * * *. See Integrations Management for the scheduling options.

5. (Optional) In Data pipeline failure notifications, select the Slack channels, Microsoft Teams channels, email addresses, or webhooks to notify when a model or seed run fails. See Pipeline Alerting.

6. Click Test connection: Sifflet checks that it can list the runs of the selected projects and download a manifest.json from one of the most recent finished runs. Then save the source.

Sifflet runs a first ingestion, which usually takes a few minutes. You can follow it in the source details, in Integrations > Sources. Once it succeeds, the dbt metadata appears on the asset pages of your models' tables and your dbt tests appear on the Monitors page. See dbt Collected Data.

Troubleshooting

Test Connection or Ingestion Fails Because No Artifacts Are Found

Symptom: Test connection or the ingestion fails with one of the following errors:

Could not fetch manifest for account <ACCOUNT_ID>, project <PROJECT_ID>, job definition
Cannot fetch valid dbt artifacts, no run found (projectId: <PROJECT_ID>)

Cause: none of the most recent finished runs of the project has a manifest.json artifact: the project has no finished run yet, its runs were all canceled, or the token can't read the project.

Fix:

1. Check that the service token has the Read-Only permission set on the project. See Step 1.
2. Check that the Base URL matches the access URL of your account, with the /api/v2 suffix. See Create the Source.
3. Run a job of the project in dbt Cloud, wait until it finishes, and test the connection again.

dbt Tests or Models Show No Status or Last Run Information

Symptom: dbt tests on the Monitors page, or models in the catalog, have no status or last run date, or the status is outdated.

Cause: the run that Sifflet read doesn't contain the results: its last step isn't dbt build, Generate docs on run is enabled, or another job (possibly from another environment) finished after your production job.

Fix:

1. Apply the job constraints of Other Prerequisites: one environment per project, dbt build as the last step, Generate docs on run cleared.
2. Schedule the Sifflet source right after the end of your jobs.