dbt Cloud

You can integrate with dbt Cloud with the following steps:

  1. Generate an API Service Token
  2. Add the dbt Cloud integration in Sifflet

1- Generate an API Service Token with Read-Only access

🚧

The service token can only be generated with a dbt admin account

  • Connect to dbt Cloud with a dbt admin account: https://cloud.getdbt.com
  • On the left panel, Go to Account Settings > Service tokens
  • Generate the new token by pressing "New Token":
    • Add the Permission Set "Read-Only" and the corresponding Project
    • After saving, the token will only be shown once, please store it in a safe location.
18991899

2- Add the dbt Cloud integration in Sifflet

a. Add the token as a Sifflet secret

The token previously generated on dbt Cloud should be saved in Sifflet:

  • On the left panel, choose "Integration" and then the submenu "Secrets"
  • Press "New" Secret
  • Secret format:
799799

b. Add the dbt Cloud integration

32403240

In Sifflet, go to "Integration" --> "New", choose the dbt Cloud option.
The required parameters are:

  • Name: choose a name to represent your dbt Cloud integration in Sifflet
  • Account ID: your dbt Cloud account ID (see below on how to get it)
  • Project ID: your dbt project ID (see below on how to get it)
  • Job ID: your dbt job ID (see below on how to get it)
  • Secret: the secret you just created in the previous step
  • Frequency: how often Sifflet will sync with dbt Cloud.

📘

How to get your dbt Cloud account ID, project ID and job ID

Connect to your dbt Cloud account and choose a specific project. The URL link should be similar to this: https://cloud.getdbt.com/#/accounts/12345/settings/projects/67890/

  • The dbt Cloud account ID is 12345
  • The project ID is 67890

In some cases, it is easier to create a dedicated job to generate the dbt artifacts.
You can then add the job ID in your connection.
For an URL https://cloud.getdbt.com/#/accounts/12345/projects/67890/jobs/223344/

  • the job ID would be 223344

3- General advice

  • Dedicated Sifflet job
    In order to generate all dbt artifacts for Sifflet, a dedicated job is recommended. In this job, you will only have to include the following commands, in this order:
  1. dbt docs generate
  2. dbt test
13301330
  • dbt Job configuration:
    On your dbt job > "Settings", you can see on the bottom the schedule of the job. Be mindful of it (and the average running time) when choosing the sync frequency on Sifflet.
  • dbt commands order:
    Sifflet will look by default for the latest job that ran.
    To get accurate data, the command dbt test must be the last one to run in the job in order to get a valid "run_results.json" artifact. For instance, this file gets overridden by a dbt run or dbt docs generate and will not contain any test run results.
    Consequently, please make sure to unselect the option to "Generate docs" in your job configuration.

Please find below an example of correct job configuration on dbt:

16181618

Correct configuration, the command dbt docs generate is included, dbt test is the last one to run, and the option "Generate docs" is OFF