You can integrate with dbt Cloud with the following steps:
- (Optional) Create a dedicated Sifflet job
- Generate an API Service Token with Read-Only access
- Add the dbt Cloud integration in Sifflet
- Dedicated Sifflet job
In order to generate all dbt artifacts for Sifflet, a dedicated job is recommended. In this job, you will only have to include the following commands, in this order:
dbt docs generate
dbt commands order
Sifflet will look by default for the latest job that ran.
To get accurate data, the command
dbt testmust be the last one to run in the job in order to get a valid "run_results.json" artifact. For instance, this file gets overridden by a
dbt docs generateand will not contain any test run results.
Consequently, please make sure to unselect the option to "Generate docs" in your job configuration.
Please find below an example of correct job configuration on dbt:
- dbt Job scheduling configuration:
On the Triggers section, select Schedule and provide relevant scheduling according to your needs.
Scheduling frequency, timing & runtime
Be mindful to schedule this job in comparison with the other main jobs:
- At the same frequency
- After other jobs completed
- Consider the average run time of previous jobs to complete
The service token can only be generated with a dbt admin account
- Connect to dbt Cloud with a dbt admin account: https://cloud.getdbt.com
- On the left panel, Go to Account Settings > Service tokens
- Generate the new token by pressing "New Token":
- Add the Permission Set "Read-Only" and the corresponding Project
- After saving, the token will only be shown once, please store it in a safe location.
The token previously generated on dbt Cloud should be saved in Sifflet:
- On the left panel, choose "Integration" and then the submenu "Secrets"
- Press "New" Secret
- Secret format:
In Sifflet, go to "Integration" --> "New", choose the dbt Cloud option.
The required parameters are:
- Name: choose a name to represent your dbt Cloud integration in Sifflet
- Base Url: by default
https://cloud.getdbt.com/api/v2. You can change it in case your dbt Cloud instance is hosted in another location than North-America (see here for more details). For example, in case of EMEA, the url is
- Account ID: your dbt Cloud account ID (see below on how to get it)
- Project ID: your dbt project ID (see below on how to get it)
- Job ID: your dbt job ID (see below on how to get it)
- Secret: the secret you just created in the previous step
- Frequency: how often Sifflet will sync with dbt Cloud. As mentioned above for the dbt Cloud schedule, consider adding a similar frequency and a CRON that will trigger the refresh after the dedicated Sifflet job is completed.
How to get your dbt Cloud account ID, project ID and job ID
Connect to your dbt Cloud account and choose a specific project. The URL link should be similar to this: https://cloud.getdbt.com/#/accounts/12345/settings/projects/67890/
- The dbt Cloud account ID is 12345
- The project ID is 67890
In some cases, it is easier to create a dedicated job to generate the dbt artifacts.
You can then add the job ID in your connection.
For an URL https://cloud.getdbt.com/#/accounts/12345/projects/67890/jobs/223344/
- the job ID would be 223344
Updated 21 days ago