dbt Core
You can integrate with dbt Core by pushing to Sifflet the following dbt artifacts: manifest.json, catalog.json, run_results.json.
The main steps are the following:
- Add the dbt Core integration in Sifflet
- Adapt your dbt workflow
- Send the dbt artifacts
1- Add the dbt Core integration in Sifflet
In "Integration" --> submenu "Sources" -> "New", choose the dbt Core option.
The required parameters are:
Name
: choose a name to represent your dbt Core integration in SiffletProject Name
: the name of your dbt project (the "name" in your dbt_project.yml file)Target
the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
Case sensitivity
As dbt is case sensitive when adding the Project Name and Target in the parameters, make sure that they match exactly with the names in your dbt project.

the profile is "sifflet_snowflake", hence the target is dev
2- Adapt your dbt workflow
️ dbt commands order and on specific resources
When you run "dbt test" on specific resources like models (see docs), the artifacts are generated each time and overriden by the next run.
Therefore, in order to get complete artifacts files, we recommend to run the below commands at the very end of your DAG and in this order :
- dbt docs generate, that generates a complete "manifest.json" and "catalog.json"
- dbt test, that generates a complete "run_results.json"
3- Send the dbt artifacts
Generate an API Token
In order to send programmatically the dbt artifacts, you will need first to generate an API Token.
You can find more details here.
You have several options to send the dbt artifacts to Sifflet:
- using SiffletDbtIngestOperator operator
- using the CLI
- using the API
SiffletDbtIngestOperator operator
With the API token generated previously, you can find how to configure the CLI here.
If you use Airflow as your orchestrator, you have the SiffletDbtIngestOperator
operator at your disposal.
It takes as parameters:
task_id
: unique identifier of the taskinput_folder
: directory of the dbt artifactstarget
: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)project_name
: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)
sifflet_dbt_ingest = SiffletDbtIngestOperator(
task_id="sifflet_dbt_ingest",
input_folder=<DBT_PROJ_DIR>,
target=>DBT_TARGET>,
project_name=<DBT_PROJECT_NAME>,
)
Sifflet's CLI
You can also send the dbt artifacts with Sifflet's CLI.
With the API token generated previously, you can find how to configure the CLI here.
Please refer to the sifflet ingest dbt command line reference for the usage.
Sifflet's API
Another option is to integrate the below script directly in your pipeline.
#!/bin/bash
accessToken="<INSERT_ACCESS_TOKEN>"
projectName='<project_name>'
target='<target>'
curl -v "https://<tenant_name>api.siffletdata.com/api/v1/metadata/dbt/$projectName/$target" \
-F '[email protected]/manifest.json' \
-F '[email protected]/catalog.json' \
-F '[email protected]/run_results.json' \
-H "Authorization: Bearer $accessToken"
accessToken
: is the value you previously generated and storedprojectName
: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)target
: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)tenant_name
: if you access to Sifflet with<https://abcdef.siffletdata.com
>, then your tenant would beabcdef
Updated 4 months ago