dbt Core
You can integrate with dbt Core by pushing to Sifflet the following dbt artifacts: manifest.json, run_results.json, and optionally catalog.json.
The main steps are the following:
- Add the dbt Core integration in Sifflet
- Adapt your dbt workflow
- Send the dbt artifacts
1- Add the dbt Core integration in Sifflet
In "Integration" --> submenu "Sources" -> "New", choose the dbt Core option.
The required parameters are:
Name
: choose a name to represent your dbt Core integration in SiffletProject Name
: the name of your dbt project (the "name" in your dbt_project.yml file)Target
the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
Case sensitivity
As dbt is case sensitive when adding the Project Name and Target in the parameters, make sure that they match exactly with the names in your dbt project.
2- Adapt your dbt workflow
Option 1: Using dbt docs generate
and dbt test
dbt docs generate
and dbt test
️ dbt commands order
When you run "dbt test" on specific resources like models (see docs), the generated artifacts override any existing artifacts from previous runs.
Therefore, in order to get the complete artifact files, we recommend running the below commands at the very end of your DAG and in this order :
dbt docs generate
: generates a complete "manifest.json" and "catalog.json".dbt test
: generates a complete "run_results.json" file.
Option 2: using dbt build
dbt build
Alternatively, you can use the dbt build
command (see docs) to run your different dbt assets. This command generates complete "manifest.json" and "run_results.json" files.
If you decide to use this command, you can optionally run the dbt docs generate
command beforehand (same order as option 1) to generate the "catalog.json" artifact. This ensures that Sifflet has all the necessary metadata about your dbt assets.
3- Send the dbt artifacts
- In order to send programmatically the dbt artifacts, you will need to generate an API Token with the Role
Editor
orAdmin
(see here for more details). - Then, you have several options to send the dbt artifacts to Sifflet:
- using SiffletDbtIngestOperator operator
- or using the CLI
- or using the API
SiffletDbtIngestOperator operator
With the API token generated previously, you can find how to configure the CLI here.
If you use Airflow as your orchestrator, you have the SiffletDbtIngestOperator
operator at your disposal.
It takes as parameters:
task_id
: unique identifier of the taskinput_folder
: directory of the dbt artifactstarget
: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)project_name
: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)
sifflet_dbt_ingest = SiffletDbtIngestOperator(
task_id="sifflet_dbt_ingest",
input_folder=<DBT_PROJ_DIR>,
target=<DBT_TARGET>,
project_name=<DBT_PROJECT_NAME>,
)
Sifflet's CLI
You can also send the dbt artifacts with Sifflet's CLI.
With the API token generated previously, you can find how to configure the CLI here.
Please refer to the sifflet ingest dbt command line reference for the usage.
Sifflet's API
Another option is to integrate the below script directly in your pipeline.
#!/bin/bash
accessToken="<INSERT_ACCESS_TOKEN>"
projectName='<project_name>'
target='<target>'
curl -v "https://<tenant_name>.siffletdata.com/api/v1/metadata/dbt/$projectName/$target" \
-F 'manifest=@target/manifest.json' \
-F 'catalog=@target/catalog.json' \
-F 'run_results=@target/run_results.json' \
-H "Authorization: Bearer $accessToken"
accessToken
: is the value you previously generated and storedprojectName
: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)target
: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)tenant_name
: if you access to Sifflet with<https://abcdef.siffletdata.com
>, then your tenant would beabcdef
FAQ
The dbt tests listed in the monitor section do not show any status and last runs information
If the dbt tests listed in the monitor section do not contain information about the success or failure of the last run, it is likely due to the order between the commands docs generation
and dbt test
. Please ensure to run the commands in the following order:
dbt docs generate
dbt test
For more information, see step 2 above.
Updated 4 months ago