You can integrate with dbt Core by pushing to Sifflet the following dbt artifacts: manifest.json, catalog.json, run_results.json.
The main steps are the following:

1. Add the dbt Core integration in Sifflet
2. Adapt your dbt workflow
3. Send the dbt artifacts

1- Add the dbt Core integration in Sifflet

In "Integration" --> submenu "Sources" -> "New", choose the dbt Core option.
The required parameters are:

• Name: choose a name to represent your dbt Core integration in Sifflet
• Project Name: the name of your dbt project (the "name" in your dbt_project.yml file)
• Target the target value of the profile that corresponds to your project (the "target" in your profiles.yml)

🚧

Case sensitivity

As dbt is case sensitive when adding the Project Name and Target in the parameters, make sure that they match exactly with the names in your dbt project.

2- Adapt your dbt workflow

❗

️ dbt commands order and on specific resources

When you run "dbt test" on specific resources like models (see docs), the artifacts are generated each time and overriden by the next run.
Therefore, in order to get complete artifacts files, we recommend to run the below commands at the very end of your DAG and in this order :

• dbt docs generate, that generates a complete "manifest.json" and "catalog.json"
• dbt test, that generates a complete "run_results.json"

3- Send the dbt artifacts

1. In order to send programmatically the dbt artifacts, you will need to generate an API Token with the Role Editor or Admin (see here for more details).
2. Then, you have several options to send the dbt artifacts to Sifflet:
   1. using SiffletDbtIngestOperator operator
   2. or using the CLI
   3. or using the API

SiffletDbtIngestOperator operator

With the API token generated previously, you can find how to configure the CLI here.
If you use Airflow as your orchestrator, you have the SiffletDbtIngestOperator operator at your disposal.
It takes as parameters:

• task_id: unique identifier of the task
• input_folder: directory of the dbt artifacts
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• project_name: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)

sifflet_dbt_ingest = SiffletDbtIngestOperator(
      task_id="sifflet_dbt_ingest",
      input_folder=<DBT_PROJ_DIR>,
      target=<DBT_TARGET>,
      project_name=<DBT_PROJECT_NAME>,
    )

Sifflet's CLI

You can also send the dbt artifacts with Sifflet's CLI.
With the API token generated previously, you can find how to configure the CLI here.

Please refer to the sifflet ingest dbt command line reference for the usage.

Sifflet's API

Another option is to integrate the below script directly in your pipeline.

#!/bin/bash

accessToken="<INSERT_ACCESS_TOKEN>"
projectName='<project_name>'
target='<target>'

curl -v "https://<tenant_name>.siffletdata.com/api/v1/metadata/dbt/$projectName/$target"  \
-F 'manifest=@target/manifest.json' \
-F 'catalog=@target/catalog.json' \
-F 'run_results=@target/run_results.json' \
-H "Authorization: Bearer $accessToken"

• accessToken: is the value you previously generated and stored
• projectName: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• tenant_name: if you access to Sifflet with <https://abcdef.siffletdata.com>, then your tenant would be abcdef

FAQ

The dbt tests listed in the monitor section do not show any status and last runs information

If the dbt tests listed in the monitor section do not contain information about the success or failure of the last run, it is likely due to the order between the commands docs generation and dbt test. Please ensure to run the commands in the following order:

1. dbt docs generate
2. dbt test

For more information, see step 2 above.