You can integrate with dbt Core by pushing to Sifflet the following dbt artifacts: manifest.json, run_results.json, and optionally catalog.json.

The main steps are the following:

1. Add the dbt Core integration in Sifflet
2. Adapt your dbt workflow
3. Send the dbt artifacts

1- Add the dbt Core integration in Sifflet

In "Integration" --> submenu "Sources" -> "New", choose the dbt Core option.
The required parameters are:

• Name: choose a name to represent your dbt Core integration in Sifflet
• Project Name: the name of your dbt project (the "name" in your dbt_project.yml file)
• Target the target value of the profile that corresponds to your project (the "target" in your profiles.yml)

🚧

Case sensitivity

As dbt is case sensitive when adding the Project Name and Target in the parameters, make sure that they match exactly with the names in your dbt project.

2- Adapt your dbt workflow

Option 1: Using dbt docs generate and dbt test

❗

️ dbt commands order

When you run "dbt test" on specific resources like models (see docs), the generated artifacts override any existing artifacts from previous runs.
Therefore, in order to get the complete artifact files, we recommend running the below commands at the very end of your DAG and in this order :

• dbt docs generate: generates a complete "manifest.json" and "catalog.json".
• dbt test: generates a complete "run_results.json" file.

Option 2: using dbt build

Alternatively, you can use the dbt build command (see docs) to run your different dbt assets. This command generates complete "manifest.json" and "run_results.json" files.

If you decide to use this command, you can optionally run the dbt docs generate command beforehand (same order as option 1) to generate the "catalog.json" artifact. This ensures that Sifflet has all the necessary metadata about your dbt assets.

3- Send the dbt artifacts

1. In order to send programmatically the dbt artifacts, you will need to generate an API Token with the Role Editor or Admin (see here for more details).
2. Then, you have several options to send the dbt artifacts to Sifflet:
   1. using SiffletDbtIngestOperator operator
   2. or using the CLI
   3. or using the API

SiffletDbtIngestOperator operator

With the API token generated previously, you can find how to configure the CLI here.
If you use Airflow as your orchestrator, you have the SiffletDbtIngestOperator operator at your disposal.
It takes as parameters:

• task_id: unique identifier of the task
• input_folder: directory of the dbt artifacts
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• project_name: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)

sifflet_dbt_ingest = SiffletDbtIngestOperator(
      task_id="sifflet_dbt_ingest",
      input_folder=<DBT_PROJ_DIR>,
      target=<DBT_TARGET>,
      project_name=<DBT_PROJECT_NAME>,
    )

Sifflet's CLI

You can also send the dbt artifacts with Sifflet's CLI.
With the API token generated previously, you can find how to configure the CLI here.

Please refer to the sifflet ingest dbt command line reference for the usage.

Sifflet's API

Another option is to integrate the below script directly in your pipeline.

#!/bin/bash

accessToken="<INSERT_ACCESS_TOKEN>"
projectName='<project_name>'
target='<target>'

curl -v "https://<tenant_name>.siffletdata.com/api/v1/metadata/dbt/$projectName/$target"  \
-F 'manifest=@target/manifest.json' \
-F 'catalog=@target/catalog.json' \
-F 'run_results=@target/run_results.json' \
-H "Authorization: Bearer $accessToken"

• accessToken: is the value you previously generated and stored
• projectName: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• tenant_name: if you access to Sifflet with <https://abcdef.siffletdata.com>, then your tenant would be abcdef

FAQ

The dbt tests listed in the monitor section do not show any status and last runs information

If the dbt tests listed in the monitor section do not contain information about the success or failure of the last run, it is likely due to the order between the commands docs generation and dbt test. Please ensure to run the commands in the following order:

1. dbt docs generate
2. dbt test

For more information, see step 2 above.