dbt Core

You can integrate with dbt Core by pushing to Sifflet the following dbt artifacts: manifest.json, catalog.json, run_results.json.
The main steps are the following:

1. Add the dbt Core integration in Sifflet
2. Adapt your dbt workflow
3. Send the dbt artifacts

1- Add the dbt Core integration in Sifflet

In "Integration" --> submenu "Sources" -> "New", choose the dbt Core option.
The required parameters are:

• Name: choose a name to represent your dbt Core integration in Sifflet
• Project Name: the name of your dbt project (the "name" in your dbt_project.yml file)
• Target the target value of the profile that corresponds to your project (the "target" in your profiles.yml)

🚧

Case sensitivity

As dbt is case sensitive when adding the Project Name and Target in the parameters, make sure that they match exactly with the names in your dbt project.

2- Adapt your dbt workflow

❗

️ dbt commands order and on specific resources

When you run "dbt test" on specific resources like models (see docs), the artifacts are generated each time and overriden by the next run.
Therefore, in order to get complete artifacts files, we recommend to run the below commands at the very end of your DAG and in this order :

• dbt docs generate, that generates a complete "manifest.json" and "catalog.json"
• dbt test, that generates a complete "run_results.json"

3- Send the dbt artifacts

📘

Generate an API Token

In order to send programmatically the dbt artifacts, you will need first to generate an API Token.
You can find more details here.

You have several options to send the dbt artifacts to Sifflet:

• using SiffletDbtIngestOperator operator
• using the CLI
• using the API

SiffletDbtIngestOperator operator

With the API token generated previously, you can find how to configure the CLI here.
If you use Airflow as your orchestrator, you have the SiffletDbtIngestOperator operator at your disposal.
It takes as parameters:

• task_id: unique identifier of the task
• input_folder: directory of the dbt artifacts
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• project_name: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)

sifflet_dbt_ingest = SiffletDbtIngestOperator(
      task_id="sifflet_dbt_ingest",
      input_folder=<DBT_PROJ_DIR>,
      target=>DBT_TARGET>,
      project_name=<DBT_PROJECT_NAME>,
    )

Sifflet's CLI

You can also send the dbt artifacts with Sifflet's CLI.
With the API token generated previously, you can find how to configure the CLI here.

Please refer to the sifflet ingest dbt command line reference for the usage.

Sifflet's API

Another option is to integrate the below script directly in your pipeline.

#!/bin/bash

accessToken="<INSERT_ACCESS_TOKEN>"
projectName='<project_name>'
target='<target>'

curl -v "https://<tenant_name>api.siffletdata.com/api/v1/metadata/dbt/$projectName/$target"  \
-F '[email protected]/manifest.json' \
-F '[email protected]/catalog.json' \
-F '[email protected]/run_results.json' \
-H "Authorization: Bearer $accessToken"

• accessToken: is the value you previously generated and stored
• projectName: same value as in part 1, the name of your dbt project (the "name" in your dbt_project.yml file)
• target: same value as in part 1, the target value of the profile that corresponds to your project (the "target" in your profiles.yml)
• tenant_name: if you access to Sifflet with <https://abcdef.siffletdata.com>, then your tenant would be abcdef