GitLab

Sifflet brings its lineage into your GitLab merge requests. When a merge request changes dbt models, the dbt Impact Analysis GitLab component asks Sifflet which tables and dashboards depend on these models, and posts the list as a merge request comment, so reviewers see the impact of a change before it reaches production.

Sifflet doesn't connect to GitLab and doesn't import anything from it: the component runs in your GitLab CI/CD pipelines and calls the Sifflet API.

Permissions Required

The integration uses two tokens: a Sifflet access token, which the component uses to read lineage from Sifflet, and a GitLab access token, which the component uses to read the merge request and post the comment.

The api scope gives read and write access to the GitLab API: the component only uses it to read the merge request and repository files, and to add a comment. Use a project access token to limit it to the repository.

The component sends Sifflet the names of the changed dbt models and of their dbt projects, and the path of the merge request (<PROJECT_PATH>/-/merge_requests/<NUMBER>), which Sifflet records with each analyzed model. It doesn't send any code.

Requirements

Required Roles for Setup

• In GitLab, the Maintainer role on the project, to create a project access token and CI/CD variables, and to edit the pipeline.
• In Sifflet, the Admin role to create the access token. See Access Control.

Supported Versions and Editions

• GitLab.com and GitLab self-managed. For self-managed instances, set the gitlab_server_url input.
• GitLab CI/CD components can only be included from the same GitLab instance. On a self-managed instance, mirror the component project to your instance and include it from there.
• Runners with a Docker-based executor, since the component runs in a container image.

Network Access

The connection goes from your GitLab runners to Sifflet:

• The runner must reach your Sifflet API at https://<TENANT>.siffletdata.com/api, your GitLab instance API, and the public.ecr.aws registry to pull the component image.
• If your Sifflet instance restricts API access to known IP ranges, run the job on runners whose IP addresses are allowed.

Other Prerequisites

• A dbt Core or dbt Cloud source in Sifflet for each dbt project of the repository. The name in the dbt_project.yml file must match the dbt project name known to Sifflet.
• Each dbt project of the repository has a dbt_project.yml file in the directory of its models or in a parent directory.
• Merge request pipelines: by default, the component job only runs in pipelines triggered by merge requests.

Connect Sifflet to GitLab

Step 1: Create a Sifflet Access Token

1. In Sifflet, go to Settings > Access Tokens and click New Access Token.
2. Enter a name (for example gitlab-dbt-impact-analysis), select the Viewer role, and click Generate.
3. Copy the token. Sifflet shows it only once. It expires after 24 months by default: see Access Tokens to set another expiration and to rotate it.
4. In GitLab, go to Settings > CI/CD > Variables in the project and add a variable named SIFFLET_API_TOKEN with the token as its value. Select the Masked and hidden visibility (see Mask a CI/CD variable). If you use an external secret manager with GitLab, you can store the token there instead.

Step 2: Create a GitLab Access Token

1. In the GitLab project, go to Settings > Access tokens and click Add new token (see Project access tokens).
2. Enter a name (for example sifflet-dbt-impact-analysis), select the Developer role and the api scope, set an expiration date, and click Create project access token.
3. Copy the token. GitLab shows it only once, and it stops working at its expiration date: create a new one and update the variable before then.
4. Go to Settings > CI/CD > Variables and add a variable named SIFFLET_GITLAB_TOKEN with the token as its value, with the Masked and hidden visibility.

Step 3: Add the dbt Impact Analysis Component to Your Pipeline

1. Add the following to the .gitlab-ci.yml file of your repository. Replace <TENANT> with the name of your Sifflet instance. The component is also listed in the GitLab CI/CD Catalog.

   YAML

   include:
     - component: $CI_SERVER_FQDN/siffletdata/dbt-impact-analysis-component/dbt-impact-analysis-component@v1
       inputs:
         gitlab_token: $SIFFLET_GITLAB_TOKEN
         sifflet_instance_url: https://<TENANT>.siffletdata.com
         sifflet_api_token: $SIFFLET_API_TOKEN
         impacted_asset_types: 'DASHBOARD, DATASET'
         stage: test
   
   stages: [test]

   If your pipeline already defines stages, keep your own list and set the stage input to one of its stages.

2. Adjust the inputs:

   Input	Required	Description	Example
   gitlab_token	Yes	The CI/CD variable containing the GitLab access token created in step 2	$SIFFLET_GITLAB_TOKEN
   sifflet_instance_url	Yes	The URL of your Sifflet instance, without /api and without a trailing slash	https://acme.siffletdata.com
   sifflet_api_token	Yes	The CI/CD variable containing the Sifflet access token created in step 1	$SIFFLET_API_TOKEN
   gitlab_server_url	No	The URL of your GitLab self-managed instance. Empty (default): https://gitlab.com.	https://gitlab.example.com
   impacted_asset_types	No	The types of downstream assets to list in the report, separated by a comma and a space: DATASET, DASHBOARD. Empty (default): all of them.	'DASHBOARD, DATASET'
   impacted_tags	No	Only list downstream assets that have at least one of these Sifflet tags, separated by a comma and a space. Empty (default): no tag filter.	'pii, finance'
   stage	No	The pipeline stage of the job. Default: test.	test
   rules	No	The if condition that decides when the job runs. Default: $CI_PIPELINE_SOURCE == 'merge_request_event'.	$CI_PIPELINE_SOURCE == 'merge_request_event'

3. Commit the file to your default branch.

The component now runs in every merge request pipeline. Its job is allowed to fail, so it never blocks a merge request. To check the setup, open a merge request that changes a dbt model: the component posts a Sifflet Impact Report comment. For what the report contains and how the component finds the changed models, see dbt Impact Analysis GitLab Component.

Troubleshooting

The Report Says a Model Is Not Found in Sifflet

• Symptom: the comment contains <MODEL_NAME> not found in Sifflet.
• Cause: Sifflet doesn't find exactly one dbt model with this name in a dbt project whose name matches the name of the closest dbt_project.yml file. Either the model doesn't exist in Sifflet yet (new model, or dbt source not refreshed since), or several dbt sources in Sifflet contain the same project and model.
• Fix:
  1. Check that the name in dbt_project.yml matches the project of your dbt source in Sifflet: see Other Prerequisites.
  2. For a new model, run your dbt job and refresh the dbt source in Sifflet, then push a new commit to the merge request.
  3. If several dbt sources in Sifflet contain the same dbt project (for example one source per dbt target), the component can't choose between them and skips the model: see Limitations.

The Job Fails Without Posting a Comment

• Symptom: the dbt-impact-analysis-job job fails with An error occurred: Invalid Gitlab token for repository <PROJECT_PATH>, or with Error occurred when calling the Sifflet API for DBT model <MODEL_NAME> and status code 401 or 403.
• Cause: the GitLab token is wrong, expired, or lacks the api scope or the Developer role, or the Sifflet access token is wrong, expired, or doesn't have the Viewer role.
• Fix:
  1. Check the GitLab token and the SIFFLET_GITLAB_TOKEN variable: see Step 2.
  2. Check the SIFFLET_API_TOKEN variable, or create a new Sifflet access token: see Step 1.