Fivetran

Overview

Fivetran is an EL (Extract-Load) tool for ingesting data from various sources into your data platform. Integrating it with Sifflet allows you to gain end-to-end visibility into your data's journey, from source to destination.

After adding a Fivetran integration, the Sifflet lineage graph gets enriched with two types of nodes that represent your data’s journey before it even reaches the data platform:

• Fivetran connector nodes: These clickable nodes represent your Fivetran connectors, allowing you to get all the useful information about a given connector in one click. This information includes the connector’s source and destination, its current status, sync frequency, and the timestamp of the latest sync. Accompanied by the Sifflet metadata that you’re accustomed to (owners, description, and last metadata refresh), this panel gives you all the necessary information about a given connector.
• Source asset nodes: For every Fivetran connector, Sifflet will create nodes for the source assets from which Fivetran is extracting data, giving you full visibility on your data’s journey. These nodes give you end-to-end lineage within Sifflet.

Additionally, Fivetran connectors are accessible via the Sifflet catalog, allowing you to get a bird's-eye view of their key metadata. Similar to other assets, each connector also has its own asset page.

Integrating Fivetran with Your Sifflet Environment

The steps to integrate Fivetran with your Sifflet environment are the following:

1. If you have a self-hosted Sifflet deployment, update your environment to version 269 or later.

2. Generate a Fivetran API key and secret by following the instructions in the official Fivetran documentation. Once the API key and secret are created, write them down. You'll need to use them when setting up the integration in Sifflet.

   🚧

   Scoped API Keys

   When generating the API key and secret on Fivetran, keep in mind that it uses scoped API keys. This means that the API key has access scoped to the role-based access control (RBAC) permissions of its creator.

   Create a new user for Sifflet, give it access to the connectors you want to monitor, and then generate the API key using it. For more on scoped API keys, you can refer to the Fivetran documentation.

3. Connect to your Sifflet environment with an Admin account.

4. Navigate to Integrations > Credentials and add a new credential. The format of the credential is as follows:

   {
     "user":"Your Fivetran API Key",
     "password":"Your Fivetran API Secret"
   }

5. Navigate to the Sources page via the Integrations submenu and click the "New Source" button to create a Fivetran source.

6. Provide the following configuration values:

   1. Name: the name of the new source
   2. Host: If you're using a SaaS environment, the value should be https://api.fivetran.com. Otherwise, it should correspond to the API of your self-hosted environment.
   3. Credentials: pick the credential created in step 4
   4. Similarly to other sources, you can choose a source timezone and provide tags and a description for your Fivetran source.

7. Set the refresh frequency for your Fivetran source.

8. Click on Test connection to validate that Sifflet can interact with your Fivetran account, then Save to create the new source.

9. Once the source is created, you can manually trigger a first run.