Fivetran

Overview

Fivetran is an EL (Extract-Load) tool for ingesting data from various sources into your data platform. Integrating it with Sifflet allows you to gain end-to-end visibility into your data's journey, from source to destination.

After adding a Fivetran integration, the Sifflet lineage graph gets enriched with two types of nodes that represent your data’s journey before it even reaches the data platform:

• Fivetran connector nodes: These clickable nodes represent your Fivetran connectors, allowing you to get all the useful information about a given connector with just one click. This information includes the connector’s source and destination, its current status, sync frequency, and the timestamp of the latest sync. Accompanied by the Sifflet metadata that you’re accustomed to (owners, description, and last metadata refresh), this panel gives you, at a glance, all the necessary information about a given connector.
• Source asset nodes: For every Fivetran connector, Sifflet will create nodes for the source assets from which Fivetran is extracting data, giving you full visibility on your data’s journey. These nodes, which allow you to get a truly end-to-end lineage within Sifflet, represent one of the key differentiators of our Fivetran integration.

Additionally, Fivetran connectors are accessible via the Sifflet catalog, allowing you to get a bird's-eye view of their key metadata. Similar to other assets, each connector also has its own asset page.

Integrating Fivetran with your Sifflet environment

The steps to integrate Fivetran with your Sifflet environment are the following:

1. If you have a self-hosted Sifflet deployment, update your environment to version 269 or later.

2. Generate a Fivetran API key and secret by following the instructions in the official Fivetran documentation. Once the API key and secret are created, write them down. You'll need to use them when setting up the integration in Sifflet.

   🚧

   Scoped API Keys

   When generating the API key and secret on Fivetran, keep in mind that it uses scoped API keys. This means that the API key would have access that is scoped to the RBAC permissions of its creator.

   We recommend that you create a new user for Sifflet, give it access to the connectors you want to monitor, and then generate the API key using it. For more on scoped API keys, you can refer to the Fivetran documentation.

3. Connect to your Sifflet environment with an Admin account.

4. Navigate to the Credentials page via the Integrations submenu and add a new secret. The format of the secret should be as follows:

   {
     "user":"Your Fivetran API Key",
     "password":"Your Fivetran API Secret"
   }
   

5. Navigate to the Sources page via the Integrations submenu and click the "New Source" button to create a Fivetran source.

6. Provide the following configuration values:

   1. Name: the name of the new source
   2. Host: If you're using a SaaS environment, the value should be https://api.fivetran.com. Otherwise, it should correspond to the API of your self-hosted environment.
   3. Credentials: pick the secret created in step #4
   4. Similarly to other sources, you can choose a source timezone and provide tags and a description for your Fivetran source.

7. Set the refresh frequency for your Fivetran source.

8. Click on Test connection to validate that Sifflet can interact with your Fivetran account, then Save to create the new source.

9. Once the source is created, you can manually trigger a first run.