Overview

Sifflet comes with a large number of built-in integrations spanning your entire data pipelines’ stack, from ingestion to consumption. Built-in integrations automatically collect metadata and lineage information and make it available in the Data Catalog.

In some cases though, Sifflet cannot retrieve assets metadata and lineage information, for instance because the built-in integration does not exist yet, because Sifflet cannot connect to your closed environment or does not have enough access to retrieve and compute the lineage, etc.

For these cases, you can programmatically declare data pipeline assets and lineage to ensure an end-to-end observability experience. You can for instance:

• Reflect data sources such as CRMs (e.g. Salesforce, Pipedrive, etc.), ERPs (e.g. SAP, etc.), marketing automation tools (e.g. HubSpot, Marketo, etc.), etc.
• Create assets from your preferred BI tools (e.g. Metabase, MicroStrategy, etc.)
• Surface key API calls and custom scripts
• Display machine learning models
• Catalog and reflect orchestrators on lineage
• Show custom data applications
• Link assets Sifflet cannot compute lineage for

Getting Started

To programmatically declare assets and their lineage, use the declarative assets & lineage framework. This framework uses two API endpoints that allow you to declare three main object types.

Assets

Assets are the smallest entity you can currently declare. They correspond to data-related components such as a dashboard, a table, a machine learning model, etc. Declared assets show up like regular assets in your Data Catalog.

Sources

Sources are a logical way to group your assets. You can for instance have a source corresponding to your staging assets and another one corresponding to your production assets. Declared sources show up alongside Sifflet sources. Declaring sources is optional, as Sifflet creates sources automatically from asset URIs.

Workspaces

Workspaces are the highest entity you can declare, they contain declared sources and assets. Workspaces are isolated from each other. Workspaces allow you to manage a collection of declared sources and assets without interfering with collections owned by different teams that would exist in other workspaces.

Prerequisites

See API prerequisites

Manage Declarative Assets & Lineage

Declare Assets & Lineage

To declare assets & lineage, use the POST /v1/assets/sync endpoint.

You can declare:

• Assets only, using the assets array of objects
• Lineage links only, using the lineages array of objects
• Both assets and lineage links
  • Using the assets array of objects that contains a lineages object
  • Using the assets array of objects and the lineages array of objects
  • Using a mix of the two above options

Sifflet is equipped with a set of technology icons, which allows you to have your declarative assets show up with the proper technology icon in the Sifflet application if you use the appropriate technology name when defining your assets' URIs. The list of expected technology names for URIs is available here.

Declare Sources

Declaring sources via the sources array of objects is optional. Declaring a source is useful if you want to attach it specific metadata (e.g. a name, a description, etc.). If no source is declared, Sifflet automatically adds declared assets to sources using declared assets URIs.

Modify Declarative Assets & Lineage

Once assets, lineage links, and sources are created in a workspace, they become read-only in the user interface (UI). You can only modify or delete them through the declarative assets & lineage framework. This will evolve in the future to allow you to manage your declarative assets & lineage both programmatically and from the UI.

To modify declarative assets and sources, update your JSON payload and push the new version to the POST /v1/assets/sync endpoint.

Delete Declarative Assets & Lineage

To delete declarative assets & lineage, use the DELETE /v1/assets/{name} endpoint.