Declarative Assets

Overview

Sifflet comes with a large number of built-in integrations spanning your entire data pipelines’ stack. Built-in integrations automatically collect metadata and lineage information and make it available in the Data Catalog.

In some cases though, you might want to programmatically declare some of your data pipeline assets to ensure an end-to-end observability experience:

  • Reflect data sources such as CRMs (e.g. Salesforce, Pipedrive, etc.), ERPs (e.g. SAP, etc.), marketing automation tools (e.g. HubSpot, Marketo, etc.), etc.
  • Create assets from your preferred BI tools (e.g. Metabase, MicroStrategy, etc.)
  • Surface key API calls and custom scripts
  • Display machine learning models
  • Catalog orchestrators jobs
  • Show custom data applications
  • Or even represent physical assets

Getting Started

Declarative Assets Framework

To programatically declare assets, you need to leverage the declared assets framework. This framework leverages two API endpoints that allow you to declare three main object types.

Assets

Assets are the smallest entity you can currently declare. They correspond to data-related components such as a dashboard, a table, a machine learning model, etc. Declared assets will show up like regular assets on your Data catalog.

Sources

Sources are a logic way to group your assets. You can for instance have a source corresponding to your staging assets and another one corresponding to your production assets. Declared sources will show up alongside Sifflet sources. Declaring sources is optional as it is done automatically from URI.

Workspaces

Workspaces are the highest entity you can declare, they contain declared sources and assets. Workspaces are isolated from each other. Workspaces allow you to manage a collection of declared sources and assets without interfering with collections owned by different teams that would exist in other workspaces.

Prerequisites

See API prerequisites

Manage Declared Assets

Declare Assets

To declare assets, you need to leverage the /v1/assets/sync endpoint.

Modify Declared Assets

Once sources and assets are created in a workspace, they become read-only on the User Interface (UI). Modifications or deletions are exclusively through the declared asset framework. This will evolve in the future to allow you to manage your declared assets both programmatically and from the UI.

To modify declared assets and sources, just update your JSON payload and push the new version to the /v1/assets/sync endpoint.

Deleted Declared Assets

To delete declared assets, you need to leverage the /v1/assets/{name} endpoint.