Documentation
HomeChangelogBlog

Practical Example

In this guide, you learn how to quickly push assets into the Sifflet Data Catalog. To follow this guide, you need:

  • an access token with the Admin role. How to create an access token in Sifflet
  • a way of performing API requests in your work environment. This guide provides examples using a curl command line input, but you can perform API requests using other tools like Postman.

Creation of a Workspace and Assets

In this section, you create a Sifflet workspace and add two catalog assets to this workspace.

  • Create a file representing your workspace definition in one of your folders.
    In this example, the guide uses a demo folder that contains a file named my_first_workspace.json. You can do the same in your environment.

  • Once your file is created, open it in a text editor and start defining your workspace content. For increased readability, use a text editor that understands the JSON standard (this is optional).
    First, name your workspace. This guide calls it “MyFirstDeclaredWorkspace”. To do so, write the following inside your my_first_workspace.json file:
{
  "workspace": "MyFirstDeclaredWorkspace"
}

  • Workspace names are unique inside a Sifflet instance, so you can also add custom information to the name to make sure no one else uses it. If you are afraid of overriding somebody’s workspace content when pushing yours because you are using an already existing workspace name, you can set the parameter dryRun to true to examine the changes that Sifflet performs when synchronizing your workspace, without actually committing any change. Once you are sure everything is fine, perform the same API request with the dryRun setting on false.

  • Now add some assets to your workspace. To do so, populate an array named assets with elements. Each element is identified using a unique identifier named uri. It also needs to have a mandatory primary type. For more details about URIs in general and in Sifflet, read the dedicated documentation section.
    In this example, you add assets from a MongoDB test database to your workspace. The MongoDB instance is hosted at the sifflet-mongodb-test.eu-west-1.com address, on port 27017, and the collections to put in the workspace are located inside a database named testDB on this instance. Their names are sampleCollection1 and minimalInputCollection2. Their URIs are:
    - mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1
    - mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.minimalInputCollection2
    Give the primary type “Dataset” to your collections (currently available types are “Dataset”, “Dashboard”, “Pipeline”, “MlModel” and “Generic”).
    Your my_first_workspace.json file now looks like this:

    {
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "type": "Dataset"
      },
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.minimalInputCollection2",
      "type": "Dataset"
    }
  ]
}

You can push your workspace now, but first add a few more details to it.

  • Make some small adjustments to your file to surface more information when pushing the workspace. This example focuses on the first asset of the file, so you can compare the final result with the minimal input used in the second asset.
    • First, add a name to the asset so the displayed name is more readable. In this example, it is “Sample Dataset 1”.
    • Then add a subType: in MongoDB it is a collection, so call it “collection” to give Sifflet more context.
    • Add a description as well: “Sample of document collection stored in MongoDB for test purpose”
    • Finally, add an href pointing to your collection URL (the example payload points to the MongoDB documentation, but you can add a real link).
  • The content of your file now looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    },
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.minimalInputCollection2",
      "type": "Dataset"
    }
  ]
}
  • You are now ready to sync your new workspace with your Sifflet instance. Open a terminal (or another way to perform an API request) and type (from the folder containing your demo folder):
curl --request POST \
     --url https://{your_sifflet_tenant_host_here}.siffletdata.com/api/v1/assets/sync?dryRun=false \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {your_access_token_here}'  \
     --header 'content-type: application/json' \
     --data '@demo/my_first_workspace.json'
  • In this case, the parameter dryRun is set to false directly, which means Sifflet creates the assets and workspace. If you want to check what happens because you are afraid to erase somebody else's workspace, set it to true. You get a report of the changes expected after writing the assets. When you are ready, perform the request again with dryRun set to false.
  • You can now go to your Sifflet instance UI and browse the Data Catalog. Your two assets appear there. If you navigate to the Integration section, you can find the source that Sifflet created to contain your new catalog assets.

Updating Your Workspace by Adding and Removing Assets

Now that you have created your workspace, update it by adding some assets and removing others.

  • First, remove the second MongoDB asset from your catalog. To do so, remove its reference inside the file my_first_workspace.json. It now looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    }
  ]
}
  • This example uses a Metabase integration that contains a couple of assets to push to Sifflet. Add them to your workspace.
    • The Metabase instance is located at sifflet-metabase-test.eu-west-1.com on port 8443. The assets are ordered inside a Metabase collection (a folder) named SampleData. You can use that information to craft the asset URIs based on their names.
    • The first asset is named “model1”. Use the “Generic” type for it, as no non-generic type fits here. Label its subType “model”.
    • The second asset is a dashboard named “dashboard1”. Use the “Dashboard” type for it.
    • Add these fields to your JSON file. It now looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1",
      "name": "Sample Model 1",
      "type": "Generic",
      "subType": "model",
      "href": "https://www.metabase.com/docs/latest/data-modeling/models",
      "description": "Sample of a Metabase model for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.dashboard1",
      "name": "Sample Dashboard 1",
      "type": "Dashboard",
      "href": "https://www.metabase.com/docs/latest/dashboards/introduction",
      "description": "Sample of a Metabase dashboard for test purpose"
    }
  ]
}
  • Sync your workspace again, using the same API request as before:
curl --request POST \
     --url https://{your_sifflet_tenant_host_here}.siffletdata.com/api/v1/assets/sync?dryRun=false \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {your_access_token_here}'  \
     --header 'content-type: application/json' \
     --data '@demo/my_first_workspace.json'
  • If you browse the Sifflet Data Catalog again, you find your new assets. The MongoDB collection you removed no longer appears in the Data Catalog. If you browse the Integration page, you can find the source containing your new Metabase asset.

Adding Sources Information

You can improve the information displayed on the Integration page by adding this information to your workspace payload. This part of the guide shows how to do so.

  • First, add a sources field inside your my_first_workspace.json file. It is an array, same as the assets field. It looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1",
      "name": "Sample Model 1",
      "type": "Generic",
      "subType": "model",
      "href": "https://www.metabase.com/docs/latest/data-modeling/models",
      "description": "Sample of a Metabase model for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.dashboard1",
      "name": "Sample Dashboard 1",
      "type": "Dashboard",
      "href": "https://www.metabase.com/docs/latest/dashboards/introduction",
      "description": "Sample of a Metabase dashboard for test purpose"
    }
  ],
  "sources": [

  ]
}
  • Now add one entry inside your file for each of the sources on the Integration page. Each source is identified by its uri field, same as assets. You can write the URI of a source based on the URI of one of the assets it contains.
    • For example, the MongoDB source has mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB as uri. It corresponds to the prefix of the URI of the asset it contains, mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1.
    • Add a name and a description to your sources. Your file now looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1",
      "name": "Sample Model 1",
      "type": "Generic",
      "subType": "model",
      "href": "https://www.metabase.com/docs/latest/data-modeling/models",
      "description": "Sample of a Metabase model for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.dashboard1",
      "name": "Sample Dashboard 1",
      "type": "Dashboard",
      "href": "https://www.metabase.com/docs/latest/dashboards/introduction",
      "description": "Sample of a Metabase dashboard for test purpose"
    }
  ],
  "sources": [
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData",
      "name": "Metabase Sample Data Collection",
      "description": "This is a sample datasource representing a Metabase collection"
    },
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB",
      "name": "MongoDB test database",
      "description": "This is a sample datasource representing a MongoDB database"
    }
  ]
}
  • Sync your workspace again, using the same API request as before:
curl --request POST \
     --url https://{your_sifflet_tenant_host_here}.siffletdata.com/api/v1/assets/sync?dryRun=false \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {your_access_token_here}'  \
     --header 'content-type: application/json' \
     --data '@demo/my_first_workspace.json'
  • You can navigate through the Integration page and see the elements you have pushed; Sifflet renamed the previous sources on the Integration page and inside the Data Catalog. If you browse a specific source page, you also find the description you set.

Add Lineage Between Your Assets

You can apply lineage to your assets using the same API. In this example, the Metabase model is built from the MongoDB table, and a Metabase dashboard uses the information of the model.

  • To represent this link in Sifflet, add a section named lineages to your file and add tuples representing each link.
  • Each tuple has an upstream element and a downstream element (or a parent and a child, or a source and a target, depending on the vocabulary you are used to). Put the upstream element in the field from and the downstream element in the field to. Those fields have to contain a URI, using the same format as in the assets definition.
  • The resulting file looks like this:
{
  "workspace": "MyFirstDeclaredWorkspace",
  "assets": [
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "name": "Sample Dataset 1",
      "type": "Dataset",
      "href": "https://www.mongodb.com/docs/manual/core/databases-and-collections",
      "subType": "collection",
      "description": "Sample of document collection stored in MongoDB for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1",
      "name": "Sample Model 1",
      "type": "Generic",
      "subType": "model",
      "href": "https://www.metabase.com/docs/latest/data-modeling/models",
      "description": "Sample of a Metabase model for test purpose"
    },
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.dashboard1",
      "name": "Sample Dashboard 1",
      "type": "Dashboard",
      "href": "https://www.metabase.com/docs/latest/dashboards/introduction",
      "description": "Sample of a Metabase dashboard for test purpose"
    }
  ],
  "sources": [
    {
      "uri": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData",
      "name": "Metabase Sample Data Collection",
      "description": "This is a sample datasource representing a Metabase collection"
    },
    {
      "uri": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB",
      "name": "MongoDB test database",
      "description": "This is a sample datasource representing a MongoDB database"
    }
  ],
  "lineages": [
    {
      "from": "mongodb://sifflet-mongodb-test.eu-west-1.com:27017/testDB.sampleCollection1",
      "to": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1"
    },
    {
      "from": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.model1",
      "to": "metabase://sifflet-metabase-test.eu-west-1.com:8443/SampleData.dashboard1"
    }
  ]
}
  • Sync your workspace again, using the same API request as before:
curl --request POST \
     --url https://{your_sifflet_tenant_host_here}.siffletdata.com/api/v1/assets/sync?dryRun=false \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {your_access_token_here}'  \
     --header 'content-type: application/json' \
     --data '@demo/my_first_workspace.json'
  • If you browse your Data Catalog and select the MongoDB asset, you can navigate to the lineage panel of the asset and see that your three assets are now linked:

Delete Workspace

To clean up, fully remove your workspace by making an API request to the dedicated deletion endpoint with the following command. The Sifflet server deletes your assets and your workspace and cleans up your sources.

curl --request DELETE \
     --url https://{your_sifflet_tenant_host_here}.siffletdata.com/api/v1/assets/MyFirstDeclaredWorkspace?dryRun=false \
     --header 'authorization: Bearer {yourtokenhere}'

You now know how to use the Sifflet API to push assets into the Data Catalog. For more information, check the other documentation sections.

Updated 5 days ago

Did this page help you?