Databricks Integration Gets a Major Upgrade: Field-Level Lineage and More!

We're excited to announce significant improvements to our Databricks integration, bringing deeper context and visibility to your data assets. You can now leverage Databricks tags directly within Sifflet and better trace dependencies with our new field-level lineage for Databricks.

What's New?

• Databricks Tags Now in Sifflet: We now automatically retrieve tags associated with your tables and columns in Databricks and display them in the Sifflet catalog. This enriches your data discovery and governance workflows, allowing you to use your existing Databricks tagging strategy to organize and find assets within Sifflet.

  

• Field-Level Lineage: Our data lineage graphs for Databricks now go deeper. We've added support for field-level (or column-level) lineage, allowing you to see exactly how individual fields are connected across your data pipeline. This granularity is crucial for precise impact analysis and root cause investigation.

  

Coming Soon

This is just the beginning. We're continuing to improve our Databricks integration and will be releasing more enhancements (like Databricks Workflows support) in the near future. Stay tuned!

How to Get Started

To enable these new features, you'll need to ensure your Databricks environment is configured to allow Sifflet access to system tables. For detailed instructions, please see our updated Databricks integration documentation.

App version: v501