Data Products

Overview

The term 'data product' has a range of definitions across the industry. Within Sifflet, we define a Data Product as a logical grouping of multiple data assets (like datasets, dashboards, and transformation tasks) that collectively serve a common business use case. This approach provides a flexible framework for you to implement your own definition of a data product while gaining powerful new capabilities for monitoring, ownership, and governance.

By grouping assets into Data Products, you can move from monitoring individual tables and dashboards to monitoring the health of an entire end-to-end pipeline or business-critical initiative at a glance.

Key Benefits of Using Data Products:

• Holistic Monitoring: Understand the health of an entire data pipeline (e.g., "Product Analytics Pipeline") instead of tracking dozens of individual assets separately.
• Clear Ownership: Assign clear owners to specific business use cases, simplifying reporting and accountability at the team level.
• Value-Driven Insights: Directly connect technical data assets to the business value they generate (e.g., "This Reverse ETL pipeline costs $X and supports $Y in sales operations").
• Governance at Scale: Standardize alert notifications for your critical use cases by defining default notification channels at the product level. This ensures all alerts from the data product's assets are consistently routed to the right stakeholders, streamlining communication and incident response.

Creating and Managing Data Products

System Admins and Domain Editors can create and manage Data Products. All other users can view Data Products, subject to their existing domain-level permissions.

Creating a Data Product

1. Navigate to Data Products from the main menu in the Sifflet platform.
2. Click the New Data Product button to open the configuration page.
3. Fill in the Data Product Details:
   • Domain: The Domain this product belongs to. Every data product belongs to one domain in Sifflet, but you can also use the default "All" domain.

     📘

     Once the data product is created, its domain is no longer changeable.

   • Name: A unique and descriptive name for the Data Product (e.g., "Marketing Campaigns Analysis").
   • Description: A detailed explanation of the use case this product serves.
   • Owners: The Sifflet users responsible for this Data Product.
   • Tags: Add relevant tags for filtering and organization.
   • SLA: Define the Service Level Agreement for the data delivered by this product (e.g., "Data must be updated by 08:00 AM daily").
4. Add Assets: In the second section, select the assets (datasets, dashboards, pipelines, etc.) that constitute this Data Product. You can either select the assets one by one or leverage the filtering conditions for dynamic at-scale selection.
5. Configure Notifications: In the final section, define the default notification channels (e.g., Slack, Email) for this Data Product.

   📘

   All alerts and notifications related to assets within the product will be sent to these channels by default. You can still override this for specific assets or monitors.

The Data Products List

The main Data Products page provides a comprehensive list of all products in your environment. The table includes:

• Data Product Name: The name of the product.
• Status: The aggregated health status of the product.
• Number of Assets: The count of assets included in the product.
• Owner(s), SLA, Description: Key metadata for quick reference.

Understanding Data Product Status

The health of a Data Product is displayed as a summary of the health of all its underlying assets. Instead of a single status, it provides a breakdown of assets by their most critical incident level.

This summary allows you to immediately gauge where issues are occurring and how severe they are:

• 🔴 Urgent Incidents: Shows the total count of assets within the Data Product that are currently affected by one or more urgent incidents.

• 🟡 High-Risk Incidents: Displays the total count of assets within the Data Product that have active high-risk incidents.

• 🟢 No Incidents: Indicates that all assets, monitors, and jobs within the product are running as expected.

This breakdown helps you prioritize your team's focus by instantly showing the number of assets that require attention at each level of severity.

Viewing Data Product Details

Clicking on a Data Product from the list will open a side panel with two tabs:

1. Overview: Displays all the metadata you configured, such as owners, SLA, and tags.
2. Assets: Lists all the data assets that are part of the Data Product. Clicking on an asset will take you to its dedicated page.

Integration Throughout Sifflet

To make Data Products a core part of your workflow, they are integrated into other areas of the platform:

• Asset Page: For any asset that belongs to a Data Product, you will see a "Data Products" attribute in its Overview tab, linking directly to the relevant product(s).