Data Products

Overview

The term 'data product' has a range of definitions across the industry. Within Sifflet, a data product is a logical grouping of multiple data assets (like datasets, dashboards, and transformation tasks) that collectively serve a common business use case. This approach provides a flexible framework for you to implement your own definition of a data product while gaining capabilities for monitoring, ownership, and governance.

By grouping assets into data products, you can move from monitoring individual tables and dashboards to monitoring the health of an entire end-to-end pipeline or business-critical initiative at a glance.

Key Benefits of Using Data Products:

• Holistic Monitoring: Understand the health of an entire data pipeline (e.g., "Product Analytics Pipeline") instead of tracking dozens of individual assets separately.
• Clear Ownership: Assign clear owners to specific business use cases, simplifying reporting and accountability at the team level.
• Value-Driven Insights: Directly connect technical data assets to the business value they generate (e.g., "This Reverse ETL pipeline costs $X and supports $Y in sales operations").
• Governance at Scale: Standardize alert notifications for your critical use cases by defining default notification channels at the product level. This ensures all alerts from the data product's assets are consistently routed to the right stakeholders, streamlining communication and incident response.

Creating and Managing Data Products

System Admins and Domain Editors can create and manage data products. All other users can view data products, subject to their existing domain-level permissions.

Creating a Data Product

1. Navigate to Data Products from the main menu in the Sifflet platform.

2. Click the New Data Product button to open the configuration page.

3. Fill in the Data Product Details:

   • Domain: The domain this product belongs to. Every data product belongs to one domain in Sifflet, but you can also use the default "All" domain.

     📘

     Once the data product is created, its domain is no longer changeable.

   • Name: A unique and descriptive name for the data product (e.g., "Marketing Campaigns Analysis").
   • Description: A detailed explanation of the use case this product serves.
   • Owners: The Sifflet users responsible for this data product.
   • Tags: Add relevant tags for filtering and organization.
   • SLA: Define the Service Level Agreement for the data delivered by this product (e.g., "Data must be updated by 08:00 AM daily").
   • Add Assets: In the second section, select the assets (datasets, dashboards, pipelines, etc.) that constitute this data product. You can either select the assets one by one or use the filtering conditions for dynamic at-scale selection.
   • Configure Notifications: In the final section, you can configure two types of notifications for your data product:
     • Status Change Notifications: Receive alerts whenever the status of the data product changes:
       • 🟢 Healthy → 🟡 At Risk
       • 🟢 Healthy → 🔴 Critical
       • 🟡 At Risk → 🔴 Critical
     • Summary Notifications: Receive periodic summaries of activity and events related to the data product:
       • Hourly: Runs at the beginning of every hour (e.g., 10:00, 11:00, 12:00).
       • Daily: Runs once per day at the selected hour.
       • Weekly: Runs once per week on the selected day and at the selected hour.

Message Example

Marketing Campaigns Analysis - 🔴 Critical

🔴 1 critical incident
- Campaign spend table not refreshed - 1 failing monitor

🔵 1 low incident
- Null values in campaign_id - 1 failing monitor

The Data Products List

The main Data Products page provides a list of all products in your environment. The table includes:

• Data Product Name: The name of the product.
• Status: Health status of the product based on incidents
• Number of Assets: The count of assets included in the product.
• Owners, SLA, Description: Key metadata for quick reference.

Understanding Data Product Status

The severity of its active incidents determines the health of a data product, which provides a high-level view of the overall reliability of the product.

Sifflet assigns each data product one of three statuses:

🟢 Healthy: No active incidents are affecting the data product.

🟡 At Risk: The data product has one or more active Low or Moderate incidents that may impact data quality or reliability, and no High or Critical incidents.

🔴 Critical: The data product has one or more active High or Critical incidents requiring immediate attention.

Viewing Data Product Details

Clicking a data product in the list opens a side panel with three tabs:

1. Overview: Displays all the metadata you configured, such as owners, SLA, and tags.
2. Assets: Lists all the data assets that are part of the data product. Clicking an asset takes you to its dedicated page.
3. Incident List: Quickly identify and investigate issues impacting your data products.

Integration Throughout Sifflet

Sifflet integrates data products into other areas of the platform:

• Asset Page: For any asset that belongs to a data product, you see a "Data Products" attribute in its Overview tab, linking directly to the relevant products.

  ---

Upstream alerts for data products learn more here.