Dashboards

Insights dashboards show metrics about your data mesh, such as data products, reuse, access requests and Time to Insights, as numbers, charts and tables. You arrange them in a visual editor or write them as YAML.

How it fits together

• A dashboard is a YAML document in the Perses dashboard format, the format Dash0 and the Perses operator use. It lists panels and places them on a grid.
• A panel shows one or more metrics in a chart: a number, a line over time, bars, a table, a pie, a funnel, a scatter plot, or Markdown text. The Chart Gallery shows every chart with its YAML.
• A metric is defined in a semantic model, an Apache Ossie document under Governance → Insights → Semantic Models. A panel refers to a metric as <semantic model>.<metric>, for example entropy_data_marketplace_analytics.reuse_rate.
• The metrics read the datasets of Entropy Data Marketplace Analytics: the data products, output ports and access agreements of your organization as fact tables, with its domains and teams. Data Contract on the Insights page shows the data contract of these datasets, with every table and column. The same datasets can also be downloaded for your own BI tools, see Marketplace Analytics datasets.

Every organization has the built-in Overview dashboard. Further dashboards appear as tabs next to it.

Who sees and edits dashboards

A dashboard is visible to everyone in the organization unless it names teams in entropy-data.com/visible-to (in the visual editor: Visible to). Tabs of restricted dashboards carry a lock.

Reading a dashboard

• Filters. A dashboard can offer a Domain filter, a Team filter, or both. A domain includes its subdomains, a team its subteams. The filters apply to every panel.
• Period. Viewers choose Last 30 days, Last 90 days or This year. Charts over time show weeks for the last 30 or 90 days and months for this year. The dashboard opens with its duration.
• Today. Some metrics have no history, such as contract coverage. Their panels say Today and ignore the period. Running totals, such as the number of data products, also say Today in a number or bar chart.
• Details. The table icon on a panel opens Metric details: the SQL with the dashboard's filters filled in, the first rows of each dataset the metric reads, and Download CSV with all of them. The info icon explains how the metric is calculated.

Create a dashboard

Choose Edit → New dashboard on the Insights page. The editor has two modes; switch between them with Visual and YAML at the top.

Visual editor

• Name, Description and Visible to describe the dashboard.
• Add metric opens a panel to choose the Metric, how to Show as (Number, Over time, By group, Table, Pie chart, Funnel, Scatter chart), what to Group by or Color by, the Width and, for bar charts, the direction of the Bars. Options that don't work for the metric are disabled with the reason, for example Over time for a metric that is measured today.
• Add text adds a Markdown panel, for example to explain the takeaways of a section.
• Drag a widget to move it, or use the arrows on the widget.

Saving from the visual editor writes the dashboard's YAML anew, so comments in the YAML are lost.

YAML editor

The YAML editor validates as you type. It completes metric references and groupBy values, lists every problem with its line, and previews the dashboard on the right. A dashboard with problems can't be saved.

To change an existing dashboard, open it and choose Edit → Edit dashboard or Edit → Edit YAML.

A complete example

This dashboard has a domain filter, three numbers in a summary section and two charts in a collapsible section:

apiVersion: "perses.dev/v1alpha1"
kind: "PersesDashboard"
metadata:
  name: "reuse"
  annotations:
    entropy-data.com/owner: "governance-team"
    entropy-data.com/visible-to: "governance-team, platform-team"
spec:
  display:
    name: "Reuse"
    description: "How much of our data other teams use."
  duration: "90d"
  variables:
    - kind: "ListVariable"
      spec:
        name: "domain"
        allowAllValue: true
        plugin:
          kind: "EntropyDataDomainVariable"
  panels:
    data_products:
      kind: "Panel"
      spec:
        plugin:
          kind: "StatChart"
          spec: {}
        queries:
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.data_products"
    published:
      kind: "Panel"
      spec:
        plugin:
          kind: "StatChart"
          spec: {}
        queries:
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.marketplace_data_products"
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.data_products"
    reuse_rate:
      kind: "Panel"
      spec:
        plugin:
          kind: "StatChart"
          spec: {}
        queries:
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.reuse_rate"
    reuse_rate_by_team:
      kind: "Panel"
      spec:
        display:
          name: "Reuse rate by team"
        plugin:
          kind: "BarChart"
          spec: {}
        queries:
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.reuse_rate"
                  groupBy: "team"
    funnel:
      kind: "Panel"
      spec:
        display:
          name: "From creation to reuse"
        plugin:
          kind: "FunnelChart"
          spec: {}
        queries:
          - kind: "TimeSeriesQuery"
            spec:
              plugin:
                kind: "EntropyDataMetricQuery"
                spec:
                  metric: "entropy_data_marketplace_analytics.data_products"
                  groupBy: "data_product.lifecycle_stage"
  layouts:
    - kind: "Grid"
      spec:
        display:
          title: "Summary"
        items:
          - x: 0
            "y": 0
            width: 8
            height: 3
            content:
              $ref: "#/spec/panels/data_products"
          - x: 8
            "y": 0
            width: 8
            height: 3
            content:
              $ref: "#/spec/panels/published"
          - x: 16
            "y": 0
            width: 8
            height: 3
            content:
              $ref: "#/spec/panels/reuse_rate"
    - kind: "Grid"
      spec:
        display:
          title: "Details"
          collapse:
            open: true
        items:
          - x: 0
            "y": 0
            width: 24
            height: 8
            content:
              $ref: "#/spec/panels/reuse_rate_by_team"
          - x: 0
            "y": 8
            width: 24
            height: 9
            content:
              $ref: "#/spec/panels/funnel"

Replace governance-team and platform-team with IDs of your own teams; a team ID that doesn't exist is a problem. Quote the key "y": some YAML parsers read a bare y as the boolean true.

Dashboard reference

metadata

namespace, labels and other annotations are allowed, so a dashboard exported from Kubernetes or Dash0 validates, but Entropy Data ignores them.

spec

Filters

Each entry of variables adds one filter above the dashboard. A dashboard has at most one filter of each kind.

variables:
  - kind: "ListVariable"
    spec:
      name: "domain"
      allowAllValue: true
      plugin:
        kind: "EntropyDataDomainVariable"
  - kind: "ListVariable"
    spec:
      name: "team"
      allowAllValue: true
      plugin:
        kind: "EntropyDataTeamVariable"

name must be unique among the variables. A metric whose dataset has no domain or team can't be filtered by it; its panel says so.

Panels

panels:
  reuse_rate_by_team:
    kind: "Panel"
    spec:
      display:
        name: "Reuse rate by team"
      plugin:
        kind: "BarChart"
        spec:
          orientation: "vertical"
      queries:
        - kind: "TimeSeriesQuery"
          spec:
            plugin:
              kind: "EntropyDataMetricQuery"
              spec:
                metric: "entropy_data_marketplace_analytics.reuse_rate"
                groupBy: "team"

The Chart Gallery describes each chart, how many metrics it takes and which options it has:

Queries

A panel with several queries reads groupBy and stackBy from the first one.

Grouping

groupBy and stackBy take one of:

A metric can group by a field only when it reads that one dataset. The YAML editor lists the possible values, and a wrong one is a problem that names the alternatives. Bar charts, tables and pie charts show the 200 largest groups.

Links

A panel can link to pages with more about what it shows, such as the Data Product Score leaderboard behind a score chart:

panels:
  score_by_team:
    kind: "Panel"
    spec:
      display:
        name: "Average Data Product Score by team"
      links:
        - name: "Data Product Score"
          url: "/governance/dataproducts/score"
      plugin:
        kind: "BarChart"
        spec: {}
      queries:
        - kind: "TimeSeriesQuery"
          spec:
            plugin:
              kind: "EntropyDataMetricQuery"
              spec:
                metric: "entropy_data_marketplace_analytics.average_score"
                groupBy: "team"

Each link gets an icon next to the panel's title, and the title itself opens the first link.

Layout

Each entry of layouts is a grid of 24 columns. A grid with a title gets a heading; with collapse, viewers can collapse and expand it.

layouts:
  - kind: "Grid"
    spec:
      display:
        title: "Access requests"
        collapse:
          open: false
      items:
        - x: 0
          "y": 0
          width: 12
          height: 8
          content:
            $ref: "#/spec/panels/pending_access_requests_by_team"

Panels move up to close gaps between rows, and a panel that overlaps another moves below it, as in Perses. Every panel must be placed by a grid; one that isn't is a problem. Good sizes are width 6 and height 3 for four numbers in a row, width 12 and height 8 for two charts side by side, and width 24 and height 9 for a funnel.

Metrics

The built-in semantic model entropy_data_marketplace_analytics is maintained by Entropy Data. Its metrics:

Dated by is the timestamp that places a row in the period. A running total counts every row up to each point in time, open at each point in time every request still waiting then. Metrics measured today can't be shown over time. A scatter chart needs a dated metric that reads one dataset and is not a running total or open requests. Semantic Models lists each metric with its description and formula.

Besides domain, top-level-domain and team, the metrics group by these fields of their dataset:

Some fields list their values in order, which gives the steps of a funnel and the order of stacked bars:

The built-in model can't be changed. To change a built-in metric, open it under Semantic Models and choose Duplicate to change, then refer to the metric in your copy. When you rename or remove a metric that a dashboard shows, the model editor asks whether to point the panels at another metric, remove them, or keep them with a notice.

Problems

The editors and the API check the whole dashboard and report each problem with its line, for example:

• There is no metric entropy_data_marketplace_analytics.reuse.
• Can't show Reuse rate over time: it is measured today, so it has no history. Use a StatChart, BarChart, Table or PieChart instead.
• Can't group Reuse rate by data_product.score: dashboards can't group by data_product.score. Group by domain, top-level-domain, team, data_product.specification_type, …
• No layout places the panel reuse_rate_by_team, so the dashboard doesn't show it.
• The panel reaches past the grid's 24 columns: x plus width can be 24 at most.

A dashboard with problems can't be saved. A saved dashboard can get problems later, for example when a metric it shows is removed; the dashboard then lists them under Some widgets can't be shown.

The Overview dashboard

The built-in Overview has the ID overview and shows the summary, data products, access requests and trust. Its Data Product Score by team table links to the Data Product Score leaderboard. Editing it saves your own copy for the organization, which replaces the built-in one. Reset to the built-in Overview in the editor, or DELETE /api/insights/dashboards/overview, brings the built-in one back.

The Overview's YAML is a good starting point for a dashboard of your own: open Edit → Edit YAML on the Overview and copy it.

API

Dashboards can live in a Git repository and be deployed through the API. The endpoints are experimental.

metadata.name in the body must match the ID in the path. An invalid dashboard is refused with 422 and every problem with its line. The location-html response header links to the dashboard in the web app.

# Download a dashboard as YAML
curl -H "x-api-key: $API_KEY" \
  https://api.entropy-data.com/api/insights/dashboards/reuse/dashboard.yaml > reuse.dashboard.yaml

# Deploy it, converting the YAML to JSON with yq
yq -o=json reuse.dashboard.yaml | curl -X PUT \
  -H "x-api-key: $API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- \
  https://api.entropy-data.com/api/insights/dashboards/reuse

Semantic models have the same endpoints under /api/insights/semantic-models, with GET /api/insights/semantic-models/{name}/model.yaml for the YAML. Reading them takes an organization key or the user key of an organization owner or governance group member. The built-in entropy_data_marketplace_analytics can be read, but changing or deleting it is refused with 422. A PUT that removes or renames metrics that dashboards show is refused with 409 and names them; send it again with ?removedMetrics=remove to take those panels off the dashboards, or ?removedMetrics=keep to leave them with a notice.

See the OpenAPI reference for the request and response formats.

Marketplace Analytics datasets

The datasets the metrics read can be downloaded with an organization API key, to analyze them in your own BI tools. These endpoints are experimental and don't need the Insights v2 preview.

Compatibility with Perses

Entropy Data reads the perses.dev/v1alpha1 PersesDashboard resource, the format the Perses operator and Dash0 exchange. It supports Grid layouts, ListVariable variables with the Entropy Data plugins, queries with the EntropyDataMetricQuery plugin, and panel links. Perses v1alpha2, which moved the dashboard into spec.config, and Prometheus or other data source queries are not supported.