Data Product Score

The Data Product Score rates every data product against a list of rules grouped by the FAIR principles (Findable, Accessible, Interoperable, Reusable). Each rule carries a weight; the score is the share of fulfilled weight over the weight that applies to a product. An organization owner switches the score on and customizes it under Settings → Data Quality → Data Product Score; the switch at the top decides whether the leaderboard, the score pages, the list column and the API exist for the organization. Without customization, every organization scores with the built-in rules, weights and rating bands.

The settings page lists the rules grouped like the score page and shows, per principle, how many rules are enabled and their summed weight. Two organization-wide settings sit above the list:

Open leaderboard goes to the Governance leaderboard. Re-evaluate all products scores every product again from scratch: stored AI judgements are discarded and the model judges each product once more, and every other rule is recomputed right away; use it after switching the AI model or after changes that have no event. Restore defaults drops every adjustment, custom rule and stored AI result of the organization.

The default rules

An organization starts with the shipped list of rules, worded in every supported language. The first change on the settings page (editing, reordering, deleting or adding a rule) copies that list into the organization's own rules; from then on every rule, shipped or added, is edited with the same form and the same fields:

A default rule can be deleted like any other. Restore defaults drops the organization's own rules, settings and stored AI results and returns to the shipped list.

The shipped checks can be reused: Add rule → Built-in check creates a rule of your own that runs one of them under its own key, wording and parameter; the editor explains what each check tests. Two rules may run the same check with different parameters, for example a second link rule that looks for an authoritative definition of type wiki.

Expression rules

An expression rule is a rule written in JSONata over the documents of a data product. The expression must yield true when the rule is fulfilled and false otherwise; it is parsed when you save. A missing path yields nothing and counts as false, so no guards are needed, but also no warning: a valid expression over a path that does not exist fails every product, which is what Try it on a data product is for. An expression that yields something other than true or false, or fails at scoring time, counts as not applicable and is flagged on the settings page. JSONata has no side effects, and every evaluation runs under a time and depth limit (half a second, 100 nested calls), so a runaway rule ends as not applicable for that rule instead of stalling the scoring of the catalog.

Besides label, what is checked, how to fulfill (per language, with AI translation when a model is configured), category, icon, weight, references and the type scope, an expression rule declares what it is evaluated for:

Per-port and per-contract rules apply only to types that can have output ports, and count as not applicable while the product has none.

Documents

Every scope can read these roots as top-level fields of the input. Documents are the same YAML you see in the app, reachable by dot path; paths fan out over arrays (contracts.schema.properties), keys with hyphens go in backticks, and = compares.

Examples:

$count(dataProduct.customProperties[property = "data-owner-signoff" and value = "yes"]) > 0 and $count(team.members) > 0

$count(contract.slaProperties) > 0 and $length(contract.description.usage) >= 20

$count(contracts.schema.properties[$exists(description)]) >= 0.8 * $count(contracts.schema.properties)

The editor offers a helper that writes the expression from a plain-language description when an AI model is configured, and Try it on a data product evaluates the unsaved expression against a chosen product.

The definition as YAML

Edit as YAML on the settings page shows the whole definition as one document: the rules in order, the rating bands and the unranked statuses. Saving replaces the definition; a document that does not validate is refused as a whole with the reason. The same document is served and accepted by the settings API, so it can live in a repository and be pushed to every environment.

ratingBands:
  excellent: 90
  good: 75
  fair: 50
unrankedStatuses: [proposed, draft]
rules:
  - key: description          # built-in check; `check` defaults to the key
    check: description
    parameter: 150            # a number for thresholds, text for a link type
    category: findable
    weight: 5
    enabled: true
    icon: documentation
    appliesToTypes: [source-aligned, aggregate]   # omit for every type the check applies to
    label: { en: Data Product description, de: Datenprodukt-Beschreibung }
    hint: { en: The purpose is described with at least 150 characters. }
    fix: { en: Describe the purpose in the edit form. }
    references:
      - { framework: FAIR, id: F2, name: Data are described with rich metadata, href: https://www.go-fair.org/fair-principles/f2-data-described-rich-metadata/ }
  - key: tagged
    kind: expression
    scope: dataProduct        # dataProduct | outputPort | dataContract
    expression: $count(dataProduct.tags) >= 3
    category: findable
    weight: 3
    label: { en: Tagged }
  - key: readable
    kind: ai
    instructions: Judge whether a business user understands what the product is for.
    material: [product, team] # product | contracts | examples | team
    category: reusable
    weight: 3
    label: { en: Readable for business users }

Keys are the identity: a rule whose key is missing from a saved document is deleted along with its stored AI results. Wording is a map per language; nothing is translated on this path.

When scores are computed

Scores are stored per product and refreshed in the background, so the leaderboard, team and domain figures, the list column and the API read stored values. A product is recomputed within seconds of a change to anything it is scored on: the product itself, its output ports, the data contracts of those ports and their test results, its access agreements, the members of its owning team, its stars, its example data, its first lineage event (and lineage deletions), and the certifications and classification schemes of the organization. A change to the rules, weights, rating bands or unranked statuses recomputes every product of the organization.

The one input that changes without any event is the age of the latest test run. A sweep every 15 minutes recomputes scores older than APPLICATION_DATAPRODUCTSCORE_REFRESH_MAX_AGE_MINUTES (a day by default) for that, fills in products that have no stored score yet, and drops the scores of deleted products. The sweep can be switched off on an instance with APPLICATION_DATAPRODUCTSCORE_REFRESH_SWEEP_ENABLED=false. An organization with the score switched off gets no background work at all; switching it on shows current numbers because a read recomputes an outdated score on the spot.

A product's own score page always computes on the spot, so an edit shows immediately, and stores what it computed. The stored rule lines carry what each rule looked at, shown as Actual in a rule's detail dialog.

AI rules

An AI rule hands the organization's configured model your instructions, written as to a reviewer, plus the selected material of a product as YAML documents, and stores the model's result: pass or fail with a one-sentence reason that is shown on the score page. The material is rendered from the same documents that expression rules read: the data product document, the data contract documents of its active output ports (without their check runs), the example data of its output ports (the first 4,000 characters of each), and the owning team. Lineage and check results are never sent.

Results are never computed on page load. They are stored per product and refreshed whenever the product or the instructions change, by a sweep that runs every ten minutes and after a product is saved; an unchanged input means no new model call, and a new result recomputes the product's stored score. Until a product has a result, the rule is not applicable for it. Re-evaluate all products in a rule's editor discards the stored results of that rule; the button of the same name on the settings page does it for every AI rule at once. The editor shows the coverage (how many products have a result) and Try it on a data product runs the instructions once without storing anything. Like the score refresh, AI evaluation stops for an organization that switches the score off.

AI rules require a configured AI model (see AI). At most five AI rules are allowed per organization (APPLICATION_DATAPRODUCTSCORE_AI_MAX_RULES); a sweep makes at most 50 model calls per organization and run (APPLICATION_DATAPRODUCTSCORE_AI_BATCH_SIZE), and the sweep can be switched off with APPLICATION_DATAPRODUCTSCORE_AI_SWEEP_ENABLED=false.

API

GET /api/settings/dataproduct-score returns the definition above as JSON, or as YAML with Accept: application/yaml. PUT with a JSON or YAML body replaces it (organization write scope; a document that does not validate is refused with 422 and the reason). DELETE restores the shipped defaults.

GET /api/data-product-score/rules returns the effective configuration of the organization: the enabled rules with their kind, weight, type scope, threshold and references, the rating bands and the unranked statuses. GET /api/dataproducts/{id}/score and GET /api/dataproducts/scores return scores computed with that configuration; custom rules appear under their key, and countedInAggregates tells whether a product is ranked.