Databricks Integration

There are four ways of integrating Databricks with Entropy Data.

• The Ingestion reads the catalogs, schemas, tables, and views of Unity Catalog as assets, on a schedule.
• The Metadata Writer writes the descriptions and tags of your data contracts to the tables and columns in Unity Catalog, whenever a data contract changes.
• The Access Writer grants the consumers of active access agreements read access to the tables of a Databricks output port in Unity Catalog, and takes it back when the access ends.
• The Connector syncs assets and applies approved access to Databricks permissions. It is based on the Entropy Data SDK and runs in your network, for example in a DMZ if you have a complex network topology or governance restrictions do not allow a direct integration.

The ingestion, the metadata writer, and the access writer are built into Entropy Data, so no additional deployments are needed. You add them under Settings > Integrations. An integration has one purpose, so add one integration per purpose and per workspace.

What is synced

Service Principals and Permissions

Every Databricks integration authenticates with a service principal and an OAuth secret. Use one service principal per integration, so each holds only the permissions its purpose needs.

Why each integration needs what it needs:

• Ingestion: reads catalogs, schemas, tables, columns, and tags through the Unity Catalog REST API. It runs no SQL and creates no compute, so the narrowest entitlement, Consumer access, is enough.
• Metadata writer: writes tags through the Unity Catalog REST API, and table and column descriptions as COMMENT statements on a SQL warehouse. This needs Databricks SQL access, the sql scope, and Can use on the warehouse. Without a warehouse it writes tags only, and the unity-catalog scope is enough.
• Access writer: creates one account group per output port, adds and removes its members, and grants the group USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on each table the output port's data contract names. Other tables in the schema stay unreadable. Managing account groups from a workspace needs Admin access. Databricks accepts no narrower scope than all APIs for account groups. MANAGE lets the service principal grant on the catalog's schemas and tables; it reads no data itself.

Keep every other entitlement off: Admin access for the ingestion and the metadata writer, Allow unrestricted cluster creation, Allow instance pool creation, and Workspace access. With Admin access on, Databricks turns the other entitlements on and keeps them on.

A service principal also gets every entitlement the workspace's users group grants. By default that group grants Workspace access and Databricks SQL access. The narrow entitlements above only reduce what a service principal can do once your workspace no longer grants these to every member; see Migrate workspace entitlement control.

Grant the Unity Catalog privileges per catalog, as the owner of the catalog or as a metastore admin. Replace the placeholders with the Application ID of each service principal and keep the backticks; the display name does not work in a GRANT statement:

-- Ingestion
GRANT USE CATALOG, USE SCHEMA, SELECT ON CATALOG <catalog> TO `<ingestion-application-id>`;
-- Metadata writer
GRANT USE CATALOG, USE SCHEMA, SELECT, MODIFY, APPLY TAG ON CATALOG <catalog> TO `<metadata-writer-application-id>`;
-- Access writer
GRANT USE CATALOG, USE SCHEMA, MANAGE ON CATALOG <catalog> TO `<access-writer-application-id>`;

A secret with all APIs works for every integration. A scoped secret refuses a token for all APIs; the ingestion and the metadata writer request only the scopes in the table from the release that introduced the access writer on. With an earlier release, use a secret with all APIs.

Ingestion

Select the Integration Type

Navigate to Settings > Integrations > Add Integration. This opens a wizard that guides you through configuring the integration.

Select Databricks from the list of available integrations.

Select the Purpose

Choose Read Unity Catalog Metadata. The purpose Write Unity Catalog Metadata creates a metadata writer, and Provision Databricks Unity Catalog Permissions an access writer.

Configure the Credentials

The integration uses OAuth for service principals as authentication method. See Service Principals and Permissions for the entitlements, the secret's scopes, and the grants it needs, and the Databricks documentation for adding a service principal to your account and obtaining an OAuth secret.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Configure Filters

Configure filters to limit which assets are synchronized. Both include and exclude filters are supported. For Databricks, filters can be applied to Catalogs, Schemas, and Tables.

Filters support '*' as a wildcard character to match any number of characters.

Configure Schedule

Set a schedule to automatically synchronize assets. You can choose from predefined schedules or define a custom schedule using the cron expression format.

Note: All schedules use the UTC timezone, so make sure to take this into account when configuring your schedule. Please do not synchronize the assets more than once or twice per day. We reserve the right to disable the integration if this happens. You will be able to trigger a synchronization manually if you need an immediate update.

Complete the Integration Configuration

Choose a unique name for the integration, review your configuration, and click Create Integration.

Next Steps

The ingestion is now configured and will run according to the schedule. To check the integration status, navigate to Settings > Integrations. Here you'll find the current status and the last 10 runs.

You can adjust the integration configuration and credentials at any time. The configuration is saved in YAML format with syntax validation support in the editor.

Note: The previously stored client secret is not displayed in the edit view for security reasons.

Deselecting the Enabled checkbox disables the automatic schedule. Manual integration runs are still possible.

Metadata Writer

The metadata writer is the counterpart of the ingestion. The ingestion reads the metadata of Unity Catalog into Entropy Data. The metadata writer writes the descriptions and tags of your data contracts to the tables and columns in Unity Catalog. It runs whenever a data contract changes. There is no filter and no schedule.

Prerequisites

The metadata writer uses OAuth for service principals as authentication method, like the ingestion. Use a service principal of its own, so it holds only the permissions below. Service Principals and Permissions compares them with the other integrations.

To create the service principal in your workspace:

1. Go to Settings > Identity and access > Service principals and click Add service principal.
2. Choose Databricks managed and enter a name, for example entropy-data-metadata-writer.
3. Keep Databricks SQL access on. The metadata writer needs it for the SQL warehouse.
4. Keep Admin access, Allow unrestricted cluster creation, Allow instance pool creation, Workspace access, and Consumer access off. The metadata writer creates no compute and needs no admin rights.
5. Click Add service principal.
6. Open the service principal, go to the Secrets tab, and click Generate secret. Under Scopes, select Other APIs, then unity-catalog and sql.
7. Copy the secret right away; Databricks shows it only once. Copy the Client ID as well. It is the same as the service principal's Application ID.

See the Databricks documentation for more on service principals and OAuth secrets.

The service principal needs USE CATALOG, USE SCHEMA, SELECT, MODIFY, and APPLY TAG on the tables your data contracts name. Grant them per catalog to cover every current and future schema and table in it:

GRANT USE CATALOG, USE SCHEMA, SELECT, MODIFY, APPLY TAG ON CATALOG <catalog> TO `<application-id>`;

Or grant them per table. This covers only the tables your data contracts name today, so a new data contract needs a new grant:

GRANT USE CATALOG ON CATALOG <catalog> TO `<application-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<application-id>`;
GRANT SELECT, MODIFY, APPLY TAG ON TABLE <catalog>.<schema>.<table> TO `<application-id>`;

Replace <application-id> with the Application ID of the service principal and keep the backticks. The display name does not work in a GRANT statement. Run the statements as the owner of the catalog or as a metastore admin.

Descriptions are written as COMMENT statements and need a SQL warehouse. Give the service principal Can use on that warehouse (SQL Warehouses > Permissions) and the Databricks SQL access entitlement (Settings > Identity and access). Tags are written through the Unity Catalog REST API and need no warehouse. Without a warehouse, the metadata writer writes tags only.

Select the Integration Type and Purpose

Navigate to Settings > Integrations > Add Integration, select Databricks from the list of available integrations, and choose the purpose Write Unity Catalog Metadata.

Configure the Credentials

Test Connection fetches a token for the service principal and runs a query on the warehouse, if one is set. Grants shows the statements from the prerequisites for the tables your data contracts name today.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Complete the Integration Configuration

Choose a unique name for the integration and click Create Integration. From now on, the metadata writer writes the tables your data contracts name.

You can adjust the integration configuration and credentials at any time. Deselecting the Enabled checkbox pauses the metadata writer. It keeps the changes and writes them once it is enabled again. Deleting the integration leaves what it wrote in Unity Catalog as it is.

Which data contracts are written

The metadata writer considers data contracts in the ODCS format with the status active. Drafts, other statuses, branches, and data contracts in the DCS format are skipped.

A data contract names its tables in its servers. A server needs the type databricks, a catalog, and a schema. If it names a table, that table is written. Otherwise, every schema object of the data contract is written as a table in that schema. A server with a host is only written by the metadata writer of that workspace.

What is written

A tag key:value or key=value becomes a tag with that key and value. A tag without a value becomes a key with an empty value. In keys, the characters ., ,, -, =, /, :, and whitespace are replaced by _.

A table is written when its schema object has a description or links a semantic concept. A column is written when it has a description or tags. Every written comment ends with the marker line entropy-data:<integration id>, so the metadata writer recognizes its own text later on.

In addition, the metadata writer sets tags that link back to Entropy Data:

If an element links a semantic concept but has no description of its own, the description of the concept is written and marked with entropy_data_comment_source. When the ingestion reads such a table, it keeps the concept as the semantic link of the asset instead of taking the comment as its description.

What is left untouched

• Tags set by other tools. A tag key the metadata writer did not set before is left untouched, even when a data contract carries the same key. Only keys the metadata writer set itself are updated or removed. The entropy_data_* keys always belong to the metadata writer.
• Views, materialized views, streaming tables, and foreign tables only receive tags. Descriptions on views need ownership, and MODIFY is not grantable on foreign tables. The table on the integration page notes this.
• Limits of Unity Catalog. A table or column takes 50 tags, keys and values of at most 256 characters, and a table takes 1,000 column tags. Tags beyond a limit are dropped with a note. The tags set by other tools count first. Tags that a tag policy refuses are noted as well.
• Tables that do not exist are not created. The table is marked as unresolved until it appears in Unity Catalog, or the data contract no longer names it.

When it writes

Creating, updating, or deleting a data contract puts an event in the queue of the metadata writer. Updating or deleting a semantic concept that data contracts reference does the same. The metadata writer writes about a minute later. Several changes of the same data contract in that time result in one write. If a table already has the same text and tags, nothing is written.

A failed write is retried with growing intervals and given up after eight attempts. If the credentials are refused, the waiting events are given up. If the workspace is unreachable or rate limited, the queue pauses and continues later.

When a data contract is no longer active, is deleted, or no longer names a table, the metadata writer retires the table. It removes the tags it set and clears the comments it wrote. A comment written by someone else is left untouched. When an element loses its description, its comment is cleared and the tags set for it are removed.

After a write, the tables that exist as assets in Entropy Data are read again through the ingestion that imported them. This way, the asset shows what was just written. The ingestion skips the entropy_data_* tags and strips the marker line from comments.

Next Steps

To check the status of the metadata writer, navigate to Settings > Integrations and open the integration. The page shows:

• Queue: the pending events, and the events processed in the last 24 hours, with their attempts and errors. Discard pending drops the waiting events. Write again puts a processed event back in the queue. Retry given up puts the events given up in the last day back in the queue.
• Unity Catalog: the tables the metadata writer wrote, as a tree of catalogs, schemas, and tables. A table is In sync, Unresolved (there is no such table in Unity Catalog), or Failed. Retry failed writes the data contracts with failed or unresolved tables again.
• Credentials: the service principal, with Test Connection and Grants.

Click a table to see the comment and tags the metadata writer last wrote to it and its columns, the asset it was imported as, and the changes that triggered the writes. It shows what was last written, not the data contract as it is today.

API

You can also manage metadata writers via the REST API under /api/integrations/unity-catalog-metadata:

Ingestions are managed under /api/integrations/ingest. The same paths directly under /api/integrations are deprecated and keep working.

Access Writer

The access writer grants access agreements in Unity Catalog. When an access agreement on a Databricks output port becomes active, it gives the consumer read access to the tables the output port's data contract names. When the access ends, it takes the access back. It reports every step to the access agreement's provisioning, so the access page shows whether the grant exists. There is no filter and no schedule.

Prerequisites

The access writer uses OAuth for service principals as authentication method, like the ingestion. Use a service principal of its own, so it holds only the permissions below. Service Principals and Permissions compares them with the other integrations.

The access writer manages account groups through the workspace, so the workspace must be identity-federated. Neither an account admin nor the account console is needed.

To create the service principal in your workspace:

1. Go to Settings > Identity and access > Service principals and click Add service principal.
2. Choose Databricks managed and enter a name, for example entropy-data-access-writer.
3. Turn Admin access on. Managing account groups from a workspace needs it.
4. Click Add service principal.
5. Open the service principal, go to the Secrets tab, and click Generate secret. Under Scopes, keep All APIs. Databricks accepts no narrower scope for account groups.
6. Copy the secret right away; Databricks shows it only once. Copy the Client ID as well. It is the same as the service principal's Application ID.

See the Databricks documentation for more on service principals and OAuth secrets.

The service principal needs USE CATALOG, USE SCHEMA, and MANAGE on every catalog of your Databricks output ports:

GRANT USE CATALOG, USE SCHEMA, MANAGE ON CATALOG <catalog> TO `<application-id>`;

MANAGE lets the service principal grant on the schemas and tables in the catalog. It reads no data itself. Replace <application-id> with the Application ID of the service principal and keep the backticks. Run the statement as the owner of the catalog or as a metastore admin.

Select the Integration Type and Purpose

Navigate to Settings > Integrations > Add Integration, select Databricks from the list of available integrations, and choose the purpose Provision Databricks Unity Catalog Permissions.

Configure the Credentials

Test Connection fetches a token for the service principal and checks that it may manage account groups from the workspace. Grants shows the statement from the prerequisites.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Complete the Integration Configuration

Choose a unique name for the integration and click Create Integration. From now on, the access writer grants the access agreements that become active.

Access agreements that are already active when you create the integration are not granted. The access writer grants them once they are activated again.

You can adjust the integration configuration and credentials at any time. Deselecting the Enabled checkbox pauses the access writer. It keeps the changes and grants them once it is enabled again. Deleting the integration leaves the groups, their members, and their grants in Databricks. Deactivate the access agreements first to take the access back.

Which access agreements are granted

The access writer considers active access agreements on output ports with a server of the type databricks and the host of its workspace. The server names the catalog and the schema. The tables are the ones the output port's data contract names, as the metadata writer reads them: the server's table, or else every schema object of the data contract.

What is granted

The access writer creates one account group per output port and grants it:

Other tables in the schema stay unreadable. The consumers of the output port's active access agreements are the group's members. Five users on one output port are one group with five members, so the provider sees one principal on the schema.

The access writer names everything it creates in Databricks after the integration, so two integrations on one Databricks account never share a group:

<integration> is the first eight characters of the integration's id. A name longer than the 255 characters Databricks allows is cut and ends in a hash of the full name.

When a data contract changes, the access writer brings the grants of its output ports in step. A table the contract adds is granted. A table it drops loses SELECT. When an output port moves to another schema, the grants on the old schema are taken back.

Consumers

Consuming teams decides how a team reads:

• Groups managed by Entropy Data: the access writer creates a group per team and fills it with the team's members who exist in Databricks. When a member joins or leaves the team, the group follows.
• Microsoft Entra ID groups: the access writer uses the team's group as your identity provider syncs it to Databricks, found by the SCIM group name of the team's members. Teams without one get a managed group.

A data product reads with the service principal its custom property databricksServicePrincipal names. Without it, the access writer creates the service principal entropy-data-<integration>-dataproduct-<data product>.

An agent reads with the service principal of its linked identity Databricks service principal, under Linked identities on the agent's page. Enter the service principal's Application ID; the display name works as well. Only the agent reads, not its team. The access writer creates no service principal for an agent: an agent without such a linked identity is not granted, and the access agreement shows why.

What is left untouched

• Groups the access writer did not create. A group with the name the access writer wants, but which it did not create, is never used. The access agreement fails with a note, because the members of such a group are unknown.
• Members added by hand. When an access ends, the access writer takes out only the members it put in. Members you added to a group yourself stay.
• Other access agreements on the output port. When an access ends, a consumer stays in the group as long as another active access agreement on the same output port needs it.
• Tables that do not exist. A table the data contract names but Unity Catalog does not have is not granted. The access agreement fails with the table's name, while the consumer reads the other tables. Once the table exists, Retry grants it.
• Groups of ended access agreements. The group of an output port stays when its last access agreement ends, and is used again by the next one.

When it writes

Activating or deactivating an access agreement puts an event in the queue of the access writer, and the access writer grants it a few seconds later. Changes of a team and of a data contract are written after half a minute, so several changes in a row result in one write.

A failed write is retried with growing intervals and given up after eight attempts. If Databricks is unreachable or rate limited, the queue pauses and continues later.

The access writer reports the provisioning of every access agreement it grants: provisioning, then provisioned or provisioning-failed, and deprovisioning, then deprovisioned or deprovisioning-failed when the access ends. With provisioning grants access, an access agreement counts as access only once the access writer has granted it. The consumer receives the usual notifications when access is granted and when it ends. When an access agreement fails for good, for example because the user does not exist in Databricks or the last attempt failed, the team that owns the output port is notified.

Moving from the Connector

The access writer replaces the access management of the connector. Turn off the connector's access management before you create the access writer, so the two do not grant the same access agreements.

The groups the connector created, access-<access id>, are not taken over. The access writer grants an access agreement once it is activated again. After that, delete the connector's group of that access agreement.

Next Steps

To check the status of the access writer, navigate to Settings > Integrations and open the integration. The page shows:

• Queue: the pending events, and the events processed in the last 24 hours, with their attempts and errors. A click on an event shows what the access writer does for it.
• Unity Catalog: the catalogs, schemas, and tables of your Databricks output ports, each table with its data product, its output port, and how many principals can read it. Show permissions in Databricks opens the schema's permissions in Catalog Explorer. A table Unity Catalog does not have is marked.
• Credentials: the service principal, with Test Connection and Grants.

Click a table to see the principals that can read it: users, team groups with their members, and service principals, each with the access agreements that put them there. A failed access agreement shows the reason and a Retry. Grants in Databricks shows what is granted on the table as SQL.

API

You can also manage access writers via the REST API under /api/integrations/unity-catalog-access:

A PUT body may carry the externalId, as a GET returns it. It must match the path. The same applies to the metadata writer.

Connector

The Databricks Connector is an open-source component that integrates Entropy Data with Databricks. It is based on the SDK and available as a Docker image. The source code can be forked to implement custom integrations.

Features

• Asset Synchronization: Sync catalogs, schemas, and tables from Databricks to Entropy Data as Assets.
• Access Management: Listen for AccessActivated and AccessDeactivated events in Entropy Data and grant access on Databricks to the data consumer.

Deploying the Connector

To get started, we need to set up docker to run the connector. Beforehand you will need to get some secrets and configuration data from Databricks to enable the Connector to fetch the correct data.

You find an overview of the configuration parameters in the README of the connector.

• ENTROPYDATA_CLIENT_APIKEY: API Key for Entropy Data (see: Authentication)
• ENTROPYDATA_CLIENT_HOST: URL of the Entropy Data API (default: https://api.entropy-data.com/api)
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_HOST: URL of the Databricks workspace (e.g. adb-1973318500000000.17.azuredatabricks.net, see Databricks Authentication)
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTID: Client ID of the Databricks workspace
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTSECRET: Client Secret of the Databricks workspace
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_HOST: URL of the Databricks account (e.g. https://accounts.azuredatabricks.com, see Databricks Account Settings)
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTID: Client ID of the Databricks account
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTSECRET: Client Secret of the Databricks account

You can use different service principals (hence, different Client IDs and Client Secrets) for the workspace and account authentication. The connector will use the workspace authentication to fetch the data from Databricks and the account authentication to manage authentication.

The service principal for the workspace authentication needs the USE CATALOG, USE SCHEMA, and MODIFY permissions on the catalogs and schemas it manages, and the workspace admin role to grant access to other principals. The service principal for the account authentication needs the account admin role, because the connector creates and deletes account groups and service principals.

If you have all configuration in place, you can now start the connector with the following command. Adopt these if you run the container in managed container environments like Kubernetes, Azure Container Apps, or AWS EKS.

    docker run \
    -e ENTROPYDATA_CLIENT_APIKEY='dmm_live_5TJYRn6CrsWELldNiffii6oKdYNMuEYWinBoOoxRrvXaLW4y9A5Xck12as9dasw9' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_HOST='https://adb-1973318500000000.17.azuredatabricks.net' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTID='d2e11498-4b63-43cd-9ebe-a00000000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTSECRET='dose79375ba79041c9c5250a190002a71cd9' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_HOST='https://accounts.azuredatabricks.net' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_ACCOUNTID='4675280120000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTID='d2e11498-4b63-43cd-9ebe-a00000000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTSECRET='dose000000000000c9c5250a190002a71cd9' \
    entropydata/entropy-data-connector-databricks:latest

After a successful start of the connector, you will find the following logs for your Connector Docker container:

2025-05-15T07:42:39.596Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : databricks-access-management: Start polling for events
2025-05-15T07:42:39.663Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : Fetching events with lastEventId=1f030d95-7eeb-67c4-b68c-7f053555cc14
2025-05-15T07:42:39.726Z  INFO 1 --- [           main] e.s.EntropyDataConnectorRegistration     : Registering integration connector databricks-assets
2025-05-15T07:42:39.748Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : Processing event 1f030d95-7f45-6d15-b68c-9fcae55fcd9a of type com.datamesh-manager.events.AccessDeactivatedEvent
2025-05-15T07:42:39.753Z  INFO 1 --- [cTaskExecutor-1] e.d.DatabricksAccessManagementHandler    : Processing AccessDeactivatedEvent 5sHEK6r9PydbE5CVpSk9BN
2025-05-15T07:42:39.789Z  INFO 1 --- [cTaskExecutor-2] e.s.EntropyDataAssetsSynchronizer      : databricks-assets: start syncing assets

The connector will subscribe to the Events API and save the current state (lastEventId) directly in Entropy Data.

Using Access Management

For access management without a deployment, use the access writer. See Moving from the Connector.

The connector will automatically create the permissions in Databricks for approved Access Requests for a Data Product's Output Port in Entropy Data: one account group access-<access id> per access, holding the consuming user, the consuming team's group team-<team id>, or a service principal dataproduct-<data product id> for a consuming data product, with USE CATALOG on the catalog and USE SCHEMA and SELECT on the output port's schema. Deactivation removes the group. Finer grants, masks, row filters, or ABAC policies are customizations of the connector; see Provisioning for the model and for the pattern of keying Unity Catalog policies on these groups.

Look at the documentation, how the access management flow works in detail.

Using Asset Synchronization

Assets are representations of actual data structures in Databricks. You can leverage them to import models and schemas from Databricks into Entropy Data. The connector will automatically create the assets in Entropy Data. You can link them with data products and derive data contracts from them.

The connector will automatically create assets in Entropy Data for all tables and views in the Databricks Unity Catalog.

Links

• Source Code: Entropy Data Connector for Databricks on GitHub

Databricks Asset Bundles

Databricks Asset Bundles (DABs) provide a standardized format for bundling data products on Databricks. They contain all necessary files to run a data product, including code and configuration. Our extended template builds on the standard open-source Databricks Asset Bundle template, adding Entropy Data integration that automatically registers the data product and its data contract via the REST API.

# Create a new Databricks Asset Bundle using the Entropy Data template
databricks bundle init https://github.com/datamesh-manager/databricks-bundle-python-template

Links

• Tutorial: How To Build a Data Product with Databricks
• Source Code: Entropy Data Asset Bundle Template For Databricks on GitHub

Query Execution

Once a Databricks data product is available, users can execute read-only SQL queries against its output ports through:

• MCP Interface: Use the execute_query tool to run SQL queries on Databricks output ports.
• Entropy Intelligence: Chat-based query execution with automatic connection setup.

To execute queries, each user needs to configure a personal Databricks connection in their profile under Connections. This ensures queries run on behalf of the individual user with their own Databricks credentials and permissions.