Databricks Integration

Integration of Databricks with Entropy Data.

Entropy Data integrates with Databricks in three ways:

1. Ingestion (Built-In)

An ingestion is an integration that reads the metadata of Unity Catalog. It imports catalogs, schemas, tables, and views as assets on a schedule. It is managed within Entropy Data: configure the credentials, filters, and schedule to start syncing with Databricks.

No additional deployments are needed.

2. Metadata Writer (Built-In)

A metadata writer is an integration that writes metadata to Unity Catalog. Whenever a data contract changes, it writes the descriptions and tags of the data contract to the tables and columns the contract names. It is managed within Entropy Data as well.

No additional deployments are needed.

3. Connector

Our connectors are based on the Entropy Data SDK and provide support for more use cases. You get the asset syncing capabilities of the ingestion, plus direct integration with Databricks permissions. Changes through the Access Approval Workflow can be dynamically applied to Databricks user permissions.

If you have a complex network topology or direct integration with Entropy Data is not possible due to governance restrictions, the Databricks connector can be deployed in a DMZ of your network to provide additional security.

An ingestion and a metadata writer are both integrations that you add under Settings > Integrations. An integration has one purpose. Add one integration per purpose and per workspace.

1. Ingestion

You can directly integrate Databricks with Entropy Data. An ingestion reads the metadata of Unity Catalog and creates assets in Entropy Data.

Prerequisites

To start, navigate to Settings > Integrations > Add Integration. This opens a wizard that guides you through configuring the integration.

Select the Integration Type

Select Databricks from the list of available integrations.

Select the Purpose

Choose Read Unity Catalog Metadata. The other purpose, Write Unity Catalog Metadata, creates a metadata writer.

Configure the Credentials

The integration uses OAuth for service principals as authentication method. Refer to the Databricks documentation for adding a service principal to your account and obtaining an OAuth secret.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Configure Filters

Configure filters to limit which assets are synchronized. Both include and exclude filters are supported. For Databricks, filters can be applied to Catalogs, Schemas, and Tables.

Filters support '*' as a wildcard character to match any number of characters.

Configure Schedule

Set a schedule to automatically synchronize assets. You can choose from predefined schedules or define a custom schedule using the cron expression format.

Note: All schedules use the UTC timezone, so make sure to take this into account when configuring your schedule. Please do not synchronize the assets more than once or twice per day. We reserve the right to disable the integration if this happens. You will be able to trigger a synchronization manually if you need an immediate update.

Complete the Integration Configuration

Choose a unique name for the integration, review your configuration, and click Create Integration.

What is synced

Next Steps

The ingestion is now configured and will run according to the schedule. To check the integration status, navigate to Settings > Integrations. Here you'll find the current status and the last 10 runs.

You can adjust the integration configuration and credentials at any time. The configuration is saved in YAML format with syntax validation support in the editor.

Note: The previously stored client secret is not displayed in the edit view for security reasons.

Deselecting the Enabled checkbox disables the automatic schedule. Manual integration runs are still possible.

2. Metadata Writer

The metadata writer is the counterpart of the ingestion. The ingestion reads the metadata of Unity Catalog into Entropy Data. The metadata writer writes the descriptions and tags of your data contracts to the tables and columns in Unity Catalog. It runs whenever a data contract changes. There is no filter and no schedule.

Prerequisites

The metadata writer uses OAuth for service principals as authentication method, like the ingestion. Refer to the Databricks documentation for adding a service principal to your account and obtaining an OAuth secret.

The service principal needs USE CATALOG, USE SCHEMA, SELECT, MODIFY, and APPLY TAG on the tables your data contracts name. Grant them per catalog to cover every current and future schema and table in it:

GRANT USE CATALOG, USE SCHEMA, SELECT, MODIFY, APPLY TAG ON CATALOG <catalog> TO `<application-id>`;

Or grant them per table. This covers only the tables your data contracts name today, so a new data contract needs a new grant:

GRANT USE CATALOG ON CATALOG <catalog> TO `<application-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<application-id>`;
GRANT SELECT, MODIFY, APPLY TAG ON TABLE <catalog>.<schema>.<table> TO `<application-id>`;

Descriptions are written as COMMENT statements and need a SQL warehouse. Give the service principal Can use on that warehouse (SQL Warehouses > Permissions) and the Databricks SQL access entitlement (Settings > Identity and access). Tags are written through the Unity Catalog REST API and need no warehouse. Without a warehouse, the metadata writer writes tags only.

Select the Integration Type and Purpose

Navigate to Settings > Integrations > Add Integration, select Databricks from the list of available integrations, and choose the purpose Write Unity Catalog Metadata.

Configure the Credentials

Test Connection fetches a token for the service principal and runs a query on the warehouse, if one is set. Grants shows the statements from the prerequisites for the tables your data contracts name today.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Complete the Integration Configuration

Choose a unique name for the integration and click Create Integration. From now on, the metadata writer writes the tables your data contracts name.

You can adjust the integration configuration and credentials at any time. Deselecting the Enabled checkbox pauses the metadata writer. It keeps the changes and writes them once it is enabled again. Deleting the integration leaves what it wrote in Unity Catalog as it is.

Which data contracts are written

The metadata writer considers data contracts in the ODCS format with the status active. Drafts, other statuses, branches, and data contracts in the DCS format are skipped.

A data contract names its tables in its servers. A server needs the type databricks, a catalog, and a schema. If it names a table, that table is written. Otherwise, every schema object of the data contract is written as a table in that schema. A server with a host is only written by the metadata writer of that workspace.

What is written

A tag key:value or key=value becomes a tag with that key and value. A tag without a value becomes a key with an empty value. In keys, the characters ., ,, -, =, /, :, and whitespace are replaced by _.

A table is written when its schema object has a description or links a semantic concept. A column is written when it has a description or tags. Every written comment ends with the marker line entropy-data:<integration id>, so the metadata writer recognizes its own text later on.

In addition, the metadata writer sets tags that link back to Entropy Data:

If an element links a semantic concept but has no description of its own, the description of the concept is written and marked with entropy_data_comment_source. When the ingestion reads such a table, it keeps the concept as the semantic link of the asset instead of taking the comment as its description.

What is left untouched

• Tags set by other tools. A tag key the metadata writer did not set before is left untouched, even when a data contract carries the same key. Only keys the metadata writer set itself are updated or removed. The entropy_data_* keys always belong to the metadata writer.
• Views, materialized views, streaming tables, and foreign tables only receive tags. Descriptions on views need ownership, and MODIFY is not grantable on foreign tables. The table on the integration page notes this.
• Limits of Unity Catalog. A table or column takes 50 tags, keys and values of at most 256 characters, and a table takes 1,000 column tags. Tags beyond a limit are dropped with a note. The tags set by other tools count first. Tags that a tag policy refuses are noted as well.
• Tables that do not exist are not created. The table is marked as unresolved until it appears in Unity Catalog, or the data contract no longer names it.

When it writes

Creating, updating, or deleting a data contract puts an event in the queue of the metadata writer. Updating or deleting a semantic concept that data contracts reference does the same. The metadata writer writes about a minute later. Several changes of the same data contract in that time result in one write. If a table already has the same text and tags, nothing is written.

A failed write is retried with growing intervals and given up after eight attempts. If the credentials are refused, the waiting events are given up. If the workspace is unreachable or rate limited, the queue pauses and continues later.

When a data contract is no longer active, is deleted, or no longer names a table, the metadata writer retires the table. It removes the tags it set and clears the comments it wrote. A comment written by someone else is left untouched. When an element loses its description, its comment is cleared and the tags set for it are removed.

After a write, the tables that exist as assets in Entropy Data are read again through the ingestion that imported them. This way, the asset shows what was just written. The ingestion skips the entropy_data_* tags and strips the marker line from comments.

Next Steps

To check the status of the metadata writer, navigate to Settings > Integrations and open the integration. The page shows:

• Queue: the pending events, and the events processed in the last 24 hours, with their attempts and errors. Discard pending drops the waiting events. Write again puts a processed event back in the queue. Retry given up puts the events given up in the last day back in the queue.
• Unity Catalog: the tables the metadata writer wrote, as a tree of catalogs, schemas, and tables. A table is In sync, Unresolved (there is no such table in Unity Catalog), or Failed. Retry failed writes the data contracts with failed or unresolved tables again.
• Credentials: the service principal, with Test Connection and Grants.

Click a table to see the comment and tags the metadata writer last wrote to it and its columns, the asset it was imported as, and the changes that triggered the writes. It shows what was last written, not the data contract as it is today.

API

You can also manage metadata writers via the REST API under /api/integrations/unity-catalog-metadata:

Ingestions are managed under /api/integrations/ingest. The same paths directly under /api/integrations are deprecated and keep working.

3. Connector

The Databricks Connector is an open-source component that integrates Entropy Data with Databricks. It is based on the SDK and available as a Docker image. The source code can be forked to implement custom integrations.

Features

• Asset Synchronization: Sync catalogs, schemas, and tables from Databricks to Entropy Data as Assets.
• Access Management: Listen for AccessActivated and AccessDeactivated events in Entropy Data and grant access on Databricks to the data consumer.

Deploying the Connector

To get started, we need to set up docker to run the connector. Beforehand you will need to get some secrets and configuration data from Databricks to enable the Connector to fetch the correct data.

You find an overview of the configuration parameters in the README of the connector.

• ENTROPYDATA_CLIENT_APIKEY: API Key for Entropy Data (see: Authentication)
• ENTROPYDATA_CLIENT_HOST: URL of the Entropy Data API (default: https://api.entropy-data.com/api)
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_HOST: URL of the Databricks workspace (e.g. adb-1973318500000000.17.azuredatabricks.net, see Databricks Authentication)
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTID: Client ID of the Databricks workspace
• ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTSECRET: Client Secret of the Databricks workspace
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_HOST: URL of the Databricks account (e.g. https://accounts.azuredatabricks.com, see Databricks Account Settings)
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTID: Client ID of the Databricks account
• ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTSECRET: Client Secret of the Databricks account

You can use different service principals (hence, different Client IDs and Client Secrets) for the workspace and account authentication. The connector will use the workspace authentication to fetch the data from Databricks and the account authentication to manage authentication.

The service principal for the workspace authentication needs the USE CATALOG, USE SCHEMA, and MODIFY permissions on the catalogs and schemas it manages, and the workspace admin role to grant access to other principals. The service principal for the account authentication needs the account admin role, because the connector creates and deletes account groups and service principals.

If you have all configuration in place, you can now start the connector with the following command. Adopt these if you run the container in managed container environments like Kubernetes, Azuer Container Apps, or AWS EKS.

    docker run \
    -e ENTROPYDATA_CLIENT_APIKEY='dmm_live_5TJYRn6CrsWELldNiffii6oKdYNMuEYWinBoOoxRrvXaLW4y9A5Xck12as9dasw9' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_HOST='https://adb-1973318500000000.17.azuredatabricks.net' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTID='d2e11498-4b63-43cd-9ebe-a00000000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_WORKSPACE_CLIENTSECRET='dose79375ba79041c9c5250a190002a71cd9' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_HOST='https://accounts.azuredatabricks.net' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_ACCOUNTID='4675280120000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTID='d2e11498-4b63-43cd-9ebe-a00000000000' \
    -e ENTROPYDATA_CLIENT_DATABRICKS_ACCOUNT_CLIENTSECRET='dose000000000000c9c5250a190002a71cd9' \
    entropydata/entropy-data-connector-databricks:latest

After a successful start of the connector, you will find the following logs for your Connector Docker container:

2025-05-15T07:42:39.596Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : databricks-access-management: Start polling for events
2025-05-15T07:42:39.663Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : Fetching events with lastEventId=1f030d95-7eeb-67c4-b68c-7f053555cc14
2025-05-15T07:42:39.726Z  INFO 1 --- [           main] e.s.EntropyDataConnectorRegistration     : Registering integration connector databricks-assets
2025-05-15T07:42:39.748Z  INFO 1 --- [cTaskExecutor-1] e.sdk.EntropyDataEventListener            : Processing event 1f030d95-7f45-6d15-b68c-9fcae55fcd9a of type com.datamesh-manager.events.AccessDeactivatedEvent
2025-05-15T07:42:39.753Z  INFO 1 --- [cTaskExecutor-1] e.d.DatabricksAccessManagementHandler    : Processing AccessDeactivatedEvent 5sHEK6r9PydbE5CVpSk9BN
2025-05-15T07:42:39.789Z  INFO 1 --- [cTaskExecutor-2] e.s.EntropyDataAssetsSynchronizer      : databricks-assets: start syncing assets

The connector will subscribe to the Events API and save the current state (lastEventId) directly in Entropy Data.

Using Access Management

The connector will automatically create the permissions in Databricks for approved Access Requests for a Data Product's Output Port in Entropy Data: one account group access-<access id> per access, holding the consuming user, the consuming team's group team-<team id>, or a service principal dataproduct-<data product id> for a consuming data product, with USE CATALOG on the catalog and USE SCHEMA and SELECT on the output port's schema. Deactivation removes the group. Finer grants, masks, row filters, or ABAC policies are customizations of the connector; see Provisioning for the model and for the pattern of keying Unity Catalog policies on these groups.

Look at the documentation, how the access management flow works in detail.

Using Asset Synchronization

Assets are representations of actual data structures in Databricks. You can leverage them to import models and schemas from Databricks into Entropy Data. The connector will automatically create the assets in Entropy Data. You can link them with dataproducts and build derive Data Contracts from them.

The connector will automatically create assets in Entropy Data for all tables and views in the Databricks Unity Catalog.

Links

• Source Code: Entropy Data Connector for Databricks on GitHub

Databricks Asset Bundles

Databricks Asset Bundles (DABs) provide a standardized format for bundling data products on Databricks. They contain all necessary files to run a data product, including code and configuration. Our extended template builds on the standard open-source Databricks Asset Bundle template, adding Entropy Data integration that automatically registers the data product and its data contract via the REST API.

# Create a new Databricks Asset Bundle using the Entropy Data template
databricks bundle init https://github.com/datamesh-manager/databricks-bundle-python-template

Links

• Tutorial: How To Build a Data Product with Databricks
• Source Code: Entropy Data Asset Bundle Template For Databricks on GitHub

Query Execution

Once a Databricks data product is available, users can execute read-only SQL queries against its output ports through:

• MCP Interface: Use the execute_query tool to run SQL queries on Databricks output ports.
• Entropy Intelligence: Chat-based query execution with automatic connection setup.

To execute queries, each user needs to configure a personal Databricks connection in their profile under Connections. This ensures queries run on behalf of the individual user with their own Databricks credentials and permissions.