Databricks Integration

Integration of Databricks with the Entropy Data.

Databricks Connector

Databricks Connector is an open-source component that integrates Entropy Data with Databricks. It is based on the SDK and available as a Docker image. The source code can also be forked to implement custom integrations.

Features

  • Asset Synchronization: Sync tables and schemas of the Unity catalog to the Entropy Data as Assets.
  • Access Management: Listen for AccessActivated and AccessDeactivated events in the Entropy Data and grants access on Databricks to the data consumer.

Links

Databricks Asset Bundles

Databricks Asset Bundles (DABs) are the new format to bundle data products on Databricks. They contain all files necessary to run a data product on Databricks, including the code and the configuration. We've extended the standard open source template for a Databricks Asset Bundle with the Entropy Data integration. Our extended template automatically registers the data product with its data contract at the Entropy Data through the REST API.

# Create a new Databricks Asset Bundle using the Entropy Data template
databricks bundle init https://github.com/datamesh-manager/databricks-bundle-python-template

Links