Dremio Integration

Integration of Dremio with Entropy Data.

The Dremio integration is managed within Entropy Data. Configure the connection and sync schedule to start syncing with Dremio.

No additional deployments are needed. Entropy Data reads the metadata through the Dremio REST API, so no JDBC or Arrow Flight driver is required.

Features

• Asset Synchronization: Sync sources, spaces, folders, tables and views from Dremio to Entropy Data as Assets, including column types.
• Data Products and Data Contracts: Create data products and data contracts from Dremio assets. Dremio is available as a server type for output ports and data contracts.
• Query Execution: Run read-only SQL queries on Dremio output ports through MCP and Entropy Intelligence, with each user's personal Dremio connection.

Prerequisites

You need a Dremio Software cluster (Community or Enterprise Edition) and a Dremio user that can sign in with a username and password, for example a local user or an LDAP user. The integration queries INFORMATION_SCHEMA, so it only finds the datasets the user is allowed to see.

To start, navigate to Settings > Integrations > Add Integration. This opens a wizard that guides you through configuring the integration.

Configuration

Select the Integration Type

Select Dremio from the list of available integrations.

Configure the Credentials

Provide the following connection details:

Use the HTTP URL of the web console, not a JDBC (jdbc:dremio:...) or Arrow Flight connection string.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Configure Filters

Configure filters to limit which assets are synchronized. Both include and exclude filters are supported. For Dremio, filters can be applied to three levels:

• Sources and spaces: the top-level containers, matched by name, for example prod_*.
• Schemas: the full dotted path of a source or space and its folders, for example sales.marts. Use * to match any part of the path, for example sales.* or *.marts.
• Tables: table and view names, for example dim_*.

Filters support '*' as a wildcard character to match any number of characters.

filters:
  sourceIncludePatterns:
    - "sales"
    - "lakehouse_*"
  sourceExcludePatterns:
    - "$scratch"
  schemaIncludePatterns:
    - "*.marts"
  tableExcludePatterns:
    - "tmp_*"

The sys and INFORMATION_SCHEMA schemas and the users' home spaces are never synchronized.

Configure Schedule

Set a schedule to automatically synchronize assets. You can choose from predefined schedules or define a custom schedule using the cron expression format.

Note: All schedules use the UTC timezone, so make sure to take this into account when configuring your schedule. Please do not synchronize the assets more than once or twice per day. We reserve the right to disable the integration if this happens. You will be able to trigger a synchronization manually if you need an immediate update.

Complete the Integration Configuration

Choose a unique name for the integration, review your configuration, and click Create Integration.

Synchronized Assets

In Dremio, a schema is the dotted path to a dataset's container, such as sales.marts. Every level of that path becomes a dremio_schema asset linked to its parent, so a view in sales.marts is linked to sales.marts, which is linked to sales. Columns carry their Dremio type with precision, such as DECIMAL(10,2), and columns that cannot be null are marked as required.

Data Products and Data Contracts

When you create a data product or a data contract from a Dremio asset, Entropy Data sets up a dremio server with the host, port and schema of the asset.

servers:
  - server: "production"
    type: "dremio"
    host: "dremio.example.com"
    port: 9047
    schema: "sales.marts"

The port is taken from the Dremio Server URL of the integration. Clients that connect over JDBC (default port 31010) or Arrow Flight (default port 32010) use a different port, so adjust it in the contract if your tools read it from there.

Query Execution

Users can run read-only SQL queries on the output ports of Dremio data products through:

• MCP Interface: Use the execute_query tool to run SQL queries on Dremio output ports.
• Entropy Intelligence: Chat-based query execution. When a connection is missing, the chat asks you to add one.

To execute queries, each user configures a personal Dremio connection in their profile under Connections, with the Server URL of the Dremio web console and REST API (for example https://dremio.example.com:9047), a username and a password. Queries run as that user, with their own Dremio permissions.

A connection serves the output ports whose host is the host of its Server URL; the port is not compared, so a contract that names the JDBC or Arrow Flight port still matches. The schema of the output port becomes the query context, so SELECT * FROM order_summary on an output port with schema sales.marts reads sales.marts.order_summary. Results are limited to 100 rows, and a query that runs longer than 40 seconds is canceled.

Next Steps

The integration is now configured and will run according to the schedule. To check the integration status, navigate to Settings > Integrations. Here you'll find the current status and the last 10 integration runs.

You can adjust the integration configuration at any time. The configuration is saved in YAML format with syntax validation support in the editor.

You can also change and test credentials in the edit view.

Note: The previously stored credentials are not displayed in the edit view for security reasons. If you want to change them, add new credentials and save the integration.

Deselecting the Enabled checkbox disables the automatic schedule. Manual integration runs are still possible.