Ingest Assets

Build your own integration to sync assets from your data platform to Entropy Data when prebuilt connectors don't fit your needs.

Before building your own integration, check if a prebuilt connector is available for your data platform (Databricks, Snowflake, AWS, Azure, Google Cloud, Kafka). Connectors provide the best experience with minimal setup.

Overview

Asset ingestion synchronizes metadata about your physical data sources (tables, views, schemas, topics, etc.) from your data platform to Entropy Data. This enables you to:

Automatically generate data contracts from existing table structures
Import assets as data products with minimal manual effort
Link technical resources to business concepts (data products and output ports)
Track data lineage across your data platform

You can implement asset ingestion using:

SDK: Java library for building integrations (recommended for long-running applications)
REST API: Direct API access for any programming language

Understanding the Asset Data Model

An asset in Entropy Data consists of four main parts:

1. Info Object

The info object contains metadata about the asset:

source: Identifies the data source (e.g., snowflake, unity, purview, postgres). This is not about the data catalog, but rather the data source itself. Do not use values like openmetadata or collibra here, as they represent catalogs, not data sources.
type: Describes the asset type, prefixed by the source (e.g., snowflake_table, unity_schema, kafka_topic)
name: Name of the asset. It must not be the full qualified name, but rather the last segment (e.g., for a table CUSTOMERS, not SALES_DB.PUBLIC.CUSTOMERS)
qualifiedName (deprecated): Unique identifier, often used to extract connection information. Deprecated in favor of storing identifiers in the custom map. Still supported for backwards compatibility.
metadataSource (optional): Identifies the metadata catalog that provided this asset (e.g., openmetadata, collibra, purview). Use this to distinguish between the data source and the catalog.
metadataSourceId (optional): The unique identifier of the asset in the metadata source.
metadataSourceUrl (optional): A URL pointing to the asset in the metadata source.

2. Custom Properties

The custom map stores additional metadata and connection details as key-value pairs. Provide server details here so that they will be picked up when converting an asset to a data product. For example, for Snowflake, include account, database, and schema. Use the environment property to indicate which environment the asset belongs to (e.g., dev, staging, prod). You find a detailed description of the needed fields below.

In older API versions, this field was called properties. The old field name is still accepted for backwards compatibility, but new integrations should use custom.

3. Columns

For table-like assets, the columns array defines the schema structure. Use the types from the type system of the data source, e.g., for Snowflake use the SQL types supported by Snowflake. This enables automatic generation of data contracts. The types will become the physicalType in the data contract, and the logicalType will be automatically derived.

Each column supports the following fields:

name (required): The column name
type (required): The physical data type from the data source (e.g., VARCHAR(255), BIGINT, STRING)
description: Column description
required: Whether the column is required
unique: Whether values must be unique
primaryKey: Whether this is a primary key column
tags: Array of tags associated with the column
links: Links associated with the column
custom: Additional custom properties as key-value map

Nested Structures

Columns support nested structures for complex data types:

properties: For record/object types, an array of child columns defining the fields of the record
items: For array types, a single column definition describing the schema of the array elements

"columns": [
  {
    "name": "orderId",
    "type": "string",
    "required": true,
    "primaryKey": true
  },
  {
    "name": "shippingAddress",
    "type": "record",
    "description": "The shipping address",
    "properties": [
      { "name": "street", "type": "string" },
      { "name": "city", "type": "string" },
      { "name": "zip", "type": "string" }
    ]
  },
  {
    "name": "items",
    "type": "array",
    "description": "The order line items",
    "items": {
      "name": "items",
      "type": "record",
      "properties": [
        { "name": "productId", "type": "string" },
        { "name": "quantity", "type": "int" }
      ]
    }
  }
]

4. Relationships

Assets can have hierarchical relationships using the relationships array. Common patterns include:

Two-tier: Schema → Table
Three-tier: Database → Schema → Table

Indicate that relationship at the child element pointing to its parent using relationshipType: "parent".

Example: Databricks Catalog (Top-Level Parent)

{
  "id": "prod-catalog",
  "info": {
    "source": "databricks",
    "type": "databricks_catalog",
    "name": "production",
    "qualifiedName": "production"
  },
  "custom": {
    "catalog": "production",
    "environment": "prod"
  }
}

Example: Databricks Schema (Child of Catalog)

{
  "id": "prod-sales-schema",
  "info": {
    "source": "databricks",
    "type": "databricks_schema",
    "name": "sales",
    "qualifiedName": "production.sales"
  },
  "custom": {
    "catalog": "production",
    "schema": "sales",
    "environment": "prod"
  },
  "relationships": [
    {
      "assetId": "prod-catalog",
      "relationshipType": "parent"
    }
  ]
}

Example: Databricks Table (Child of Schema)

{
  "id": "prod-sales-customers",
  "info": {
    "source": "databricks",
    "type": "databricks_table",
    "name": "customers",
    "qualifiedName": "production.sales.customers"
  },
  "custom": {
    "catalog": "production",
    "schema": "sales",
    "environment": "prod"
  },
  "relationships": [
    {
      "assetId": "prod-sales-schema",
      "relationshipType": "parent"
    }
  ]
}

Mapping Your Data Source to Assets

Select the right values for source and type that represent your data platform. Use server types from the Open Data Contract Standard when available.

Platform	source	type examples	Common custom fields	Notes
API	`api`	`api_endpoint`	`location`
AWS Athena	`athena`	`athena_database`, `athena_table`	`schema`, `catalog`, `stagingDir`, `regionName`
AWS Glue	`glue`	`glue_database`, `glue_table`	`account`, `database`, `location`, `format`	Also used for Glue Catalog
AWS Kinesis	`kinesis`	`kinesis_stream`	`stream`, `region`, `format`
AWS Redshift	`redshift`	`redshift_database`, `redshift_schema`, `redshift_table`	`host`, `database`, `schema`, `region`, `account`
AWS S3	`s3`	`s3_bucket`, `s3_folder`, `s3_object`	`location`, `format`, `delimiter`, `endpointUrl`
Azure Storage	`azure`	`azure_container`, `azure_blob`	`location`, `format`, `delimiter`
Azure Synapse	`synapse`	`synapse_database`, `synapse_schema`, `synapse_table`	`host`, `port`, `database`
ClickHouse	`clickhouse`	`clickhouse_database`, `clickhouse_table`	`host`, `port`, `database`
Custom Platform	`custom`	`custom_<type>`	Define your own properties
Databricks	`databricks`	`databricks_catalog`, `databricks_schema`, `databricks_table`, `databricks_view`	`host`, `catalog`, `schema`	Modern Databricks with Unity Catalog
Dremio	`dremio`	`dremio_source`, `dremio_schema`, `dremio_table`	`host`, `port`, `schema`
DuckDB	`duckdb`	`duckdb_schema`, `duckdb_table`	`database`, `schema`
Google BigQuery	`bigquery`	`bigquery_project`, `bigquery_dataset`, `bigquery_table`, `bigquery_view`	`project`, `dataset`
Google Cloud SQL	`cloudsql`	`cloudsql_database`, `cloudsql_table`	`host`, `port`, `database`, `schema`
Google Pub/Sub	`pubsub`	`pubsub_topic`, `pubsub_subscription`	`project`
IBM DB2	`db2`	`db2_database`, `db2_schema`, `db2_table`	`host`, `port`, `database`, `schema`
Kafka	`kafka`	`kafka_cluster`, `kafka_topic`	`host`, `format`
Microsoft Purview	`purview`	`purview_database`, `purview_schema`, `purview_table`	Varies by source	Prefer underlying data source type (e.g., `sqlserver`) when possible
MySQL	`mysql`	`mysql_database`, `mysql_schema`, `mysql_table`	`host`, `port`, `database`
Oracle	`oracle`	`oracle_database`, `oracle_schema`, `oracle_table`	`host`, `port`, `serviceName`
PostgreSQL	`postgresql`	`postgresql_database`, `postgresql_schema`, `postgresql_table`, `postgresql_view`	`host`, `port`, `database`, `schema`
Presto	`presto`	`presto_catalog`, `presto_schema`, `presto_table`	`host`, `catalog`, `schema`
SFTP	`sftp`	`sftp_folder`, `sftp_file`	`location`, `format`, `delimiter`
Snowflake	`snowflake`	`snowflake_database`, `snowflake_schema`, `snowflake_table`, `snowflake_view`	`account`, `database`, `schema`, `warehouse`
SQL Server	`sqlserver`	`sqlserver_database`, `sqlserver_schema`, `sqlserver_table`	`host`, `port`, `database`, `schema`
Trino	`trino`	`trino_catalog`, `trino_schema`, `trino_table`	`host`, `port`, `catalog`, `schema`
Vertica	`vertica`	`vertica_database`, `vertica_schema`, `vertica_table`	`host`, `port`, `database`, `schema`

Naming Conventions

For source:

Use the server type from ODCS when available
Use lowercase, no spaces or special characters
Prefer data source name over catalog name (e.g., sqlserver not purview)

For type:

Follow the pattern: <source>_<asset_type>
Common asset types: database, catalog, schema, table, view, topic, bucket, folder, file
Examples: snowflake_table, kafka_topic, s3_bucket
Use the same terminology as your data platform (e.g., Databricks uses catalog, Kafka uses topic)

Complete Examples

Example 1: Snowflake Table Asset

{
  "id": "sales-customers",
  "info": {
    "source": "snowflake",
    "type": "snowflake_table",
    "name": "CUSTOMERS",
    "qualifiedName": "SALES_DB.PUBLIC.CUSTOMERS",
    "description": "Customer master data"
  },
  "custom": {
    "account": "mycompany",
    "database": "SALES_DB",
    "schema": "PUBLIC",
    "environment": "prod"
  },
  "columns": [
    {
      "name": "CUSTOMER_ID",
      "type": "NUMBER(38,0)",
      "description": "Unique customer identifier",
      "required": true
    },
    {
      "name": "EMAIL",
      "type": "VARCHAR(255)",
      "description": "Customer email address",
      "required": true
    },
    {
      "name": "FIRST_NAME",
      "type": "VARCHAR(100)",
      "description": "Customer first name",
      "required": false
    },
    {
      "name": "LAST_NAME",
      "type": "VARCHAR(100)",
      "description": "Customer last name",
      "required": false
    },
    {
      "name": "CREATED_AT",
      "type": "TIMESTAMP_NTZ(9)",
      "description": "Account creation timestamp",
      "required": true
    }
  ]
}

Example 2: Databricks Unity Table Asset

{
  "id": "prod-sales-customers",
  "info": {
    "source": "unity",
    "type": "unity_table",
    "name": "customers",
    "qualifiedName": "production.sales.customers",
    "description": "Customer dimension table"
  },
  "custom": {
    "host": "adb-1234567890.5.azuredatabricks.net",
    "path": "/mnt/production/sales/customers",
    "format": "delta",
    "environment": "prod"
  },
  "columns": [
    {
      "name": "customer_id",
      "type": "BIGINT",
      "description": "Unique customer identifier",
      "required": true
    },
    {
      "name": "email",
      "type": "STRING",
      "description": "Customer email address",
      "required": true
    },
    {
      "name": "signup_date",
      "type": "DATE",
      "description": "Date customer signed up",
      "required": true
    }
  ],
  "relationships": [
    {
      "assetId": "prod-sales-schema",
      "relationshipType": "parent"
    }
  ]
}

Example 3: Kafka Topic Asset

{
  "id": "orders-topic",
  "info": {
    "source": "kafka",
    "type": "kafka_topic",
    "name": "orders",
    "qualifiedName": "prod-cluster.orders",
    "description": "Order events stream"
  },
  "custom": {
    "bootstrap_servers": "kafka-1.example.com:9092,kafka-2.example.com:9092",
    "cluster": "prod-cluster",
    "partitions": "12",
    "replication_factor": "3",
    "environment": "prod"
  },
  "relationships": [
    {
      "assetId": "prod-kafka-cluster",
      "relationshipType": "parent"
    }
  ]
}

Example 4: Custom Data Platform

{
  "id": "customer-dataset",
  "info": {
    "source": "custom_platform",
    "type": "custom_dataset",
    "name": "customers",
    "qualifiedName": "custom://prod/sales/customers",
    "description": "Customer dataset"
  },
  "custom": {
    "location": "s3://my-bucket/datasets/customers",
    "format": "parquet",
    "environment": "prod"
  },
  "columns": [
    {
      "name": "id",
      "type": "STRING",
      "description": "Customer ID",
      "required": true
    },
    {
      "name": "name",
      "type": "STRING",
      "description": "Customer name",
      "required": true
    }
  ]
}

Next Steps

Once you've ingested your assets:

For questions or support, refer to the SDK documentation or API reference.