AWS Glue Integration

Integration of AWS Glue Data Catalog with Entropy Data.

The AWS Glue integration is managed within Entropy Data. Configure the connection and sync schedule to start syncing with AWS Glue Data Catalog.

No additional deployments are needed.

Features

• Asset Synchronization: Sync databases and tables from AWS Glue Data Catalog to Entropy Data as Assets.
• S3 Folder Ingestion: Sync S3 folders as data sources, in addition to Glue tables. For unsupported Glue table types (e.g., external tables pointing to S3), the integration automatically falls back to ingesting the underlying S3 folder.
• Iceberg Tables: Iceberg tables in the Data Catalog keep their format and the location of their metadata file. Columns that Iceberg dropped from the current schema are left out.
• S3 Tables: Sync the table buckets, namespaces and tables of Amazon S3 Tables through the s3tablescatalog catalog. See S3 Tables.
• Metadata Writer: Write the descriptions and tags of your data contracts to the tables and columns in the Glue Data Catalog. See Metadata Writer.
• Business Context Writer: Write your semantic namespaces as business glossaries to the Glue Data Catalog and link the tables and columns of your data contracts to the terms. See Business Context Writer.

Ingestion

The ingestion reads the databases and tables of the Glue Data Catalog into Entropy Data as assets, on a schedule.

Prerequisites

To start, navigate to Settings > Integrations > Add Integration. This opens a wizard that guides you through configuring the integration.

Select the Integration Type

Select the Purpose

Choose the purpose Read Glue Assets and click Next. The other two purposes are the metadata writer and the business context writer.

Select AWS Glue from the list of available integrations, and choose the purpose Read Glue Assets.

Configure the Credentials

AWS Glue supports two authentication methods. Choose one in the Authentication Method section of the wizard.

Whichever method you use, the effective AWS identity needs permissions for glue:GetDatabases and glue:GetTables. To sync S3 Tables, it also needs glue:GetCatalog and glue:GetCatalogs, plus read permissions on the S3 Tables service itself: the Data Catalog forwards every call for a table bucket to S3 Tables, which checks the caller's identity again. You may use the following AWS policy to grant them:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetCatalog",
                "glue:GetCatalogs",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3tables:ListTableBuckets",
                "s3tables:GetTableBucket",
                "s3tables:ListNamespaces",
                "s3tables:GetNamespace",
                "s3tables:ListTables",
                "s3tables:GetTable",
                "s3tables:GetTableMetadataLocation",
                "s3tables:GetTableData"
            ],
            "Resource": "*"
        }
    ]
}

Leave out the second statement if you have no table buckets. When a permission is missing, the run still succeeds for everything else and carries a note: without glue:GetCatalogs the S3 Tables are skipped, without s3tables:ListTableBuckets no table bucket is visible, and a bucket the identity may not read is skipped with the reason AWS gives. Assets from earlier runs are kept in all three cases, so nothing disappears because of a permission change.

IAM User Access Credentials

Authenticate with a long-lived AWS access key. Refer to the AWS documentation for creating a new access key. While creating the access key, select the use case 'Other'.

Cross-account access. When you set an Assume Role ARN, Entropy Data calls AWS STS AssumeRole with the access key and uses the returned temporary credentials to read the target account's Glue catalog. The IAM user additionally needs the sts:AssumeRole permission, and the target role's trust policy must allow that IAM user to assume it.

Entra Federation

Authenticate to AWS using Microsoft Entra ID as an OIDC identity provider via STS AssumeRoleWithWebIdentity — no IAM user access keys are stored. Entropy Data obtains a token from Entra and exchanges it for temporary AWS credentials on the target role.

This option is only selectable once an administrator has configured Entra federation for your organization; otherwise it is disabled in the wizard.

Note: The central Entra credentials are configured at deployment time by an administrator, per organization, via application.integration.aws.entra.<organization-id>.* properties — either client credentials (tenant-id, client-id, client-secret, scope) or a user-assigned managed identity (managed-identity-client-id, resource). See Configuration.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Configure Filters

Configure filters to limit which assets are synchronized. Both include and exclude filters are supported. For AWS Glue, filters can be applied to Databases, Tables and S3 Table Buckets. The database and table filters also apply to the namespaces and tables of a table bucket.

Filters support '*' as a wildcard character to match any number of characters.

Configure Schedule

Set a schedule to automatically synchronize assets. You can choose from predefined schedules or define a custom schedule using the cron expression format.

Note: All schedules use the UTC timezone, so make sure to take this into account when configuring your schedule. Please do not synchronize the assets more than once or twice per day. We reserve the right to disable the integration if this happens. You will be able to trigger a synchronization manually if you need an immediate update.

Complete the Integration Configuration

Choose a unique name for the integration, review your configuration, and click Create Integration.

Next Steps

The integration is now configured and will run according to the schedule. To check the integration status, navigate to Settings > Integrations. Here you'll find the current status and the last 10 integration runs.

You can adjust the integration configuration and credentials at any time. The configuration is saved in YAML format with syntax validation support in the editor.

Note: The previously stored secret access key is not displayed in the edit view for security reasons.

Deselecting the Enabled checkbox disables the automatic schedule. Manual integration runs are still possible.

S3 Tables

Amazon S3 Tables are Apache Iceberg tables that S3 manages in table buckets. Once a table bucket is integrated with AWS analytics services, the AWS Glue Data Catalog shows it under the catalog s3tablescatalog: one child catalog per table bucket, each namespace as a database, each table as a table. The integration reads that hierarchy on every run, after the default catalog.

What is synced

The qualified name of a table is <account>:s3tablescatalog/<bucket>.<namespace>.<table>. A table bucket itself is not an asset; use the table bucket filters to choose buckets.

Data contract servers

When a data contract is created from an s3_table asset, it gets two servers:

• an athena server with the catalog s3tablescatalog/<bucket> and the namespace as schema, which Amazon Athena and the Data Contract CLI can query, and
• an iceberg server (ODCS 3.2.0) with the AWS Glue Iceberg REST endpoint as catalogUrl and <account>:s3tablescatalog/<bucket> as warehouse, for Iceberg clients.

Iceberg tables in the default catalog get an iceberg server next to their s3 server as well.

Notes

• Table and column names must be lowercase, otherwise the Data Catalog does not show the table.
• In most regions the integration works with IAM alone. In regions where AWS requires Lake Formation for the S3 Tables integration, the identity also needs the Lake Formation DESCRIBE permission on the catalog objects.
• A table's storage location is an internal bucket of S3 Tables that cannot be browsed, so s3_table assets link to the Glue console only.

Metadata Writer

The metadata writer is the counterpart of the ingestion. The ingestion reads the metadata of the Glue Data Catalog into Entropy Data. The metadata writer writes the descriptions and tags of your data contracts to the tables and columns in the Glue Data Catalog. It runs whenever a data contract changes. There is no filter and no schedule.

Prerequisites

The metadata writer authenticates like the ingestion: with the access key of an IAM user, optionally assuming a role, or through Entra federation with a role. Use an identity of its own, so it holds only the permissions below.

The identity needs glue:GetTable and glue:UpdateTable on the tables your data contracts name, and glue:GetDatabases for Test Connection. Grant them for the whole catalog to cover every current and future table:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:UpdateTable", "glue:GetCatalog", "glue:GetCatalogs"],
      "Resource": [
        "arn:aws:glue:<region>:<account>:catalog",
        "arn:aws:glue:<region>:<account>:database/*",
        "arn:aws:glue:<region>:<account>:table/*/*"
      ]
    }
  ]
}

Or grant them per table. This covers only the tables your data contracts name today, so a new data contract needs a new statement. Policy on the integration page shows both variants for the tables your data contracts name.

Tables of S3 table buckets are written through their Iceberg metadata, which S3 Tables checks once more behind the catalog. For them the identity also needs, on the table buckets and their tables, s3tables:ListTableBuckets, s3tables:GetTableBucket, s3tables:ListNamespaces, s3tables:GetNamespace, s3tables:ListTables, s3tables:GetTable, s3tables:GetTableMetadataLocation, s3tables:GetTableData, s3tables:UpdateTableMetadataLocation and s3tables:PutTableData. The policy on the integration page lists them when a data contract names such a table.

An Iceberg table in the account's own catalog is read and committed through its metadata file in S3, as the identity. For such tables the identity also needs s3:GetObject, s3:PutObject and s3:ListBucket on the table's location:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
  "Resource": ["arn:aws:s3:::<bucket>", "arn:aws:s3:::<bucket>/<prefix>/*"]
}

Where Lake Formation governs the catalog, also grant the identity DESCRIBE and ALTER on the tables in Lake Formation.

Select the Integration Type and Purpose

Navigate to Settings > Integrations > Add Integration, select AWS Glue from the list of available integrations, and choose the purpose Write Glue Catalog Metadata.

Configure the Credentials

Test Connection signs one read of the catalog with the credential. Policy shows the IAM policy from the prerequisites for the tables your data contracts name today.

Note: Credentials are stored encrypted in the Entropy Data database. To enable encryption in your environment, set a 64 hex character APPLICATION_ENCRYPTION_KEYS in your environment (see Configuration).

Complete the Integration Configuration

Choose a unique name for the integration and click Create Integration. From now on, the metadata writer writes the tables your data contracts name.

You can adjust the integration configuration and credentials at any time. Deselecting the Enabled checkbox pauses the metadata writer. It keeps the changes and writes them once it is enabled again. Deleting the integration leaves what it wrote in the Glue Data Catalog as it is.

Which data contracts are written

The metadata writer considers data contracts in the ODCS format with the status active. Drafts, other statuses, branches, and data contracts in the DCS format are skipped.

A data contract names its tables in its servers. The metadata writer reads four server types:

If a server names a table, that table is written. Otherwise, every schema object of the data contract is written as a table in that database. A server is written by the metadata writer of its regionName; a server without a region by every metadata writer.

What is written

Glue tables take no AWS resource tags, so tags become table and column parameters under their own keys. A tag key:value or key=value becomes a parameter with that key and value. A tag without a value becomes a key with an empty value. Parameters appear as table properties in the Glue console and in Athena's SHOW TBLPROPERTIES; column parameters are visible through the API and the ingestion only.

A table is written when its schema object has a description or links a semantic concept. A column is written when it has a description or tags. Every written table description ends with the marker line entropy-data:<integration id>, so the metadata writer recognizes its own text later on.

Glue holds 2,048 characters in a table description and 255 in a column comment, without line breaks. A longer text is cut at a word boundary with an ellipsis, line breaks in a column comment become spaces, and the full text is stored in the parameter entropy_data_description next to it. The ingestion reads the full text back as long as the comment is still what the metadata writer rendered of it.

In addition, the metadata writer sets parameters that link back to Entropy Data:

If an element links a semantic concept but has no description of its own, the description of the concept is written and marked with entropy_data_comment_source. When the ingestion reads such a table, it keeps the concept as the semantic link of the asset instead of taking the comment as its description.

Iceberg tables and S3 Tables

An Iceberg table, in the account's catalog or in an S3 table bucket, is not written through its Glue entry: the next commit of any engine would rewrite the entry from the Iceberg metadata. The metadata writer commits to the Iceberg metadata instead, through the AWS Glue Iceberg REST endpoint:

• the description becomes the table property comment, with the marker line,
• the tags and the entropy_data_* keys become table properties,
• the column descriptions become the field docs of a new schema version, which Glue shows as the column comments,
• the tags of a column become table properties entropy_data_column.<column>.<key>, since Iceberg columns carry no properties of their own.

The entry of an Iceberg table in the account's catalog is updated afterwards, so the table description shows in Glue and Athena; a column comment Glue mirrored from a long or multi-line doc is sent back on one line within 255 characters, the doc keeps the full text. The entry of a table in an S3 table bucket cannot be updated and shows none of the Iceberg properties: the description, the tags and the entropy_data_* keys of such a table live in the Iceberg metadata only. Athena shows them with SELECT key, value FROM "s3tablescatalog/<bucket>"."<namespace>"."<table>$properties"; Iceberg clients see them as table properties; the ingestion reads them through the REST catalog for the tables the metadata writer wrote. The identity needs the same glue:UpdateTable permission, S3 access on the table's location for tables in the account's catalog, and the s3tables actions from the prerequisites for tables of table buckets.

What is left untouched

• Parameters set by other tools. A parameter key the metadata writer did not set before is left untouched, even when a data contract carries the same key. Only keys the metadata writer set itself are updated or removed. The entropy_data_* keys always belong to the metadata writer.
• Parameters engines read. Keys such as classification, EXTERNAL, table_type, metadata_location, projection.*, iceberg.*, spark.* or write.* steer Athena, Spark, crawlers and Iceberg. A tag under such a key is not written, with a note on the table.
• Everything else in the table's entry. The storage descriptor, the partition keys, the view definition and the other parameters go back as they were read.
• Tables that do not exist are not created. The table is marked as unresolved until it appears in the Glue Data Catalog, or the data contract no longer names it.

When it writes

Creating, updating, or deleting a data contract puts an event in the queue of the metadata writer. Updating or deleting a semantic concept that data contracts reference does the same. The metadata writer writes about a minute later. Several changes of the same data contract in that time result in one write. If a table already has the same text and parameters, nothing is written.

A failed write is retried with growing intervals and given up after eight attempts. If the credentials are refused, the waiting events are given up. If the catalog is unreachable or throttles, the queue pauses and continues later.

When a data contract is no longer active, is deleted, or no longer names a table, the metadata writer retires the table. It removes the parameters it set and clears the descriptions and comments it wrote. A description written by someone else is left untouched. When an element loses its description, its comment is cleared and the parameters set for it are removed.

After a write, the tables that exist as assets in Entropy Data are read again through the ingestion that imported them, including the s3_folder assets of Glue tables over S3. This way, the asset shows what was just written. For written tables the ingestion reads the parameters as tags, skips the entropy_data_* keys and the keys engines read, and strips the marker line from the description.

Next Steps

To check the status of the metadata writer, navigate to Settings > Integrations and open the integration. The page shows:

• Queue: the pending events, and the events processed in the last 24 hours, with their attempts and errors. Discard pending drops the waiting events. Write again puts a processed event back in the queue. Retry given up puts the events given up in the last day back in the queue.
• Glue Data Catalog: the tables the metadata writer wrote, as a tree of catalogs, databases, and tables. A table is In sync, Unresolved (there is no such table in the catalog), or Failed. Retry failed writes the data contracts with failed or unresolved tables again.
• Credentials: the AWS identity, with Test Connection and Policy.

Click a table to see the description and parameters the metadata writer last wrote to it and its columns, the asset it was imported as, and the changes that triggered the writes. It shows what was last written, not the data contract as it is today.

API

You can also manage metadata writers via the REST API under /api/integrations/glue-catalog-metadata:

The credential in a PUT takes type (iamUserAccessCredentials or entraFederation), glueRegion, glueAccessKeyId, glueSecretAccessKey and glueAssumeRoleArn. Ingestions are managed under /api/integrations/ingest.

Business Context Writer

The business context writer writes your semantics and data contracts into the business context of the Glue Data Catalog. Each semantic namespace becomes a business glossary with a term per concept. Each table a data contract names gets the terms of the concepts its schema object and columns link, and the data contract itself as a form attached to the table. Agents and users that search the catalog find the tables by the meaning of the terms, and read the data contract behind a table.

Amazon Quick's Glue Data Catalog connector, itself in preview, reads the table metadata only and does not pick up business context yet (checked in October 2026); what Quick sees of a table is what the metadata writer puts there.

The business context writer does not change the tables themselves. It can run next to the metadata writer on the same tables: the metadata writer writes the descriptions and tags into the table's entry, the business context writer writes the terms and the form into the business context of the table.

Prerequisites

The business context writer authenticates like the ingestion and the metadata writer: with the access key of an IAM user, optionally assuming a role, or through Entra federation with a role. Use an identity of its own, so it holds only the permissions below.

Business context objects have no ARNs of their own yet, so the policy names every resource:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetAsset", "glue:ListIterableForms", "glue:BatchGetIterableForms",
        "glue:AssociateGlossaryTerms", "glue:DisassociateGlossaryTerms",
        "glue:PutFormType", "glue:GetFormType", "glue:PutAttachment", "glue:DeleteAttachment",
        "glue:CreateGlossary", "glue:UpdateGlossary", "glue:GetGlossary", "glue:ListGlossaries", "glue:DeleteGlossary",
        "glue:CreateGlossaryTerm", "glue:UpdateGlossaryTerm", "glue:GetGlossaryTerm", "glue:ListGlossaryTerms", "glue:DeleteGlossaryTerm"
      ],
      "Resource": "*"
    }
  ]
}

Policy on the integration page shows this policy.

Select the Integration Type and Purpose

Navigate to Settings > Integrations > Add Integration, select AWS Glue from the list of available integrations, and choose the purpose Write Glue Business Context.

Configure the Credentials

The credentials are the same as for the metadata writer: the authentication method, the AWS region, the access key or the role. One integration writes one region; choose a region in which AWS offers business context.

Test Connection lists the glossaries of the catalog with the credential. It fails in a region without business context.

Complete the Integration Configuration

Choose a unique name for the integration and click Create Integration. The business context writer then writes every semantic namespace as a glossary, and the tables every active data contract names in the catalog.

You can adjust the integration configuration and credentials at any time. Deselecting the Enabled checkbox pauses the business context writer. It keeps the changes and writes them once it is enabled again. Deleting the integration leaves the glossaries, terms and attachments in the Glue Data Catalog as they are.

What is written

A glossary carries the description of the namespace. A term carries the concept's name; the first paragraph of the concept's description as its short description; and the whole description, followed by the link to the concept in Entropy Data, as its long description. An inline property is named with its entity, such as Order: total amount. Groups structure an ontology and become no terms. Branches of a namespace are not written.

Glue takes 10 terms on a table and 5 on a column. A schema object or property that links more concepts gets the first ones, with a note on the table. A concept whose namespace has no glossary yet is noted as well; the table is written again with the retry once the glossary holds the term.

The form EntropyDataContract is a form type the integration creates in your account. It carries the id, name, version, status and link of the data contract, the element of the contract the table stands for, and the tags of the schema object. The fields are searchable through the Glue search API.

Which data contracts are written

The business context writer considers the same data contracts and servers as the metadata writer: data contracts in the ODCS format with the status active, on glue, athena, iceberg and s3 servers of the integration's region. A table of an S3 table bucket is addressed by the ARN of its nested catalog; whether business context covers such tables depends on the AWS preview.

What is left untouched

• The table's entry. The description, the parameters and the columns of a table are not changed. The metadata writer does that.
• Glossaries, terms and attachments of other tools. Only the terms and the attachment the business context writer put on a table are ever removed again. A glossary or a term of the same name that exists already is taken over and brought up to date, not doubled.
• Business descriptions. The business name and description of a table asset are not written.
• Tables that do not exist are not created. The table is marked as unresolved until it appears in the Glue Data Catalog, or the data contract no longer names it.

When it writes

Creating, updating or deleting a concept, a namespace or a data contract puts an event in the queue of the business context writer. It writes about a minute later; glossaries first, so the terms are there when the tables are linked. Several changes in that time result in one write. If a glossary or a table already shows what the namespace or the contract says, nothing is written.

A failed write is retried with growing intervals and given up after eight attempts. If the credentials are refused, the waiting events are given up. If the catalog is unreachable or throttles, the queue pauses and continues later.

When a concept is deleted, its term is taken off every table and column the writer put it on, then deleted. When a namespace is deleted, its terms and its glossary go. When a data contract is no longer active, is deleted, or no longer names a table, the writer takes its terms and its form off the table.

Next Steps

To check the status of the business context writer, navigate to Settings > Integrations and open the integration. The page shows:

• Queue: the pending events, and the events processed in the last 24 hours, with their attempts and errors. Discard pending drops the waiting events. Write again puts a processed event back in the queue. Retry given up puts the events given up in the last day back in the queue.
• Glossaries: one row per semantic namespace, with the glossary's name and id in Glue, the number of terms and the objects whose write failed.
• Tables: the tables the writer linked, as a tree of catalogs, databases and tables. A table is In sync, Unresolved (there is no such table in the catalog), or Failed. Retry failed writes the namespaces and data contracts with failed or unresolved items again.
• Credentials: the AWS identity, with Test Connection and Policy.

Click a table to see the terms on the table and its columns and the form the writer last put there. Click a glossary to see the glossary as written, with its id in Glue, and its terms by concept: the name each term was written under, its id, its state with the last error, and when it was last written. Failed terms come first.

API

You can also manage business context writers via the REST API under /api/integrations/glue-business-context:

The credential in a PUT takes the same fields as the metadata writer's.