Open Data Product Standard Details

Overview

This document describes how existing Data Product Specification (DPS) fields are mapped to the Open Data Product Standard (ODPS) specification format in Entropy Data. When migrating from DPS to ODPS or when working with both formats, understanding this mapping is crucial for maintaining data consistency and ensuring proper functionality.

Core Entity Mapping

Data Product Level Mapping

Extended ODPS Fields

ODPS introduces several fields not present in DPS that are stored as custom fields in the internal model:

apiVersion and kind are handled completely independent in the migration of a DPS to ODPS. kind is always set to DataProduct and apiVersion is always set to v1.0.0, as this is the current version of the ODPS specification.

Output Port Mapping

Direct Field Mapping

Custom Property Mapping for Output Ports

Fields without direct ODPS equivalents are mapped to customProperties:

ODPS-Specific Output Port Fields

Input Port Mapping

Core Field Mapping

ODPS-Required Input Port Fields

Transformation Logic

DPS to ODPS Conversion

The conversion process follows these principles:

1. Direct Mapping: Fields with direct equivalents are mapped 1:1
2. Custom Properties: DPS custom fields become ODPS custom properties with structure:

{
  "property": "fieldName",
  "value": "fieldValue",
  "description": null
}

3. Reserved Properties: Certain custom properties are reserved for system fields (see exclusion lists below)
4. Link Transformation: DPS links are converted to ODPS authoritative definitions:

{
  "type": "linkName",
  "url": "linkHref",
  "description": null
}

Exclusion Filters

Data Product Level Exclusions

The following custom properties are handled specially and not included in the general custom properties list:

• icon
• assets
• version
• tenant
• description.limitations
• description.usage
• description.authoritativeDefinitions
• description.customProperties
• managementPorts
• support
• productCreatedTs

Output Port Level Exclusions

These properties are mapped to specific OutputPort fields:

• environment
• location
• autoApprove
• containsPii
• server
• status
• assets

Input Port Level Exclusions

These properties are mapped to specific SourceSystemInputPort fields:

• sourceSystemInputPortId
• sourceSystemId
• displayName
• description
• type
• location
• version
• contractId

Best Practices

For API Consumers

1. Use appropriate endpoints based on desired format (?specification=odps vs default DPS)
2. Handle custom properties when working with ODPS format
3. Validate required fields are present when creating/updating
4. Consider field limitations - some DPS fields may be lost in ODPS conversion if not in custom properties

For SDK Users

1. Data Product parsing: Make sure to parse the correct format
2. Test conversion with sample data to verify field preservation
3. Update client applications to handle both formats if necessary
4. Plan for custom property usage in ODPS format
5. Consider feature flags to control specification type availability

Version Compatibility

• DPS Version: 0.0.1 (default for existing data products)
• ODPS Version: v1.0.0 (current implementation)
• Backward Compatibility: Maintained through service layer abstraction
• Forward Compatibility: New ODPS fields added as custom properties in DPS model

Note: This mapping is implementation-specific to Entropy Data and may differ from other ODPS implementations. Always refer to the actual service code for the most current mapping logic.