Knogin
[Core Modules]

Entity Schema Evolution

When an operational ontology grows, field names change, new attributes are added, and existing attributes are retired.

Module metadata

When an operational ontology grows, field names change, new attributes are added, and existing attributes are retired.

Back to All Modules

Source reference

content/modules/entity-schema-evolution.md

Last Updated

Apr 14, 2026

Category

Core Modules

Content checksum

662e61be1241013b

Tags

modulescompliancegeospatial

Overview#

When an operational ontology grows, field names change, new attributes are added, and existing attributes are retired. Without automated migration, every schema change requires a coordinated manual migration across all stored entities — a brittle, error-prone process that slows down platform evolution and increases the risk of data loss.

The Entity Schema Evolution module provides automated schema versioning and data migration for all entity types in the Argus ontology. Each registered schema version is validated against its predecessor using Apache Avro compatibility rules before it is persisted. When an existing entity is read under a new schema version, the migration engine applies the minimum set of field-level transforms required to bring it forward, without touching the canonical stored record. Operators gain the ability to evolve the ontology continuously without coordinating manual database migrations or accepting backward-incompatible breaks.

Diagram

graph LR
    A[New Schema Definition] --> B[Compatibility Checker]
    B --> C{BACKWARD / FORWARD / FULL}
    C -->|Pass| D[Schema Registry]
    C -->|Fail| E[Violation Report]
    D --> F[Migration Transform Rules]
    F --> G[Existing Entity Data]
    G --> H[Auto-Migration Engine]
    H --> I[Updated Entities]

Last Reviewed: 2026-04-14 Last Updated: 2026-04-14

Key Features#

  • Apache Avro Compatibility Enforcement: Every new schema version is checked against the current version in the selected compatibility mode before it is accepted. BACKWARD mode ensures new readers can still read data written under the old schema; FORWARD mode ensures old readers can still read data written under the new schema; FULL mode enforces both directions simultaneously. The compatibility checker reports all violations with precise field-level explanations, enabling analysts to correct schema definitions before they cause operational disruption.

  • Schema Version Registry: Each entity type maintains a version chain in the entity_type_schema_registry table. Versions are immutable once persisted. The currently active version is flagged for quick lookup, and each version links back to its predecessor, giving a complete audit chain of ontology evolution. Organization-scoped schemas override platform-wide defaults, supporting multi-tenant ontology customisation within a single deployment.

  • Automated Field-Level Migration Transforms: Migration transforms describe the minimum set of operations required to move entity data between consecutive schema versions: ADD_FIELD (injects a default value for newly required fields), REMOVE_FIELD (drops deprecated fields), RENAME_FIELD (renames a key in place), and CHANGE_TYPE (signals a type coercion for the caller). Transforms are applied in sequence, in created_at order, so the migration path from any version to any later version is resolved by composing the transforms across intermediate steps.

  • JSON Schema Draft 2020-12 Definitions: Schemas are stored and validated as JSON Schema Draft 2020-12 documents, the current IETF standard for JSON data description. The required field set and per-field default values in the schema definition are the authoritative source for compatibility checking, ensuring the validation logic is deterministic and tool-independent.

  • Schema Registry Admin UI: The Schema Registry admin page provides a task-first interface for registering new schema versions and reviewing version history for any entity type. Each entry displays a SchemaVersionBadge showing the version number and compatibility mode. Administrators can inspect the full schema history, register a new version with inline JSON editor, and see compatibility mode transitions across the version chain.

  • Organization-Scoped Schemas: Schema definitions can be scoped to a specific organization or set as platform-wide defaults. This allows national instances to customise entity types for local data models without affecting other tenants, while inheriting the platform ontology as a baseline.

Use Cases#

  • Ontology Evolution Without Downtime: Add new required fields to entity types by registering a BACKWARD-compatible schema update with defaults. Existing entities are automatically presented with the new field populated from the default when read, without a bulk database migration.
  • Field Deprecation: Register a new schema version that removes an optional field and record a REMOVE_FIELD transform. The migration engine strips the deprecated field from entity data as it is processed, preventing stale data from surfacing in new versions of the operational UI.
  • Field Renaming: Rename a field across the ontology by registering a schema with a RENAME_FIELD transform. The transform encodes the mapping as old_name:new_name. Entities read under the new schema version have the field transparently renamed, preserving all stored values.
  • Multi-Tenant Schema Customisation: Deploy a national instance with a customised entity schema for a specific entity type while retaining the platform-wide schema for all other types. The schema registry resolves the correct definition by checking organisation scope before falling back to the platform default.
  • Schema Audit and Compliance: Maintain a full audit chain of ontology changes, including who registered each version and when. Export the version history for entity types as evidence for EDF compliance and PESCO interoperability assessments.

Integration#

  • Ontology Management: Schema versions are the authoritative type definitions consumed by the ontology management layer when resolving entity field sets.
  • Intelligence Correlation: Correlation rules that reference entity fields are checked against the current schema version to ensure field references remain valid across schema updates.
  • Evidence Management: Entity evidence records are stamped with the schema version active at the time of capture, enabling the migration engine to automatically upgrade records as they are accessed.
  • Audit Trail: All schema registration and migration events are logged to the immutable audit trail with userId, organizationId, action, resourceId, and timestamp, satisfying EDF/PESCO audit requirements.

Open Standards Compliance#