Transform Domain

Overview#

An analyst imports a batch of financial records from a legacy accounting system that uses a proprietary CSV dialect. Before any investigative work can begin, those records need to be normalised into the platform's standard transaction format, dates converted, currency codes mapped, and null fields handled consistently. Rather than writing a one-off script, she selects the appropriate transform from the registry and the pipeline handles the rest. That reusable, catalogued approach is what the Transform domain provides.

The domain offers a registry of available data transforms within the platform. Transforms are operations that convert, enrich, or process data from one format to another, enabling automated data processing pipelines and format conversions across investigative and analytical workflows. Transform definitions are stored in PostgreSQL and are discoverable through a browse and lookup API.

Key Features#

• Transform Registry: Browse a catalog of all available data transforms with descriptions of their input requirements, output formats, and processing capabilities.

• Transform Discovery: Look up specific transforms by identifier or browse the complete registry to find the right transformation for a given data processing need.

• Format Conversion: Access transforms that convert data between different formats, enabling interoperability between diverse data sources and platform components.

• Data Enrichment: Use transforms that add value to existing data by enriching it with additional context, calculations, or derived attributes.

• Pipeline Integration: Reference registered transforms within automated data processing pipelines for consistent, repeatable data operations.

Mermaid Diagram#

Use Cases#

• Law Enforcement: Apply transforms to incoming data from external case management or records systems to normalise it into platform-compatible formats for analysis and investigation.

• Intelligence Analysis: Convert data between formats when moving between different platform components or preparing intelligence products for export to partner agencies.

• Financial Crime: Chain transforms in automated pipelines to process transaction data through multiple stages of conversion, currency normalisation, and entity enrichment.

• Critical Infrastructure: Use transforms to bridge format differences when exchanging telemetry or operational data with external systems and partner organisations running different standards.

Integration#

The Transform domain supports data processing across the platform:

• Data Ingestion: Transforms normalise incoming data from diverse sources.
• Evidence Management: Evidence data can be processed through registered transforms.
• Export: Outbound data transforms support external format requirements.
• Workflow Automation: Transforms are components in automated processing pipelines.

Open Standards#

• GraphQL (June 2018 specification): The transform registry and discovery API is exposed as a GraphQL query (listTransforms), with permission enforcement and JSON scalar responses defined in the platform's Strawberry-based schema.
• OASIS STIX 2.1: Transforms operating on cyber-threat intelligence data produce and validate output against the STIX 2.1 object model (SDOs, SROs, SCOs), using the two-phase extraction and structural mapping architecture defined in the connector base classes.
• NIEM (National Information Exchange Model) v5.0: Transforms serving justice, emergency management, and defence data exchange use the NIEM JSON Specification for extraction output, with structural JSON-LD @context validation before any Argus type mapping occurs.
• OASIS CAP v1.2 (Common Alerting Protocol): Transforms handling emergency and public-safety alerting data validate output against the mandatory CAP v1.2 alert fields (identifier, sender, sent, status, msgType, scope) before normalisation into the platform's entity format.
• OpenLineage Specification: Pipeline runs that invoke registered transforms emit OpenLineage RunEvent records, enabling downstream data lineage queries over the directed acyclic graph of transform executions.
• W3C PROV-DM (Provenance Data Model): The ingestion service records a provenance event via PROV-DM after each transform-driven ingestion job, linking output entities to their originating activity and agent.
• ISO 8601 (Date and Time): The entity extractor and schema mapper normalise all timestamp fields to ISO 8601 format during transform execution, ensuring consistent datetime representations across heterogeneous source data.

Last Reviewed: 2026-02-05 Last Updated: 2026-04-14