Data Source Integration Domain

Overview#

A financial intelligence unit connects a new external data provider to the platform. The provider's data format does not match the internal schema. Records arrive daily with timestamps in a non-standard format, entity names in a different field order, and a custom risk score field that needs mapping to the platform's internal scoring model. The Data Source Integration domain handles that transformation layer: the connection configuration, the data mapping definitions, the synchronisation schedule, and the error handling when something goes wrong.

Key Features#

Data source configuration and connection management.
Synchronisation scheduling with configurable intervals.
Data mapping definitions for transforming external data to internal schemas.
Ingestion workflow management with error handling and retry logic.
Integration health monitoring and alerting.
Schema mapping validation for data quality assurance.
Batch and incremental sync support.
Organisation-scoped integration management.

Use Cases#

Intelligence platforms configure connections to external intelligence data sources with scheduled synchronisation, ensuring that new data from partner feeds is available to analysts without manual import steps.

Financial crime units define data mappings for transforming incoming financial transaction feeds into the platform's internal entity schema, standardising field formats across multiple provider feeds.

Operations teams monitor integration health across all configured data sources, receiving alerts when synchronisation failures occur and tracking error rates to identify sources that consistently produce data quality issues.

Platform administrators configure batch versus incremental sync modes for each integration based on data volume and update frequency, optimising synchronisation efficiency without manually managing each provider's data pipeline.

Integration#

Integrates with data source definitions, datasource management, and connector domains for the complete data integration lifecycle.

Open Standards#

GraphQL (June 2018 specification): the entire API surface for managing connections, testing them, and discovering schemas is exposed through a GraphQL query and mutation layer, giving clients a typed, self-describing interface.
OpenAPI Specification (OAS 3.x / Swagger 2.x): API schema discovery fetches and parses openapi.json and swagger.json documents from remote data sources to extract paths, parameters, and types automatically.
OAuth 2.0 Bearer Token (RFC 6750): outbound requests to external API sources support Authorization: Bearer header authentication as one of the configurable auth strategies.
HTTP Basic Authentication (RFC 7617): outbound connector requests support Base64-encoded Authorization: Basic credentials for data sources that require this scheme.
ISO 8601 (date and time): all connection and schema timestamps (created, updated, last tested, discovered) are serialised as ISO 8601 strings using UTC-aware datetime values.
JSON (RFC 8259): connection configurations, authentication parameters, data mapping rules, transformation configs, schema field definitions, and sync results are all stored and exchanged as JSON documents.
Apache Kafka protocol: the Kafka connection type is modelled as a first-class integration point, accepting bootstrap_servers configuration for future stream ingestion alongside SQL, REST, file, and webhook source types.

Last Reviewed: 2026-02-05 Last Updated: 2026-04-14