Data Connector Framework

Overview#

The 153 pre-built integrations in the Argus marketplace cover the most common data sources across intelligence, financial crime, and government domains. But every organisation has at least one system that is not common: a proprietary case management platform built in-house fifteen years ago, a classified database with a bespoke API that no commercial vendor supports, or a partner feed delivered over a protocol that predates modern standards. For those cases, the Data Connector Framework provides the tools to build production-ready connectors without writing integration boilerplate from scratch.

The framework supplies SDKs across eight programming languages, standardised connector interfaces, built-in authentication handling for nine authentication methods, and deployment tools that take a connector from local development through to managed production operation. Teams in intelligence agencies, law enforcement, and critical infrastructure can build connectors that meet the same security, reliability, and maintainability standards as the connectors already in the marketplace, then share them across their organisation or publish them for broader use.

Key Features#

• Multi-Language SDK Support: Build connectors in Python, Java, JavaScript/TypeScript, Go, C#, Rust, Scala, or Ruby using native SDKs tailored to each language's ecosystem and strengths. Teams can work in the language they already know.
• Standardised Connector Interfaces: Consistent source and destination connector APIs across all languages ensure compatibility with orchestration, monitoring, and marketplace infrastructure. A connector built in Go and one built in Python behave identically from the platform's perspective.
• Authentication Framework: Built-in support for nine authentication methods including OAuth 2.0, API keys, client certificates, mutual TLS, and cloud provider identity services, with automatic token refresh and credential rotation. No boilerplate security code required.
• Configuration Validation: Schema-based configuration validation with type checking, required field enforcement, constraint verification, secret masking, and actionable error messages that identify problems before a connector reaches production.
• Change Data Capture Support: Built-in framework for real-time data replication using database transaction logs, enabling low-latency synchronisation without impacting source system performance.
• Schema Discovery: Automatically discover and map source and destination schemas to simplify connector setup and reduce configuration errors during initial deployment.
• Testing and Quality Assurance: Integrated testing framework with unit test, integration test, and load testing utilities to confirm connector reliability before production deployment.
• Error Handling and Retry Logic: Built-in exponential backoff, circuit breakers, and health check patterns handle transient failures and prevent cascade issues in production.
• Secrets Management Integration: Connect to enterprise secrets management solutions for secure credential storage and rotation. Secrets are never stored in connector configuration files.
• Deployment Flexibility: Deploy connectors locally with hot-reload during development, in containers, as serverless functions, or as managed services on the integration platform.

Use Cases#

• Proprietary System Integration: Build connectors for internal or legacy systems that lack pre-built marketplace integrations, using the SDK that matches your team's preferred programming language. Classified or sensitive systems that cannot be exposed to commercial connector vendors can still be integrated.
• ISV Integration Development: SaaS vendors can create and publish connectors for their platforms, making it straightforward for customers to integrate with the broader data ecosystem.
• High-Performance Data Pipelines: Use the Go or Rust SDK for performance-critical connectors requiring low memory footprint, high concurrency, or safety-critical operation, particularly relevant for real-time threat feed ingestion.
• Rapid Prototyping: Quickly build and iterate on connector designs using the Python or Ruby SDK, with built-in testing utilities and hot-reload development workflows. Production hardening comes later using the same framework.
• Enterprise Authentication: Connect to systems with complex authentication requirements such as mutual TLS, cloud provider identity services, or custom authentication flows without writing boilerplate security code.

Integration#

Connectors built with the framework automatically integrate with the Argus ingestion pipeline, workflow orchestration engine, monitoring dashboards, and managed services infrastructure. All ingested data flows into PostgreSQL as the primary data store. The 153 third-party integrations available in the marketplace were built using the same framework, so custom connectors inherit the same operational characteristics as marketplace connectors.

Open Standards#

• OASIS STIX 2.1: Every connector built with the framework must declare an open standard as its primary output contract; STIX 2.1 is the supported contract for threat-intelligence connectors, and the ingestion pipeline validates extracted data against it before structural type mapping.
• OAuth 2.0 (RFC 6749): The built-in authentication framework supports the OAuth 2.0 client credentials grant, with automatic token refresh and credential rotation, so connector authors need not write authentication boilerplate.
• OpenAPI Specification (OAS 3.x): Schema discovery for REST API data sources probes well-known OpenAPI/Swagger document paths and parses the retrieved specification to map endpoint parameters to connector field definitions.
• OASIS CAP 1.2: The Common Alerting Protocol version 1.2 is a recognised open standard output contract for connectors delivering emergency and alerting data into the ingestion pipeline.
• NIEM (National Information Exchange Model): NIEM is a supported output contract for connectors serving justice, law enforcement, and government data-sharing use cases, ensuring interoperability with partner agencies.
• OpenLineage Specification: The ingestion pipeline emits START, COMPLETE, and FAIL run events conforming to the OpenLineage specification so that data lineage can be tracked across connector extraction jobs.
• W3C PROV-DM: Provenance records for each ingested entity are written according to the W3C Provenance Data Model, capturing wasGeneratedBy and wasAttributedTo relationships for audit and chain-of-custody purposes.
• GraphQL: All connector registry operations, schema discovery results, and ingestion pipeline queries are exposed through a GraphQL API, with compile-time type checking used as the structural mapping contract between connector output and the platform data model.

Last Reviewed: 2026-02-05 Last Updated: 2026-04-14