Overview#
A fusion centre analyst finishes their shift with two hundred new intelligence documents sitting unprocessed. By the next morning, a scheduled batch job has run entity extraction across every one of them, populated the knowledge graph with new relationships, and flagged three high-confidence links to an active investigation. That is what the Analysis Jobs System does: it handles the analytical workload that would otherwise accumulate overnight and across weekends, keeping investigative pipelines moving without demanding continuous analyst attention.
The system is an asynchronous job orchestration platform that executes AI-powered analytical workflows across your entire investigation portfolio. It provides real-time visibility into running operations, predictive cost management, and seamless AI integration for autonomous investigation support.
Key Features#
Parallel Processing at Scale#
Execute hundreds of simultaneous analytical jobs across distributed worker infrastructure. Process entire investigation backlogs with zero manual intervention through scheduled batch operations. Kafka Streams manages high-throughput event ingestion for jobs that pull from streaming data sources.
Cost-Aware AI Orchestration#
Real-time token tracking and predictive cost estimation prevent budget overruns. The system automatically routes jobs to optimal AI model tiers based on complexity, budget constraints, and SLA requirements, keeping spend predictable across large-volume analytical runs.
Transparent Progress Monitoring#
Web-based dashboards display live job execution status, estimated completion times, and incremental findings as they emerge. Percentage-based progress indicators with stage-specific status updates eliminate uncertainty during long-running analyses. Progress events stream via GraphQL subscriptions.
Investigation Lifecycle Integration#
Jobs automatically link to investigations, alert clusters, and entity networks. Results populate knowledge graphs, trigger workflow actions, and generate audit-ready documentation without manual data entry. All data writes go to the PostgreSQL primary data store with organisation-level isolation enforced throughout.
Enterprise Reliability#
Automatic retry with exponential backoff, dead letter queue handling, and graceful degradation ensure high job completion rates. Failed jobs generate detailed diagnostics for rapid troubleshooting.
Use Cases#
- Automated Nightly Entity Extraction: Scheduled jobs enumerate investigations with new documents, run entity extraction across the portfolio, and update knowledge graphs overnight. Eliminates multi-week backlogs and shifts analysts to strategic case development.
- Real-Time Threat Assessment: Transaction monitoring alerts trigger threat modelling jobs that execute within seconds, generating threat scores with evidence citations. Low-scoring alerts auto-close with audit trails while high-scoring items escalate immediately.
- Pattern Recognition for Organised Crime: Weekly pattern recognition jobs analyse combined multi-agency datasets for behavioural signatures, operational methodologies, and organisational structures across large incident sets.
Integration#
Available through a complete API with Python SDK integration. Pre-built templates for common workflows including entity extraction, link analysis, anomaly detection, and predictive modelling accelerate time-to-value. The system connects with 153 third-party integrations through the connector registry for data ingestion, and feeds results into investigation, alert, and case management workflows.
Open Standards#
- GraphQL (June 2018 specification): All job queries, statistics, and real-time progress notifications are exposed through a typed GraphQL API, including cursor-based pagination and subscription events that stream incremental findings to analyst dashboards.
- JSON Web Token / RFC 7519 (RS256 / JWKS): Every GraphQL resolver enforces RS256-signed JWT validation via a JWKS endpoint before granting access to job data, ensuring only authenticated, tenant-scoped callers can read or trigger analytical workloads.
- ISO 8601 / RFC 3339: All job lifecycle timestamps (created, started, completed) are serialised in UTC ISO 8601 format across the API and stored representation, ensuring unambiguous cross-system interoperability.
- OASIS STIX 2.1: Threat modelling and entity extraction jobs produce and consume STIX 2.1 structured domain objects; the platform's connector base validates output against the OASIS STIX 2.1 JSON schema before writing to the knowledge graph.
- Apache Kafka (distributed event streaming): High-throughput job triggers and streaming data source ingestion use the Kafka event streaming protocol; the platform references NATO STANAG 4559 interoperability guidance for message-oriented middleware deployment in federated environments.
- SQL / ISO/IEC 9075: All job state, progress, token usage, and cost data are persisted in a PostgreSQL relational store, with organisation-level tenant isolation enforced through SQL WHERE predicates on every query.
- JSON / RFC 8259: Job findings, configuration parameters, and runtime metadata are stored and transmitted as JSON documents, providing a broadly interoperable serialisation format for downstream consumers and audit systems.
Last Reviewed: 2026-02-05 Last Updated: 2026-04-14