Data Sources Management

Overview#

An investigation platform is only as useful as the data feeding it. The Data Sources Admin Panel centralises every external integration feeding the platform, from threat intelligence feeds and OSINT sources to SIEM exports and custom internal APIs. Administrators configure connections once, monitor them continuously, and resolve ingestion failures without touching individual integration code.

This matters for organisations that aggregate from multiple sources: a law enforcement agency pulling from national intelligence feeds alongside commercial threat data, or a financial institution correlating public data sources with internal security tooling.

Key Features#

• Data Source Catalog: Browse and manage available integrations organised by category: threat intelligence, security tools, external databases, cloud services, and custom APIs. Each source shows its current status, record count, and last sync time at a glance.

• Connection Management: Configure authentication credentials (API keys, OAuth 2.0, basic auth, certificates), proxy settings, and connection parameters per source. Built-in connection testing validates credentials and connectivity before a source is enabled.

• Flexible Sync Configuration: Choose from real-time (WebSocket/SSE), continuous polling, scheduled, or on-demand synchronisation. Supports full sync, incremental sync, delta sync, and selective sync filtered by criteria specific to the source.

• Real-Time Monitoring: Live dashboards show sync health, record counts, success/failure rates, and data freshness for all connected sources. Colour-coded status indicators provide at-a-glance health visibility.

• Data Transformation: Map source fields to platform fields, apply transformations, set default values, handle missing data, and perform type conversions during ingestion. Pre- and post-sync filtering, enrichment, and deduplication are available without custom code.

• Error Tracking and Retry: Comprehensive error logging with automatic retry logic and exponential backoff. View error messages, failed queries, rate limiting events, and timeout details per source to diagnose problems quickly.

• Data Quality Monitoring: Track completeness, accuracy, freshness, consistency, and reliability metrics per source. Automated quality checks include required field validation, format validation, range checking, and duplicate detection.

Supported Data Sources#

Threat Intelligence Feeds#

AlienVault OTX, MISP, ThreatConnect, Recorded Future, IBM X-Force, Anomali ThreatStream, RiskIQ, VirusTotal, and others.

SIEM Integrations#

Splunk, QRadar, ArcSight, LogRhythm, Elastic SIEM, Microsoft Sentinel, Sumo Logic, Chronicle.

OSINT Sources#

Social media APIs, domain registrars (WHOIS), certificate transparency logs, DNS records, IP geolocation, and company registries.

Custom APIs#

Connect any REST or GraphQL endpoint with configurable authentication, field mapping, and pagination strategies for proprietary or internal data sources.

Use Cases#

• Law enforcement agencies aggregating national and international intelligence feeds alongside commercial threat data for correlated investigative analysis.
• Government departments pulling from authoritative public registers and third-party data brokers under controlled, auditable conditions.
• Intelligence organisations ingesting OSINT at scale with automated quality checks and deduplication that prevent noisy data from degrading analysis quality.
• Financial institutions connecting internal SIEM exports and external threat feeds to correlate financial crime signals.
• Critical infrastructure operators integrating sector-specific threat intelligence with platform investigations.

Open Standards#

• OAuth 2.0 (RFC 6749): The connector framework implements the OAuth 2.0 client credentials flow to authenticate against external data sources, with automatic token caching and refresh.
• OASIS STIX 2.1 / TAXII 2.1: Threat intelligence data sources are ingested, polled, and exported as STIX 2.1 bundles via configured TAXII 2.1 feed endpoints, enabling interoperability with platforms such as MISP and OpenCTI.
• HTTP Basic Authentication (RFC 7617): Base64-encoded username/password credentials are supported as a connector authentication method for sources that require HTTP Basic Auth.
• Web Linking / HTTP Link Header (RFC 5988 / RFC 8288): The generic REST connector parses the HTTP Link header to drive cursor-based pagination across external APIs that follow this convention.
• JSONPath (IETF RFC 9535): Field mapping, record extraction, and cursor resolution during data transformation are expressed as JSONPath expressions applied to source API responses.
• CVE / CVSS / CWE / CPE (NIST NVD): The NVD connector ingests vulnerability records structured around Common Vulnerabilities and Exposures identifiers, Common Vulnerability Scoring System scores, Common Weakness Enumeration IDs, and CPE 2.3 platform strings.
• ISO 8601: Date and time parameters passed to external APIs (including the NVD CVE API) use ISO 8601 formatted strings to ensure unambiguous temporal interoperability.
• GraphQL: The admin panel's own management API is exposed as a GraphQL interface, and the connector framework natively supports GraphQL endpoints as a source connector type alongside REST.

Getting Started#

1. Navigate to Admin > Data Sources to browse the source catalog.
2. Select a data source and click Configure to enter credentials and sync settings.
3. Click Test Connection to verify connectivity and data access.
4. Configure sync frequency, data filters, and retention settings.
5. Click Enable Source to begin data ingestion.

Best Practices#

• Use incremental sync wherever possible to reduce bandwidth and processing load.
• Rotate API keys regularly and store them in the built-in credential management system rather than in configuration files.
• Set up proactive alerts for sync failures and data quality degradation so issues are caught before downstream analysis is affected.
• Check rate limit consumption against provider limits and adjust sync frequency accordingly to avoid throttling.
• Define deduplication rules early to maintain a clean, reliable data foundation.

Availability#

• Enterprise Plan: Included (all source types, unlimited connections)
• Professional Plan: Core integrations included; premium threat intelligence and custom API integrations available as add-on

Last Reviewed: 2026-02-05 Last Updated: 2026-04-14