Human-in-the-Loop Approvals

Overview#

Automated analytical pipelines can detect meaningful signals at a scale and speed no human team can match, but that same speed becomes a liability when the pipeline acts on a false positive without any opportunity for analyst intervention. A drift detector that incorrectly identifies a routine data distribution shift as an anomaly and immediately triggers an ingestion pause, or a changepoint model that flags a normal operational pattern as a significant behavioural change and auto-escalates an entity, wastes analyst time and erodes operator trust in the platform.

The Human-in-the-Loop Approvals module gives automated services a structured, audited channel to request analyst sign-off before acting on uncertain or high-impact detections. When a service's confidence is below an operator-configured threshold, or when policy requires mandatory human review regardless of confidence, the service creates a review request rather than acting immediately. Analysts review requests through a prioritised queue, approving or rejecting each with optional notes. Approved reviews allow the upstream process to resume; rejected reviews suppress the action and record the analyst's reasoning for audit purposes. Requests that exceed a configurable confidence threshold are auto-approved without entering the human queue, keeping the analyst workload focused on genuinely uncertain cases.

flowchart LR
    A[Automated Service] --> B[request_review]
    B --> C{confidence check}
    C -->|confidence >= threshold| D[AUTO_APPROVED]
    C -->|below threshold or no threshold| E[PENDING]
    E --> F[Analyst Review Queue]
    F --> G{Human decision}
    G -->|approves| H[APPROVED]
    G -->|rejects| I[REJECTED]
    H --> J[Process resumes]
    I --> K[Action suppressed]
    D --> J

Last Reviewed: 2026-04-14 Last Updated: 2026-04-14

Key Features#

• Prioritised Review Queue: Review requests are ordered by severity (CRITICAL, HIGH, MEDIUM, LOW) and then by creation time within each severity band. Analysts always see the most urgent cases at the top of the queue, reducing the risk of high-severity alerts being buried under routine medium-severity ones.

• Automatic Approval by Confidence Threshold: Each service can supply an auto_approve_threshold (0.0 to 1.0) when creating a review request. When the payload's confidence score meets or exceeds that threshold, the record is immediately marked AUTO_APPROVED without entering the human queue. Requests below the threshold always require analyst review.

• Process Pause and Resume: A service can associate a review request with a paused_process_id, indicating that an upstream scheduled job or ingestion pipeline should be suspended until the review is resolved. Analysts can resume the linked process directly from the review queue once they have approved the request.

• Audit Trail: Every state transition (creation, approval, rejection, auto-approval, pause, resume) is recorded with timestamps, the reviewing analyst's user ID, and their notes. This satisfies EDF/PESCO audit logging requirements and provides a traceable record for post-incident analysis.

• Organisation Isolation: All review requests are scoped to the creating organisation's organization_id. No analyst can view or act on review requests from another tenant. Organisation isolation is enforced at the database query level on every read and write operation.

• Severity Levels: Services classify their review requests as LOW, MEDIUM, HIGH, or CRITICAL based on the potential impact of acting or not acting on the detected signal. The severity badge is displayed prominently in the queue and used for priority ordering.

Use Cases#

• Drift Alert Review: An analyst reviews a data drift detection alert raised by the Evidently integration before the platform pauses ingestion from that data source. The analyst examines the drift score, checks recent source activity, and either approves the pause or rejects the alert as a known distribution shift from a scheduled data batch.
• Behavioural Changepoint Sign-off: A changepoint detection service identifies a significant shift in an entity's behaviour pattern and creates a HIGH-severity review request before auto-flagging the entity for investigation. An analyst reviews the changepoint details and confidence score, and either approves the escalation or rejects it as a false positive caused by a known change in the entity's operating context.
• Causal Discovery Edge Uncertainty: A causal discovery pipeline identifies a new uncertain causal edge between two entities and requests human validation before adding it to the causal graph. An analyst reviews the supporting evidence in the payload and either approves the edge or rejects it.
• Audit and Accountability: Security or compliance officers review the approval history for a given service or time period, verifying that all high-impact automated decisions had human sign-off and that rejection reasons are recorded.

Integration#

• Drift Detection: The Evidently drift monitoring integration calls request_review with DRIFT_DETECTED events when the drift score exceeds a configurable sensitivity threshold and confidence does not meet the auto-approve level.
• Changepoint Detection: Changepoint detector pipelines create SIGNIFICANT_CHANGEPOINT review requests for high-amplitude or low-confidence detections before triggering entity escalation.
• Causal Discovery: Causal graph construction services request review for UNCERTAIN_CAUSAL_EDGE events where the edge confidence is within the uncertainty band.
• Scheduler: Paused processes are tracked via the StateClient cache key sched:paused:{process_id}, allowing the scheduler to check pause state with a single cache lookup before each job execution.
• Audit Trail: All HITL decisions are written to the platform audit trail with userId, organizationId, action, timestamp, resourceId, and secrecyLevel fields.