Closed-loop detection engineering: from alert to drafted detection to validated rollout

Abstract. Most platforms marketed as AI-SOC are triage accelerators. They take an incoming alert, summarise it against a few enrichments, and propose a verdict. Whether the alert was true or false, whether the underlying pattern generalises, whether the next variant will be caught — these questions are left to a detection-engineering team that operates on a different cadence, with a different toolchain, and against a different copy of the data. The result is a one-way loop: alerts in, dispositions out, content static. This paper specifies the closed loop — eight stages, each with a measurable output, each gated, each reasoned against the same property graph — and reports the operational KPIs we hold ourselves to: MTTD under sixty seconds, MTTR under fifteen minutes, and a sixty-percent reduction in human time at L1.

1. The gap AI-SOC overlays leave open

An overlay product, by construction, sits above the SIEM and consumes its alerts. It does not own ingest, does not own the schema, does not own the response surface, and — critically — does not own the detection content. It can write a good summary, produce a plausible verdict, and route the ticket to the right queue. What it cannot do is the work that converts that ticket into a better detection tomorrow: re-deriving the underlying graph, generalising the pattern, validating it against adversary behaviour, replaying it against history, and rolling it out with measurable confidence.

This gap matters because the half-life of a detection is short. Adversary techniques move; environments drift; the rule that fired correctly on Monday is silent by Friday. A SOC that does not have a closed loop is, in effect, decaying — slowly, then quickly — toward a baseline of generic content that catches yesterday's threats. The closed loop is the only mechanism that keeps the detection surface aligned with the actual adversary surface.

2. The eight stages, end to end

The loop has eight stages. Each stage takes a typed input and produces a typed output. Each stage has a gate — a condition that must be satisfied before the next stage runs. Each gate is enforced by the platform, not by convention.

#	Stage	Input	Output	Gate
1	Triage	Raw alert + graph context	Disposition (true / false / suppress) + sub-graph	Confidence ≥ threshold or escalate
2	Investigate	Sub-graph + analyst questions	Expanded sub-graph, scoped to incident	All graph nodes resolved to canonical entities
3	Root-cause	Expanded sub-graph	Identified initial access edge + propagation path	Path verified against telemetry, not inferred alone
4	Draft detection	Root-cause path + analyst notes	Candidate detection (graph pattern, not regex)	Pattern compiles and binds to declared schema
5	Adversary-validate	Candidate detection	Pass/fail against safe-adversary library	≥ 1 positive trigger on intended technique; zero on negatives
6	Retro-replay	Candidate detection	Hit-count over N days of history, with sample paths	FP rate below tenant threshold; coverage gain documented
7	Rollout	Validated detection	Active rule, with tenancy and severity bound	Two-person review for high-severity; auto-rollout below
8	Feedback	Live alerts from rolled-out rule	Refinement deltas or retirement	Reviewed at fixed cadence; aged content auto-flagged

2.1 Stage 1 — Triage

Triage runs on every inbound alert. Its job is to produce a disposition and a scoped sub-graph, in seconds, with the confidence figure attached. The disposition is not "we think this is malicious" in free text; it is a structured judgement bound to specific graph edges that justify it. If the sub-graph cannot be assembled with sufficient confidence — for instance because one of the entities cannot be resolved canonically — the alert escalates rather than being silently dispositioned.

// Triage output (conceptual)
{
  "alert_id": "evt-1f3c…",
  "disposition": "true_positive",
  "confidence": 0.91,
  "subgraph_root": "Principal:tenant/abc/p-1029",
  "justifying_edges": [
    "Principal→OBSERVED_FROM→Workload (anomaly: new geo)",
    "Workload→READS→SecretStore (anomaly: never before)",
    "Workload→MODIFIED_BY→CodeArtifact (delta < 24h)"
  ],
  "retrieval_calls": ["q-71a…","q-71b…","q-71c…"]
}

2.2 Stage 2 — Investigate

Investigation expands the sub-graph along directions the analyst would otherwise have to walk by hand. The expansion is bounded by edge type and time window, never by a "show me everything" sweep — the latter is how investigations become unreadable. A useful heuristic: every expansion must be justifiable in a sentence, and the platform stores that sentence alongside the resulting nodes.

2.3 Stage 3 — Root-cause

Root-cause identifies the first edge in the propagation path that an adversary controlled, and traces forward to the impact. Crucially, this is verified against telemetry rather than inferred from the model alone. If the model proposes an initial-access edge that does not appear in the underlying event log, the gate fails and the analyst is asked to confirm or refute. This is the single most important guard against fabricated narratives.

2.4 Stage 4 — Draft detection

With a verified root-cause path in hand, the platform proposes a candidate detection. The candidate is a graph pattern — a small typed sub-graph with bound and free variables — not a regular expression and not a search string. The candidate is annotated with the analyst's notes, the original incident reference, and the technique class it is intended to catch.

// Candidate detection (conceptual)
pattern Suspicious_Secret_Access_Post_Code_Change {
  (p:Principal)-[obs:OBSERVED_FROM {age: "<= 24h", anomaly: "new_geo"}]->(w:Workload)
  (w)-[mod:MODIFIED_BY {age: "<= 24h"}]->(c:CodeArtifact)
  (w)-[read:READS {first_observed: true}]->(s:SecretStore)
  WHERE s.criticality >= "high"
  EMIT severity="high", suggested_action="isolate_workload, suspend_session"
}

2.5 Stage 5 — Adversary-validate

The candidate is exercised against a safe-adversary library: a curated set of techniques replayed in a sandboxed tenancy, with both positive cases (the technique we mean to catch) and negative cases (benign activity that superficially resembles it). The gate requires at least one positive trigger and zero negative triggers. A candidate that triggers on negatives is sent back to drafting with the offending sample attached.

2.6 Stage 6 — Retro-replay

The candidate is replayed across a configurable window of history — typically thirty to ninety days. The output is a hit-count, a distribution by tenancy and severity, and a representative sample of paths the rule would have flagged. Two numbers matter here: false-positive rate (against incidents that were investigated and closed as benign) and coverage gain (incidents that were closed as malicious but did not fire any existing rule). The gate is a tenant-specific FP threshold; coverage gain is reported but not gated.

2.7 Stage 7 — Rollout

Rollout activates the rule with a tenancy and severity bound. For high-severity content, a two-person review is required: the drafting analyst and an independent reviewer must both approve. For low and medium severity, rollout is automatic once the previous gates have passed. Every rollout records the validating evidence in an immutable log, so the rule's pedigree is queryable later.

2.8 Stage 8 — Feedback

Once live, the rule's behaviour is observed continuously. Three signals matter: FP rate drift (rising over time, often the sign of a benign change in the environment), coverage decay (the rule fires less often, often the sign that adversary technique has shifted), and analyst override frequency (the rule fires but is consistently dispositioned as benign). Each signal triggers a refinement loop; aged content with no triggers and no relevance is auto-flagged for retirement.

3. The compression metric

The point of the loop is to convert closed incidents into durable detections, with as little human time per detection as possible. We measure this as the detection-candidate yield: the ratio of closed incidents to net-new validated detections per unit time. A SOC operating without a closed loop typically yields one new detection per fifty to one hundred closed incidents, and the detection lags the incident by weeks. A SOC operating the loop yields one per ten to fifteen, with a median lag of under twenty-four hours.

Metric	Open-loop baseline	Closed loop, mature	Improvement
Closed-cases per new detection	50–100	10–15	5–7×
Lag from incident to live rule	2–6 weeks	< 24 h (median)	~ 20×
Analyst hours per drafted rule	6–12	0.5–1.5	~ 8×
Rules retired per quarter (hygiene)	Near zero	5–12% of catalogue	—
Coverage gain on retro-replay (median)	n/a	3–7 prior incidents per new rule	—

4. Why every stage has to reason against the same graph

The temptation, when wiring a system like this, is to let each stage use the data shape that suits it: triage on a flat view, investigation on a graph, detection drafting on a search index, replay on a columnar lake. We have built this and we have watched it fail. The failure mode is reliable and ugly: the candidate detection refers to an attribute that exists in one view and not another, validation succeeds in the index but fails in the lake, and the rule rolls out half-broken.

Keeping all eight stages on the same graph eliminates this class of failure. The pattern that triages the alert is the pattern that drafts the rule is the pattern that replays against history. Schema drift is detected at compile time, not at three in the morning.

5. The role of the agent layer

The agent layer participates in every stage, but its powers are deliberately bounded. It can call retrieval primitives; it can propose a disposition, an expansion, a candidate detection, or a refinement; it cannot mutate state outside the graph except via the response engine, and it cannot retrieve from any store outside the graph. This bound is what makes the loop auditable. Anything the agent did is a sequence of logged queries; anything it proposed is a typed object the analyst can inspect.

// Agent invariant (enforced at the retrieval boundary)
- inputs : RetrievalHandle[]   // never raw queries against arbitrary stores
- outputs: TypedProposal       // Disposition | Expansion | DetectionCandidate | Refinement
- side-effects: none directly  // mutations only via response engine, with two-person gate for high severity
- audit  : (handle_log, proposal_id, reviewer_id) → immutable

6. Operational KPIs we hold ourselves to

The loop is not a research artefact; it is the unit of work that produces the SOC's measurable outcomes. We hold ourselves to three numbers, in this order.

6.1 MTTD ≤ 60 seconds

Mean time to detect, measured from the moment the triggering event lands in the projector to the moment the alert is enriched with its scoped sub-graph and dispositioned at L1. The 60-second target is achievable because triage is, by construction, a few graph traversals plus a confidence calculation; there is no human in the path. The number that matters is not the average — it is the 95th percentile, which we hold under three minutes.

6.2 MTTR ≤ 15 minutes

Mean time to respond, measured from disposition to containment action. The 15-minute target assumes the action is in the platform's catalogue and is within the tenant's pre-approved response surface. Containment actions outside the catalogue — those requiring human authorisation, regulatory notification, or coordinated multi-team work — have their own clocks and are reported separately. The discipline is to make as many actions catalogue-eligible as possible without lowering the gate on reversibility.

6.3 L1 human-time reduction ≥ 60 percent

Measured per analyst-week. The reduction comes from three places, roughly equally: dispositioning is automated for the high-confidence band, investigation expansions are pre-computed before the analyst opens the ticket, and detection drafting takes minutes rather than hours. We do not promise the analyst's job becomes shorter; we promise it becomes harder in the right direction — more cases at L2 and L3, fewer at L1.

7. Failure modes and how we instrument for them

7.1 The plausible-but-wrong narrative

A model presented with a partial graph can construct a story that fits the visible edges and is wrong about the invisible ones. We guard against this with the retrieval-only rule: the model can only refer to edges that exist in the graph, and the platform refuses to render edges the model invented. The user-visible artefact is that the model says "I cannot determine X" more often than analysts initially expect; we treat that as a feature.

7.2 The over-fitted detection

A candidate that was drafted against one incident will, if not pushed, encode incident-specific artefacts: a particular IP, a particular workload name, a particular hour of day. Adversary-validation flushes the worst of these by exercising the technique against a library of variants; retro-replay catches the rest by surfacing low hit counts and analyst override frequency.

7.3 The silent rollout

A detection that rolls out cleanly but never fires is, in practice, indistinguishable from one that does not exist. We instrument every rule with an expected-hit-rate band derived from retro-replay, and we flag rules that drop out of band — high or low — within the first thirty days of rollout. This is a hygiene signal, not a quality signal, but it catches the long tail of zombie content.

8. What this looks like to the analyst

The day-to-day experience is, in a phrase, fewer tabs. The analyst opens a single incident view; the sub-graph is already assembled; the candidate disposition is already proposed; the investigation expansions are one click away with their justifications attached. When the case closes, a detection draft is waiting; the analyst either approves it, edits it, or marks it not-generalisable, and the loop continues. The work is still demanding — the cases that reach a human are, by selection, the harder ones — but the work is the right shape.

8.1 The shift in analyst skill mix

A consequence worth naming explicitly: the skill mix the SOC hires for changes. Pattern-matching attention — the ability to scan twenty alerts an hour for the one that matters — becomes less valuable, because the platform performs that scan. Detection-engineering judgement — the ability to look at a closed case and ask whether the underlying pattern generalises, whether the candidate rule is too specific, whether the technique class is already covered — becomes more valuable, because every closed case is now a drafting opportunity. SOCs that have run the loop for two quarters report that their L1 attrition falls, because the work becomes more interesting; their L2 hiring becomes harder, because the role is now closer to engineering than to monitoring.

8.2 The shift in detection-content ownership

Content authorship was historically the preserve of a small detection-engineering team, with submissions from the analyst floor handled as a separate intake. The closed loop dissolves this separation: every analyst is a content author by default, every closed case is a draft, and the detection-engineering team's role shifts toward stewardship — review, dedup, retirement, technique-class coverage. The catalogue grows faster, and it grows in directions that match the actual incident distribution rather than the team's prior anticipations.

9. Where the loop interacts with the rest of the platform

The closed loop does not run in isolation; it shares state with three other surfaces. Exposure management feeds it the standing reachability paths that act as priors for which detections matter most. Threat-informed-defence content provides the safe-adversary library and the technique catalogue. Response automation receives the validated rules' suggested actions and binds them to the tenant's pre-approved response surface. All three interactions happen against the same graph, with the same retrieval primitives, so the loop's state is queryable from any of them.

The practical consequence is that improvements to one surface compound into the others. A new technique class added to the catalogue surfaces as a safe-adversary variant the next time a candidate is validated; a new reachability path raised by exposure becomes a prior on which detections deserve high-severity treatment; a new response action becomes selectable in the candidate's suggested-action field on the next draft. Open-loop platforms do not get this compounding because their surfaces share dashboards rather than data.

About this paper

Authored by Autocops Desk — the Netgraph practitioner team. First published 26 March 2026. Confidential to recipient.