AIOps Incident-Investigation Agent

What actually happens

Alertmanager pushes; the agent reacts. No polling, no chat window that only works when someone's watching — every run leaves a file behind.

Investigatornode 1

Given an alert, pulls the same evidence an on-call engineer would check first: recent logs from Loki, the restart-count metric from Prometheus, and a pod describe via the read-only Kubernetes API. No interpretation yet — just gathering.

Reporternode 2

Turns that evidence into three things: a likely cause, the specific evidence for it, and one suggested next diagnostic step. Never a fix, never an action — it has no tool that could take one.

Security auditparallel pipeline

A scoped Kubernetes audit policy watches for exactly two signals — RBAC object changes and kubectl exec — shipped through Grafana Alloy into Loki. Everything else is dropped before it becomes noise.

Observabilityinstrumentation

Every agent run is traced with Grafana's agent-observability SDK, so the tool that investigates cluster health shows up in the same dashboards as the cluster it's investigating.

Architecture

Rides entirely on an existing EKS cluster's Prometheus and Alertmanager — no parallel cluster, no second Grafana.

Solid lines: data and alert flow. Dashed line: the agent's own instrumentation, feeding back into the same Grafana.

Read-only, enforced twice

"Don't take actions" isn't a line in a prompt here — a prompt is advisory. It's a property of the code and the cluster permissions, independently.

Layer 1 — the code itself

The tool clients the agent calls simply don't contain a write method. There's no delete_pod or patch_deployment function anywhere for a bug — or a misbehaving prompt — to reach for.

Layer 2 — cluster RBAC

The agent's service account is granted get, list, watch — nothing else. Even if the code layer had a bug, the API server itself would reject any write attempt.

See it in action

Media from a real run against a live cluster goes here.

PLACEHOLDER — screenshot Grafana security-audit dashboard, showing a real kubectl exec event and an RBAC change captured within seconds.

PLACEHOLDER — screenshot A generated incident report: the alert that fired next to the agent's Markdown summary.

PLACEHOLDER — screen recording 30–60s clip: triggering the CrashLoopBackOff pod, the alert firing, the report appearing.

PLACEHOLDER — screenshot Grafana's AI Observability view, showing a real agent run's latency and node breakdown.

Deliberately out of scope

A working two-node agent beats an elaborate multi-agent system nobody, including its builder, can fully explain.

No multi-agent "war room" with specialized roles for every alert type.
No autonomous remediation — investigation only, a human decides and acts.
No parallel cluster — this rides entirely on existing infrastructure.

An incident-investigation agent that only ever looks, never touches.