Security teams lose most of their time to mechanical work: reading raw log formats, pivoting between sources to check whether the same address appears twice, and writing up what happened. This pipeline automates that. It ingests logs from multiple sources, normalises them into a single event schema, detects incidents with deterministic rules, and uses a language model only to explain what was found. The constraint the whole system is built around is that rules detect and the AI explains. The model never decides what counts as an alert, and it cannot influence severity.
The project covers parsing, correlation, ATT&CK mapping and enrichment behind a React interface. Parsers are selected by confidence score rather than an ordered chain, so adding one cannot silently capture files belonging to another. Every event normalises to a shared schema with namespaced entity keys, which is what allows a failed SSH login and a failed cloud console login from the same address to land in a single incident instead of two unrelated ones. Incidents are formed as connected components over shared evidence keys within a time gap, severity is derived from the breadth of ATT&CK tactics rather than the number of findings, and every technique ID is validated against a committed copy of the ATT&CK v19.1 catalogue before it reaches an analyst.
The part of this project that took the most thought was the gap between a detection and an incident. A rule produces a finding, which says only that a pattern matched some evidence. What an analyst needs is the story: eight failed SSH logins, a successful one from the same address, cloud reconnaissance, a new access key, an administrator policy attached, and audit logging switched off are not six alerts. They are one intrusion, and reading them as six is exactly the alert fatigue the tool is supposed to remove. Grouping is done as connected components over the entity keys each piece of evidence touches, so the fan-out produced by a single burst and the chaining of activity across sources are handled by one mechanism rather than two that disagree at the edges.
Suppressing noise in that graph turned out to be the place where the obvious approach is backwards. My first version treated a frequently occurring entity key as ambient and excluded it from joining, which sounds reasonable and merged nothing at all. In any batch dominated by a single intrusion, the attacker's address is by definition the most frequent key, so a frequency threshold suppresses precisely the key that should be doing the work. The measure that works is co-occurrence breadth: an account name seen alongside forty hosts is ambient and joining on it would merge unrelated intrusions, while an address seen alongside one host and one account identifies something no matter how often it appears. Severity follows the same logic of preferring breadth to volume, climbing with the number of distinct ATT&CK tactics rather than the count of findings, so that twenty brute-force detections stay one story while a genuine progression through credential access, discovery, persistence and defence impairment escalates.
Most of what I learned came from defects that a passing test suite did not catch. The clearest was a false positive I found while building the sample data. Any successful AWS Security Token Service call was being categorised as authentication, which meant a routine GetCallerIdentity call, something continuous integration and the AWS CLI make constantly, could satisfy the rule for a successful login following a brute-force burst and manufacture a critical incident out of ordinary automation. The interesting part was the fix. Removing the service-level fallback closed the false positive but would have quietly dropped SAML and web identity federation out of authentication entirely, creating a detection gap inside a fix for a false alarm. Both are now mapped explicitly, and the invariant that made the class of bug possible in the first place is pinned by a test, so adding a similar mapping fails rather than ships.
Two others are worth recording. A technique I had mapped by hand turned out to have been retired in ATT&CK v19, along with its entire family, and was silently contributing no tactic and therefore under-escalating incidents. It resolved only because the catalogue check is run against the real published data rather than against my memory of it. Separately, timestamps were not normalised to UTC before being written, and because SQLite discards timezone offsets while PostgreSQL preserves them, an event recorded in a non-UTC offset arrived at the detection layer hours displaced, on one database engine and not the other. Both cases taught me the same thing: a green test suite proves considerably less than it appears to, and the useful question is what the tests are structurally unable to see.
The safety design follows from the same scepticism. The enrichment layer never reads a self-reported confidence score, because a model's stated certainty is highest exactly when it is confidently wrong, which is the case the check exists to catch. Instead every claim is verified against something external: the response must parse, its technique identifiers must resolve in the catalogue, every event it cites must exist in the incident's evidence, and it must stay inside field and length limits. Any failure discards the entire response rather than the offending part, since partial acceptance ships prose whose support has been removed. Technique selection is offered as a shortlist drawn from the incident's event categories rather than asked as an open question, because validation can reject an identifier that does not exist but cannot reject one that is real and wrong. Log content reaching the model is wrapped as untrusted data, and a deterministic summary is generated for every incident regardless, so an outage degrades the output rather than breaking the pipeline.
The system currently reads Linux SSH logs over syslog and AWS CloudTrail, with a Windows Security event parser designed but not built, and I have kept that distinction visible in the documentation rather than describing planned work as finished. I deliberately publish no false-positive rate, because the only corpus available is sample data I wrote myself, and measuring a detector against fixtures authored to trigger it measures the fixtures. What is genuinely verifiable is covered by 302 tests running in continuous integration, including golden-value pins that force a change of definition to be deliberate, subprocess probes for failures that only appear at import time, and invariant guards on the seams between layers.
