Vulnerability enrichment with LLMs

The obvious design is a router: look at the finding, work out it is a SQL injection, pick the prompt that suits it. That is what almost everyone assumes it does, and what I would have built if I had stopped thinking a week earlier.

It does the opposite. There is a fixed catalogue of about eighteen things worth working out about any finding, and the question is not which one fits but which are worth running now. Two filters, both generic: is this finding important enough to bother, and has this particular answer already been worked out. Everything surviving both runs at once. Resubmitting is nearly free, and one answer failing takes nothing else with it.

The catalogue is not eighteen variations on a theme. It is four groups doing different work: the narrative an auditor reads, a mapping into each compliance framework a customer might be audited against, the risk scoring, and the threat classification. A finding needs all four. None of them is a substitute for another, so there was never a choice to make in the first place.

Once you see it that way the router looks like a mistake you got lucky to avoid. A router has to be right about a category before it can be right about anything else, and it silently produces one wrong answer when it is wrong. A catalogue with two coarse filters cannot be wrong about the category, because it never forms an opinion about the category.

Eighteen calls per finding, across a lot of findings, and the bill stops being a rounding error. The version I started with sent everything to whichever model was configured, which meant the most expensive one, because that is the one you trust with the hardest question in the set.

The questions are not equally hard. Scoring against a fixed rubric is narrow and a small model does it correctly every time. Picking a technique out of a large threat taxonomy is open-ended judgment. So each kind of answer got its own assignment, the cheapest model that holds up for that job, with the per-token cost tracked next to it so the choice is made against a number rather than a hunch.

The part I would defend hardest is the tracking rather than the assignments. Assignments go stale: models get cheaper, get better, get retired. A number sitting next to each one means the next person can redo the decision in an afternoon instead of rediscovering that it was ever made.

Most of the catalogue is one call in, one structured answer out. Two are not, because the answer space is too big to ask about in one go and asking anyway gets a confident wrong answer. The threat mapping asks twice, picking the broad category first and the specific technique inside it second. The weakness classification does not start with a model at all: a precomputed index produces a shortlist, and only then does a model choose between them.

Both are the same trick. Do not ask a model to pick one item out of hundreds. Narrow the field with something that cannot hallucinate, then ask it to choose between a handful. The second call is easier to get right and easier to check, and when it is wrong it is wrong within a range you can see.

Running eighteen things at once means eighteen things can fail, and the useful question is what a caller sees when three of them do. Each answer retries a couple of times on its own and is then marked failed, and the others are unaffected either way. A finding comes back with fifteen answers and three marked failed, rather than not coming back.

The call back to whoever asked is a single event rather than a stream. It fires once, when the batch as a whole has finished one way or another. That is a real trade and I would make it the same way again: a stream means every caller has to hold state and decide for itself when the finding is done, and every one of them would decide differently. It does mean that if you are watching an individual answer complete and wondering why nothing has arrived, the answer is that something else in the batch is still running.

Underneath all of it, the same finding submitted twice lands on the same record. That is what makes the second filter possible, and it is why a retry from a caller costs almost nothing: everything already worked out is already there, and only the gaps run.

The decision I am happiest with was not a prompt. It was noticing this looked like a routing problem and was not one, and that a catalogue with two cheap filters came out simpler, cheaper and far more forgiving of a single failure than any router I could have written.