Generating a pentest report
Siemba · 2024
The point of a pentest is the report at the end of it. Somebody clicks a button and a couple of minutes later a PDF arrives that a customer hands to an auditor. It is the least glamorous part of the product and the part they judge it by.
The click, and the wait
A document takes long enough to build that nobody holds a request open for it, so the click returns immediately and something has to come back later to say it is ready. A callback is only worth anything if you know who sent it, so the proof is decided in advance: a row is written for the document that does not exist yet, carrying a secret generated on the spot. That secret goes out with the request and nowhere else, and the message coming back has to present it.
On the screen this is deliberately unremarkable. The button stops spinning almost at once, the list of reports refreshes itself every few seconds while anything on it is still building, and it stops refreshing when nothing is. A row grows a download link, and an email arrives with the document attached. No progress bar, because a progress bar would be a lie: the work is happening in a service that has no way to say how far through it is.
A couple of the simpler exports skip all of this and hand the file back in the response, because they are quick enough that the ceremony would cost more than the wait. That is the sort of inconsistency worth keeping. Making them go the long way round would have been tidier and slower for everybody.
The engine never sees the database
The backend does everything the engine should not have to think about: who is asking, whether they are entitled to this data, and which of it belongs in this document. Then it assembles the lot into one payload and hands it over.
The engine gets that and nothing else. No connection to the product's database, no notion of a company or a permission, no way to ask a follow-up. It takes facts and returns a document. That boundary is what lets it be a separate service at all, and it means the question about a report is never whether the engine got the data right, only whether the backend gathered it right.
The version of this that goes wrong is the one where the document builder is given a database connection so it can fetch the couple of extra things it turns out to need. It is a smaller change than assembling a bigger payload, and from that point on every permission question in the product has two places it can be answered and one of them has no idea who is asking.
Every report is a different document
There is no single template with the title swapped, because the documents are genuinely different. An executive summary is for someone who will read one page. A final report is the thing that gets filed. A certificate and an engagement letter are formal papers with fixed language.
So each type has its own rendering code, composed from page builders the types share: a cover, a findings table, a section of finding detail. The shared pieces are the paragraphs and the tables. What is not shared is the decision about which of them appear and in what order, and that is exactly the part that differs between a one-page summary and a fifty-page filing.
This is more code than one clever template with a lot of conditionals in it, and it is the only version where changing the executive summary cannot quietly change the document a customer has already sent to their auditor.
A chart is a photograph
The library that builds the pages works like a drawing program: place text at a coordinate, put an image in a box. It has no idea what a bar chart is. Draw the charts in it and you now maintain two charting implementations that have to agree forever, the real one on screen and the shadow one in the PDF. They drift within a month.
So the charts are not drawn. A small web page renders a chart from whatever data is put in its address, a headless browser loads it and photographs one element, and the image goes onto the page. The chart in the PDF is the chart from the product, pixel for pixel, because it is the same code. The cost is running a browser inside a document pipeline, which is a strange sentence and a cheaper one than maintaining the shadow.
The page being photographed is a real, buildable web application that no human has ever opened. It has no navigation and no interface. Anyone finding it cold would reasonably assume it was an internal tool somebody abandoned, and it is worth writing down somewhere that it is not, because the day somebody deletes it the reports lose their charts and nothing else breaks.
Looking back
Most of this is about drawing lines in the right places. The engine cannot see the database, so it cannot be wrong about permissions. The callback cannot be trusted, so it proves itself with something minted before it existed. The charts are not reimplemented, so they cannot drift. None of those are clever. Each is somebody deciding what a piece of the system is not allowed to know, and getting a category of bug back in exchange.