Home / DPA Insights / Investment Research
What Happened When We Let AI Read the Earnings Report
Field notes from investment research. What a near miss taught us about AI, verification, and the work only analysts can do.
We gave a language model a forty page quarterly earnings report and asked for one page back. It returned one in eleven seconds, and it looked finished. A manual check against the source found a basis point movement that was wrong. Repeat runs showed a pattern: figures lifted directly from the text were reliable, figures the model had to compute were not. The fix was not a better summarizer. It was a second pass that independently recomputes every derived number and flags what does not reconcile.
How to read this page
01 / The reading
Why is reading the earnings report not the same as doing the research?
Forty pages of quarterly results. Dense tables, pages of management commentary, guidance buried past the halfway mark. Everything a portfolio manager needs sits in that document, wrapped in an hour of reading before anyone can act on it.
And nobody reads one report. Analysts work through dozens of earnings releases, transcripts, notes, and commentary pieces, all measured against the same handful of parameters. If your research team lives this every quarter, you already know: the reading was never the work. It was what stood between the team and the work.
02 / The run
What happens when you hand a language model a forty page earnings report?
We gave the machine forty pages. It came back in eleven seconds. That was the easy part.
The instructions were simple: take the report, return one page. A clean table of earnings and EPS, the key quarter's moves in short form, the year on year and quarter on quarter changes that matter, and an outlook section capturing where management sees the business heading.
The page came back in eleven seconds.
First impression vs. the actual output
On first read, the output looked ready to ship. Clean tables, tight prose, an outlook section sharper than many sell side notes. It looked finished.
A manual check against the source told a different story. Sitting quietly inside the table was a basis point movement that was simply wrong, and it only revealed itself when we went back to the source and re-did the math independently. It had already survived one skim. Left alone, that number was on its way into a client facing deck.
03 / Where it breaks
Where does AI actually get financial numbers wrong?
Repeat runs on other reports showed the same pattern. Figures pulled directly from the text were reliable. Anything the model had to compute, percentage changes, basis point shifts, one number derived from two others, was where the gap opened up.
| What the model was asked to do | Example from the run | What we observed | What it needs |
|---|---|---|---|
| Lift a stated figure out of the document | Reported revenue, EPS, segment totals | Reliable across runs | Spot check |
| Summarise management commentary | The outlook section | Strong, in places sharper than a sell side note | Analyst read for tone and emphasis |
| Derive one number from two others | Year on year and quarter on quarter percentage change | Where the gap opened | Independent recomputation |
| Express a movement in basis points | Margin movement inside the summary table | One value simply wrong, and it survived a skim | Independent recomputation, then flag |
That gap is what makes this category of error dangerous. A near correct number survives a skim, makes it into the deck, and gets questioned by a client weeks later. A wrong number that looks wrong gets caught. A wrong number that looks right is the one that costs you the meeting.
04 / Closing the gap
How do you close the gap?
A sharper instruction Pass 1
The original prompt asked for a summary. It never specified what a financial reader actually needs, how a table should be built for a ninety second scan, or which comparisons to run against which base. Rewriting the prompt around the reader and the calculation method improved the output noticeably with each iteration.
A layer that doubts Pass 2
The real issue was never extraction. It was that a right answer and a near right answer are indistinguishable until you check. So, we built a second prompt that now runs behind the first, with one job: cross check every figure against the source, independently recompute the percentage and basis point math, and flag anything that does not reconcile. It does not summarize. It doubts.
That layer is what turned a party trick into a production step. A human still signs off on everything, but now reviews one verified page instead of reading forty. An hour of reading becomes a five minute review.
05 / What it taught us
What did the gap teach us?
Verification creates trust
The difference between a demonstration and a production process is rarely the quality of the first output. It is the quality of the validation behind it.
AI is powerful, but not magical
The best implementations combine models with workflow design, business rules, validation, and domain expertise. The model is only one component.
Human judgment becomes more valuable
As routine reading and extraction get automated, what is left is interpretation, challenge, and decision making, fundamentally human work.
06 / Beyond research
Why does this matter beyond investment research?
The lesson extends well past earnings analysis. Wherever professionals spend time reviewing documents, extracting information, and performing repetitive validation, the same pattern shows up: research teams, credit analysts, operations groups, compliance functions, and risk teams alike. The documents differ; the underlying problem is the same. The opportunity is not simply to automate reading. It is to redesign workflows around verified information.
07 / Three questions
Three questions worth asking your own team
If you want a quick read on whether this gap already exists in your process, ask:
- How many numbers in your last client deliverable were computed by a model, a macro, or a spreadsheet, and independently verified by no one?
- If a derived figure were slightly wrong, would your current review catch it, or would it survive a skim the way ours did?
- How many analyst hours each quarter go into reading that produces no judgment, only extraction?
If those answers are uncomfortable, the problem is not your analysts and it is not the AI. It is the absence of a verification layer between the two, and it is buildable.
Frequently asked questions
Can AI do financial analysis reliably?
It depends entirely on which part of the analysis you mean. In our own testing, figures a model lifts directly from the text of a report were reliable across repeat runs. Figures the model had to compute, percentage changes, basis point movements, any number derived from two others, were where errors appeared. So the honest answer is that AI reads reliably and calculates unreliably, and a process that does not separate those two things is carrying a risk it cannot see.
Where do AI models most often get financial numbers wrong?
In derived values. A percentage change calculated against the wrong base, a basis point movement expressed incorrectly, a figure that is internally consistent with the rest of the summary but does not match the source. These errors are dangerous precisely because they are near correct. A wrong number that looks wrong gets caught in review. A wrong number that looks right survives a skim, enters a client deliverable, and surfaces weeks later as a question you cannot answer.
What is a verification layer, in practice?
A second, separate pass that runs behind the summarization step and has a different job. It does not rewrite or improve the summary. It cross checks every figure against the source document, independently recomputes the percentage and basis point math, and flags anything that does not reconcile. Because it is a distinct instruction rather than a longer version of the first one, it has no stake in the first output being correct.
Does this mean a human still has to read the whole report?
No, and that is the point. A human still signs off on everything that leaves the team, but reviews one verified page with the unreconciled items already flagged, rather than reading forty pages to find them. In our own research operations that turned roughly an hour of reading into a five minute review. The judgment stays with the analyst. The reading does not.
Is this only relevant to investment research teams?
No. The same pattern appears wherever professionals review documents, extract information, and perform repetitive validation. Credit analysts working through borrower files, operations groups reconciling statements, compliance and risk functions checking disclosures. The documents differ and the domain rules differ, but the failure mode is identical: extraction holds, computation drifts, and nothing in the output tells you which is which.
How do we find out whether this gap exists in our own process?
Send us five documents your team works from. We run our two pass process against them over two weeks at no cost, and return a written memo naming every derived figure that failed to reconcile and what a verification layer would change about your review cycle. There is no obligation to proceed. Request the diagnostic.
