Grammatically Fine, Financially Wrong

Fifteen thousand seven hundred and twenty-one Korean mutual fund share classes. That was the target when the project started, and for the first few weeks, it looked achievable in a straightforward way: find the prospectuses, extract the data, translate it into English. Then the team ran their first validation pass, and the accuracy sat at 70 to 80 percent.

The real problem was not extraction

For most automation projects, that number would be a milestone. For financial reference data, it was a wall. A fund name, a fee, a share class identifier is only useful if it is correct every time, not most of the time. The team’s real problem was not extraction. It was translation that read correctly but wasn’t. Korean entity names, for example, would come back as literal English phrases instead of the legal or commercial name a fund actually uses. Grammatically fine, financially wrong.

That distinction changed the question the team was asking. Instead of “can the system pull this information out of the document,” they started asking whether it could pull the correct information, consistently, in a form usable as structured financial data. Those are different engineering problems, and the second one does not get solved by writing one better prompt.

Building a loop: extract, validate, investigate, refine

It got solved by building a loop: extract, validate against the source, investigate why a field failed, refine the prompt or the logic, extract again. Every data point was still checked by a human against the original document, not as a stopgap while the automation matured, but as a permanent part of the system, because it was the only way to tell whether a wrong answer was an extraction problem, a translation problem, or a definitions problem specific to Korean fund structures. The reviewer was not checking whether the machine had read the document. They were checking whether it had understood it. Those are not the same question.

A repeatable method

Validated accuracy climbed from that first 70 to 80 percent to 99 percent. And the human check, the part you might expect to be the bottleneck, got faster as the loop matured, with review time per fund falling from 12 minutes to 3. What the team really built was a repeatable method, drawn from source discovery, field-specific prompting, iterative validation, and human review, and it has since been reused on two more countries.

The definitions layer, not the language layer

The lesson generalizes past Korea. Multilingual financial data work fails less often at the language layer and more often at the definitions layer: what a field means, in that market’s convention, in that document’s format. Source discovery and human validation are not the parts of the workflow you automate right away once the model gets good enough. They are the parts that tell you whether the model is actually right.