Home / DPA Insights / Data Management
The Month-End That Wasn’t There
What Green Doesn’t Tell You · A Three-Part Series on Building Trustworthy Financial Data
A file only tells you what arrived. It says nothing about what the platform should hold for a month two years back. The right set of records depends on business dates: when the security started, where the cutoff was, and which month-ends fell in between. Before calling a load complete, build the full list of security and period combinations that ought to exist. Then compare it with what the platform holds. A trustworthy platform can answer two questions. What do we have? And what should we have?
On the same security master build, the historical rebuild looked clean. Every file had arrived. Every file had loaded. Then someone pulled a month-end view for an older period, and a group of securities simply wasn’t there.
Nothing had failed. The source had never sent those records.
A file only tells you what arrived
We had quietly assumed that complete meant loading everything we received. It doesn’t. A daily file shows what the source sent today. It says nothing about what the platform should hold for a month two years back.
For this platform, the right set of records depended on business dates.
- When did the security start?
- Where was the cutoff?
- Which month-ends fell in between?
None of that sits inside a delivery file.
Write down what should exist first
So we flipped the order. Before calling a load complete, we built the full list of security and period combinations that ought to exist. Then we compared it with what the platform actually held.
Missing periods stopped being a surprise someone found in a report. They became something we could see and track. The first comparison turned up gaps in periods we had assumed were long settled.
Historical completeness is a modelling problem, not a loading problem. A trustworthy platform can answer two questions. What do we have? And what should we have? Most can only answer the first.
Next: what happens when you run the same job twice.
Next — Part 3 of 3Run It Twice
Two ways to define “complete”
| Complete as loading | Complete as modelling | |
|---|---|---|
| What complete means | Loading everything we received | The full list of security and period combinations that ought to exist |
| What it can tell you | What the source sent today | What the platform should hold for a month two years back |
| Missing periods | A surprise someone found in a report | Something we could see and track |
| The question it answers | What do we have? | What should we have? |
Before you call a historical load complete
- Set the business dates for each security: when it started, where the cutoff is, and which month-ends fall in between.
- Build the full list of security and period combinations that ought to exist.
- Compare that list with what the platform actually holds.
- Track the missing periods, so they stop being a surprise someone finds in a report.
Can your platform say what it should hold?
If your team runs a security master or another financial data platform, we will walk through how your platform defines complete, and where missing periods could be hiding.
Book a working sessionFrequently asked questions
Why can a data load look complete but still miss records?
A file only tells you what arrived. On this build, every file had arrived and every file had loaded. Then a month-end view for an older period showed that a group of securities simply wasn’t there. Nothing had failed. The source had never sent those records.
How do you check historical data completeness?
Write down what should exist first. Before calling a load complete, build the full list of security and period combinations that ought to exist, based on business dates: when the security started, where the cutoff was, and which month-ends fell in between. Then compare it with what the platform actually holds.
Is historical data completeness a loading problem or a modelling problem?
A modelling problem. The right set of records depends on business dates, and none of that sits inside a delivery file. A trustworthy platform can answer two questions. What do we have? And what should we have? Most can only answer the first.
