Home / DPA Insights / Data Management
The Job Ran Green. The Data Was Wrong.
What Green Doesn’t Tell You · A Three-Part Series on Building Trustworthy Financial Data
Green only means the machine ran. Job status and row counts don’t know what the data means. A duplicate delivery, a date in the wrong format, or a column that changes type between stages can all pass every check while the business picture is still wrong. On a security master build, four checks replaced “did the job run?”: is each security unique for its date, does every expected period exist, do the values behind valuation and risk hold up, and can each curated figure trace back to its source. Validation moved out of the last step before sign-off and into each stage of the pipeline, so problems got caught where they started, not three systems later.
Every job finished. No errors, no alerts, row counts close to the day before. By any dashboard’s measure, the load worked.
Then a validation check found one security holding two records for the same business date. Both looked valid. Only one could be true. The client’s risk team was pricing from whichever one the query returned first.
Green only means the machine ran
We were building a security master platform for an investment firm, and we started the way most teams do. Land the source files, clean them in layers, publish tables for valuation, risk and reporting. We trusted job status and row counts. If both looked healthy, we moved on.
Neither signal knows what the data means. A duplicate delivery adds rows, so the count looks fine. A date in the wrong format still loads. A column that changes type between stages still writes. Every check passes, and the business picture is still wrong.
So we changed the question
We stopped asking whether the job ran. We started asking what the data should look like once it had. That gave us four checks.
- Is each security unique for its date?
- Does every expected period exist?
- Do the values behind valuation and risk hold up?
- Can each curated figure trace back to its source?
The first time we ran those checks, they flagged more than a few stray issues that job status had waved straight through. That mattered. The curated layer feeds valuation, risk and the reports our client’s teams rely on every day. One wrong record there doesn’t stay in one place.
So validation stopped being the last step before sign-off. We built it into each stage of the pipeline, so problems got caught where they started, not three systems later.
A database protects the write. It can’t tell you whether the write is right. That judgement has to be designed in, and it starts with a plain answer to one question: what does a single record stand for?
Next: the month-end records nobody noticed were missing.
Next — Part 2 of 3The Month-End That Wasn’t There
What a green run can hide
| What goes wrong | Why it still looks green |
|---|---|
| One security, two records for the same business date | Both looked valid. Only one could be true. |
| A duplicate delivery | It adds rows, so the count looks fine. |
| A date in the wrong format | It still loads. |
| A column that changes type between stages | It still writes. |
Does your pipeline know what its data means?
If your team runs a security master or another financial data platform, we will walk through the four checks against your pipeline: uniqueness by security and date, expected periods, the values behind valuation and risk, and traceability to source.
Book a working sessionFrequently asked questions
Why doesn’t a green job status mean the data is right?
Green only means the machine ran. Job status and row counts don’t know what the data means. A duplicate delivery adds rows, so the count looks fine. A date in the wrong format still loads. A column that changes type between stages still writes. Every check passes, and the business picture is still wrong.
What data quality checks does a security master need?
On this security master build, four checks replaced the question of whether the job ran. Is each security unique for its date? Does every expected period exist? Do the values behind valuation and risk hold up? Can each curated figure trace back to its source?
How did a duplicate security record get through a successful load?
Job status and row counts do not check what a record stands for. On this build, a validation check found one security holding two records for the same business date. Both looked valid. Only one could be true. The client’s risk team was pricing from whichever one the query returned first.
Where should data validation sit in a financial data pipeline?
In each stage, not only at the end. On this build, validation stopped being the last step before sign-off. The team built it into each stage of the pipeline, so problems got caught where they started, not three systems later. The curated layer feeds valuation, risk and reporting, and one wrong record there doesn’t stay in one place.
