Home / DPA Insights / Data Management
Run It Twice
What Green Doesn’t Tell You · A Three-Part Series on Building Trustworthy Financial Data
Correct code can still make an unsafe run. Two valid jobs touching one table at once made the whole execution unsafe. A rerun has three right answers. If the data changed, update it. If it’s genuinely new, add it. If nothing changed, leave the table exactly as it was. A rerun that quietly creates a second copy fails that test, even when it reports success. A reliable pipeline gives the same correct answer every time you repeat it, and does no damage when it overlaps with another job.
Late in the same build, a historical refresh was halfway through when the daily load kicked off on schedule. Both jobs reached for the same table. The logic in each was correct. The run still failed.
Correct code, unsafe run
Our first instinct was to fix the write that broke. The real cause sat further out, across scheduling, storage and timing. Two valid jobs touching one table at once made the whole execution unsafe.
That raised a question we had never answered clearly. When you rerun a load, what should actually happen?
There are three right answers
It depends on the record. If the data changed, update it. If it’s genuinely new, add it. If nothing changed, leave the table exactly as it was. A rerun that quietly creates a second copy fails that test, even when it reports success.
So we split routine daily loads from historical refreshes and stopped them colliding. Then we checked the table before and after repeated runs, including records we had already processed.
A reliable pipeline isn’t one that succeeds once. It gives the same correct answer every time you repeat it, and does no damage when it overlaps with another job. If you can’t safely press run twice, you’re not there yet.
What a rerun should do
| When you rerun a load and the record… | The right answer |
|---|---|
| Has changed | Update it |
| Is genuinely new | Add it |
| Has not changed | Leave the table exactly as it was |
| Gets quietly copied a second time | The rerun fails the test, even when it reports success |
Can you safely press run twice?
- Does a repeated run give the same correct answer every time?
- Does it do no damage when it overlaps with another job?
- Are routine daily loads split from historical refreshes, so they cannot collide?
- Have you checked the table before and after repeated runs, including records you had already processed?
- Does a rerun leave unchanged records exactly as they were, with no second copy?
Can your pipeline safely run twice?
If your team runs a security master or another financial data platform, we will walk through what your loads do on a rerun, and where daily loads and historical refreshes could collide.
Book a working sessionFrequently asked questions
What should happen when you rerun a data load?
It depends on the record. If the data changed, update it. If it’s genuinely new, add it. If nothing changed, leave the table exactly as it was. A rerun that quietly creates a second copy fails that test, even when it reports success.
Why can two correct jobs still cause a failed run?
The cause can sit outside the code, across scheduling, storage and timing. On this build, a historical refresh was halfway through when the daily load kicked off, and both jobs reached for the same table. The logic in each was correct. The run still failed, because two valid jobs touching one table at once made the whole execution unsafe.
How do you stop a historical refresh colliding with the daily load?
Split them. On this build, routine daily loads were separated from historical refreshes so they stopped colliding. The team then checked the table before and after repeated runs, including records already processed.
What makes a data load idempotent?
A load is idempotent when you can run it again and get the same correct answer, with no damage. If the data changed, the rerun updates it. If it’s genuinely new, the rerun adds it. If nothing changed, the table stays exactly as it was. A rerun that quietly creates a second copy is not idempotent, even when it reports success.
