Home / DPA Insights / Data and AI
Onboarding the Tenth Counterparty Should Not Feel Like the First
Field notes from engineering a configuration-driven FTP onboarding framework.
The scalability problem in counterparty data onboarding is rarely the file transfer. It is the implementation model. When directory paths, schedules, naming rules, authentication, retries and retention are expressed as configuration rather than code, a single execution engine can serve every counterparty, and onboarding becomes a data entry exercise instead of a development project.
How to read this page
We set out to make FTP (File Transfer Protocol) onboarding faster and assumed the work was in the file transfer. We were wrong. The scalability problem was never FTP. It was our implementation model.
That realization took a run of painful onboardings to arrive, and it changed how we build integrations across the board. This is the story of how we got there.
01 / Differences
Every Counterparty Was Slightly Different. That Was Enough to Break Everything
On paper, onboarding a new FTP counterparty is simple. Connect to a remote server, retrieve files, validate them, trigger downstream processing. The first few integrations went exactly that way, and each one shipped on time.
Then the differences started to compound. One counterparty delivered files in deeply nested folders, another dropped everything into a single directory. Naming conventions ranged from timestamps to business dates. Delivery schedules ran from hourly to monthly. Authentication methods varied, and so did retention expectations.
Each difference was small. Together, they meant every onboarding was a fresh build. We were solving the same problem repeatedly, in slightly different code, and every copy had to be maintained separately.
Our working assumption, for a long time, was that this was simply the cost of doing integrations. Counterparties differ, so integrations differ, so each one needs its own logic. The team's job was to build each one well.
The hypothesis we eventually tested was the opposite: that almost none of these differences deserved code at all.
If directory paths, schedules, naming rules, authentication, retries, and retention could be expressed as configuration, then one execution engine could serve every counterparty. Onboarding would become a data entry exercise, not a development project.
02 / Accumulation
It Didn't Fail Loudly. It Failed by Accumulation
The custom-build model did not fail loudly. It failed by accumulation. Two moments made the cost undeniable.
The first was a large file that repeatedly timed out during download. The retry handling that would have caught it existed in some integrations but not in the one that mattered, because each copy of the pipeline had drifted from the others. The fix was trivial. Finding which of the near-identical implementations needed it was not.
The second was scheduling. As hourly, daily, and monthly deliveries stacked up across counterparties, jobs began colliding, and there was no single place to see or manage the calendar. Monitoring was as fragmented as the code.
Neither incident was an FTP problem. Both were symptoms of the same root cause: we had built one solution many times instead of one platform once. That was the surprise. The thing slowing us down was not counterparty complexity. It was our own duplication.
The thing slowing us down was not counterparty complexity. It was our own duplication.
03 / Metadata
One Question Changed the Architecture: How Much of This Is Actually Data?
We built a proof of concept around a single question: how much of an onboarding can be metadata? We moved directory paths, schedules, authentication, naming rules, retry behavior, and retention settings into configuration, and wrote an execution engine that interpreted those settings dynamically.
The result was clearer than we expected. The overwhelming majority of counterparty differences turned out to be data, not logic.
We call the approach that came out of it Configuration-First Onboarding: the default for any new requirement is a configuration entry, and code changes are reserved for genuinely new behavior.
04 / Platform
From Proof of Concept to Platform
The proof of concept became a reusable framework: configuration-driven onboarding with standardized validation, configurable scheduling and retry logic, consistent retention policies, and a single monitoring view across every integration. Because every counterparty now runs through the same execution path, an improvement to the framework reaches all of them at once. Engineering effort has shifted from writing one-off pipelines to strengthening a shared platform.
| What onboarding involves | Custom-build model | Configuration-first framework |
|---|---|---|
| Directory paths and folder depth | Written into each pipeline | Configuration entry |
| Delivery schedules, hourly to monthly | Scheduled inside each pipeline, no single calendar | Configurable scheduling |
| Naming conventions | Parsing logic written per integration | Naming rule as configuration |
| Authentication methods | Implemented per counterparty | Configuration entry |
| Retry and timeout handling | In some integrations, not in others | Configurable retry logic in the engine |
| Retention expectations | Set pipeline by pipeline | Consistent retention policies |
| Monitoring | As fragmented as the code | A single monitoring view across every integration |
| A new requirement | A fresh build | A configuration entry, unless the behavior is genuinely new |
05 / Profile
A Counterparty Profile Is Complete When Nobody Has to Open an Editor
The question "how much of this is data?" has a practical test attached to it. A counterparty profile is complete when a new integration can be described end to end without opening a code editor. That is a higher bar than it sounds, because the fields that get forgotten are never the obvious ones. Connection details and directory paths are in every first version of a configuration schema. Arrival expectations, deduplication keys and ownership almost never are, and those are the three that generate incidents eighteen months later.
| Field group | What it has to express | What happens when it is missing |
|---|---|---|
| Connection | Protocol, host, port, credential reference | Credentials end up inside a pipeline and rotation becomes a code change |
| Discovery | Root path, folder pattern, recursion depth | Nested-folder counterparties get bespoke traversal logic |
| File identification | Filename pattern, business-date extraction rule | The schedule and the file contents disagree and nobody notices |
| Arrival expectation | Delivery window, expected file count, grace period | A missing file stays invisible until a downstream report is wrong |
| Validation | Required schema, checksum, row-count tolerance | Bad files process as successfully as good ones |
| Idempotency | Deduplication key, behavior on redelivery | A resent file is counted twice |
| Failure handling | Retry count, backoff, timeout, escalation path | Retry logic exists in some integrations and not in the one that matters |
| Retention | How long raw files are kept, and where | Retention becomes whatever the last engineer assumed |
| Downstream trigger | What runs on success, what must not run on failure | Partial data reaches consumers |
| Ownership | Who is alerted, who approves a change | Alerts arrive with no owner and are eventually muted |
06 / Silence
The Failure This Catches Is the One Code Usually Misses
Timeouts and collisions are visible failures. Something ran, something went wrong, and there is a log line to find. The harder failure is silence: the file that was never delivered at all.
Custom pipelines are structurally bad at this, because a pipeline triggered by a file's arrival cannot notice an arrival that did not happen. Detection requires an expectation, and an expectation only helps if it lives somewhere the platform can read. This counterparty delivers by 06:00 on every business day, and there should be four files.
That expectation is configuration. Once a delivery window and an expected file count are declared per counterparty, absence becomes a monitorable event like any other, and it can be raised before the downstream report is wrong rather than after. It is the clearest illustration of a wider point: moving a decision out of code and into configuration does not only remove duplication, it makes a class of failure detectable that was previously invisible.
07 / Redeliveries
Corrections and Redeliveries Are Not an Edge Case
External parties resend files. A counterparty corrects a business date, reissues a day of data after a fix on their side, or delivers the same file twice because their own retry fired. In financial data integration this is routine rather than exceptional.
A pipeline that assumes every delivery is new will process a redelivery without complaint, and the damage surfaces downstream as duplicated records rather than as a failed job. Failures that look like success are the expensive kind. The defence is a deduplication key declared per counterparty, typically business date plus file type plus a content hash, together with an explicit statement of what a second delivery means: ignore it, supersede the earlier one, or hold it for review.
This is the field most often absent from a first version of a configuration schema, and the most expensive to add later, because by then there is history to reconcile.
08 / At scale
What Changed Across 100+ Integrations
The framework now runs 100+ counterparty integrations in production. Onboarding a new counterparty has come down from roughly one to two days of development and testing to a single day of configuration and verification. Duplicate pipeline code has been retired, and every integration is visible in one monitoring view, so issues like the timeout incident are caught by the platform rather than discovered downstream.
09 / Limits
Where Configuration-First Stops
A framework that removes one kind of duplication can quietly create another. Three limits are worth stating plainly, because a model is only trustworthy once its edges are known.
The first is configuration sprawl. When configuration becomes the only permitted answer, teams start expressing genuinely different behavior as ever more elaborate settings, and the configuration language drifts toward a programming language with no tests, no type system and no debugger. When a counterparty's requirement needs a conditional, that is the signal it has earned code.
The second is that configuration is production state. It carries the reach of code without, by default, the controls of code. An entry capable of affecting a hundred integrations belongs under version control, moves through environments, and deserves the same review as a deploy. The blast radius of a shared platform cuts both ways: one improvement reaches every integration at once, and so does one mistake.
The third is that a shared execution path makes every counterparty a stakeholder in every change. That is a governance question before it is a technical one, and it is answered with ownership and a change process, not with more configuration.
The share of new onboardings that required a code change. Trending toward zero means the configuration schema is complete. Trending up means either a genuinely new class of counterparty, or a schema missing a field it needs.
10 / Migration
How You Get There From Twenty Custom Pipelines
Almost nobody gets to build this on a clean sheet. The realistic path is not a rewrite but a replacement in place, one counterparty at a time.
Take the simplest existing integration, express it as configuration, and run it in parallel with the pipeline it replaces. Compare outputs file by file until the two agree across a full delivery cycle, then retire the original. The comparison is the point. It converts a migration from an act of faith into an evidence exercise, and it surfaces the undocumented behavior that every long-lived pipeline accumulates.
Order the queue by simplicity rather than by importance. Early migrations are how a configuration schema earns its missing fields, and a gap is far cheaper to discover on a low-volume monthly feed than on the daily one the finance team depends on.
11 / Beyond FTP
Why this matters beyond FTP
Nothing about this lesson is specific to file transfer. Any operation that integrates with many external parties, whether custodian feeds, servicer reports, fund administrator extracts, or counterparty data files, faces the same trap: the first few custom builds feel fast, and the twentieth is where the model quietly collapses.
The question worth asking early is not "how quickly can we build this integration?" but "which parts of this integration are actually data?" In our experience, the answer is: far more than you think.
Scalable engineering does not come from writing code faster. It comes from noticing which code you should stop writing.
Sixty minutes. No commitment.
How many pipelines does your team maintain?
If counterparty onboarding is a fresh build every time, the cost is not in any single integration. It is in the copies. We will spend sixty minutes with your engineering lead separating the parts of your current onboarding that are logic from the parts that are actually data, and hand back the map. No commitment, no commercial discussion.
Book an onboarding architecture reviewFrequently asked questions
What is configuration-driven data onboarding?
Configuration-driven data onboarding expresses directory paths, schedules, naming rules, authentication, retries and retention as configuration rather than code. One execution engine interprets those settings dynamically, so it can serve every counterparty. Onboarding becomes a data entry exercise, not a development project.
Why do custom-built data pipelines break down at scale?
They do not fail loudly. They fail by accumulation. Every onboarding is a fresh build, so each copy of the pipeline drifts from the others. Retry handling exists in some integrations and not in the one that matters, and monitoring fragments as badly as the code. The root cause is building one solution many times instead of one platform once.
Which parts of a data ingestion framework should be configuration rather than code?
Directory paths, schedules, authentication, naming rules, retry behavior and retention settings. Moving these into configuration showed that the overwhelming majority of counterparty differences are data, not logic. Code changes are then reserved for genuinely new behavior.
What changes when counterparty onboarding runs on a shared framework?
Because every counterparty runs through the same execution path, an improvement to the framework reaches all of them at once. Duplicate pipeline code is retired, every integration is visible in one monitoring view, and engineering effort shifts from writing one-off pipelines to strengthening a shared platform.
How do you detect a counterparty file that never arrives?
A pipeline triggered by a file's arrival cannot notice an arrival that did not happen. Detection needs a declared expectation: a delivery window, an expected file count and a grace period, held per counterparty as configuration. Absence then becomes a monitorable event and can be raised before a downstream report is wrong.
When should counterparty onboarding still require code?
When the requirement needs a conditional. Configuration should express variation in paths, schedules, naming, authentication, retries and retention. Once teams start expressing genuinely different behavior as ever more elaborate settings, the configuration language is drifting toward a programming language without tests or a debugger, and the behavior has earned code.
Does this apply beyond FTP file transfer automation?
Nothing about this lesson is specific to file transfer. Any operation that integrates with many external parties faces the same trap, whether custodian feeds, servicer reports, fund administrator extracts or counterparty data files. The first few custom builds feel fast, and the twentieth is where the model quietly collapses.
