Overview / Data journey · B02
Follow one dataset from arrival to deletion
A shared vocabulary for where data is fragile. Most problems we see sit at the edges: arrival and retirement.
Use this page with your team. Mark the stage where you have the least confidence; that is usually where to start.
Arrive
Sources, formats, frequency
Every dataset has an origin: a supplier upload, a system export, a form, a shared folder. We list each source, its format, how often it arrives, who sends it and what happens if it is late.
Questions we ask: What triggers this file? Who notices if it does not arrive? Has the format ever changed without warning?
Check
Validation before trust
Data is checked against written rules before it is used: required fields present, types correct, values in range, keys unique, text encoding consistent. Failures are reported, not silently fixed.
Output: a validation report per run, showing counts accepted and rejected, with reasons.
Shape
Transformations you can read
Joins, lookups, calculations and reformatting are written as clear, version-controlled steps. Each step can be explained to the person who owns the business rule.
Output: transformation description in plain language alongside the job itself.
Store
A known place, a known lock
The result lives somewhere with documented location, access controls and retention. Backups follow the agreed pattern.
Output: entry in the environment record, updated whenever anything changes.
Deliver
Outputs people actually use
Files, tables or extracts are delivered to people and systems in the format they need, at the time they need them.
Output: delivery schedule and a list of recipients, reviewed periodically.
Retire
Deletion is part of the design
Data that has served its purpose is archived or deleted according to the retention schedule, and that deletion is recorded.
Output: retention actions log.
A monthly supplier file
Consider a typical case, described generically: a supplier emails a spreadsheet each month. Someone saves it, deletes the header rows, fixes the dates, pastes it into a master workbook and sends a summary to finance.
Arrive: files land in a dedicated, access-controlled location rather than an inbox. Check: rules reject rows with missing account codes and flag dates outside the month. Shape: the job reformats dates, matches codes to your reference list and calculates totals. Store: the result is appended to a hosted table with a backup pattern. Deliver: finance receives the same summary as before, plus the rejection report. Retire: raw files older than the agreed period are deleted and logged.
Nothing exotic, but the process no longer depends on one person’s memory.