Mon–Fri 09:00–18:00 UK

Overview / Data journey · B02

Follow one dataset from arrival to deletion

A shared vocabulary for where data is fragile. Most problems we see sit at the edges: arrival and retirement.

Use this page with your team. Mark the stage where you have the least confidence; that is usually where to start.

01

Arrive

Sources, formats, frequency

Every dataset has an origin: a supplier upload, a system export, a form, a shared folder. We list each source, its format, how often it arrives, who sends it and what happens if it is late.

Questions we ask: What triggers this file? Who notices if it does not arrive? Has the format ever changed without warning?

02

Check

Validation before trust

Data is checked against written rules before it is used: required fields present, types correct, values in range, keys unique, text encoding consistent. Failures are reported, not silently fixed.

Output: a validation report per run, showing counts accepted and rejected, with reasons.

03

Shape

Transformations you can read

Joins, lookups, calculations and reformatting are written as clear, version-controlled steps. Each step can be explained to the person who owns the business rule.

Output: transformation description in plain language alongside the job itself.

04

Store

A known place, a known lock

The result lives somewhere with documented location, access controls and retention. Backups follow the agreed pattern.

Output: entry in the environment record, updated whenever anything changes.

05

Deliver

Outputs people actually use

Files, tables or extracts are delivered to people and systems in the format they need, at the time they need them.

Output: delivery schedule and a list of recipients, reviewed periodically.

06

Retire

Deletion is part of the design

Data that has served its purpose is archived or deleted according to the retention schedule, and that deletion is recorded.

Output: retention actions log.

Worked example

A monthly supplier file

Consider a typical case, described generically: a supplier emails a spreadsheet each month. Someone saves it, deletes the header rows, fixes the dates, pastes it into a master workbook and sends a summary to finance.

Arrive: files land in a dedicated, access-controlled location rather than an inbox. Check: rules reject rows with missing account codes and flag dates outside the month. Shape: the job reformats dates, matches codes to your reference list and calculates totals. Store: the result is appended to a hosted table with a backup pattern. Deliver: finance receives the same summary as before, plus the rejection report. Retire: raw files older than the agreed period are deleted and logged.

Nothing exotic, but the process no longer depends on one person’s memory.