Ekho-Labs / ton-app · pull request #6

M5 Export, retention, production
the promise becomes code

The last milestone makes the data leavable, destroyable and deployable: a three-sheet workbook built entirely in the browser, a purge that clears R2, D1 and KV together with a swept, time-boxed debug exception, and a production environment that cannot be deployed to by accident.

branchmilestone/m5-export-retention
basemilestone/m4-forecast-screen
head7df79c5
commits6
files27 changed
diff+1,268 −66
merged2026-08-14
M5 gate PASS · spec 2/2 + m5.sh
00

What M5 actually built

After M4 the product forecasts and renders. What it could not do was let the data leave, let the data be destroyed, or run anywhere but dev. M5 closes all three — and turns the retention sentence from copy into enforced behaviour.

01 · export

Three sheets, built in the browser

Summary, Forecast and Exceptions, assembled entirely from payloads the page already holds — no export endpoint, no second round trip. The vendored 925 kB spreadsheet library is code-split and fetched only when someone actually clicks export.

SheetJS 0.20.3vendored client-side onlylazy
02 · retention

The promise, enforced

Deleting a dataset purges both R2 prefixes, the narrative KV keys and the D1 rows. A qa-only, time-boxed debug flag is the single exception, and a daily cron sweeps it — with a §0.3-signed hook so the gate can prove the sweep really deletes.

purge7-day debug daily crontype-to-confirm
03 · production

A prod that cannot be reached by accident

The wrangler config's top level becomes an inert base, dev and prod become explicit environments, and production ships through a manual workflow that provisions, migrates, deploys both halves, pushes secrets and smoke-tests the custom domain.

base: do-not-deployenv.dev env.prodmanual dispatch
export
3 sheets
Summary · Forecast · Exceptions
purge
R2 + D1 + KV
one call, three stores
cron
17 3 * * *
declared in dev and prod
prod
ton-app
manual workflow, ids pending first provision
seeding
seed-client.sh
out-of-band, signed route, no open signup
01

Retention enforcement, end to end

Two paths destroy data: the one the user asks for, and the one that runs whether anyone asks or not. Hover any node to isolate its edges.

call chain signed / privileged trigger response state destroyed / written hover a node to isolate · scroll horizontally if clipped
tier 1Browser
copy
UploadPage
The §3 promise is on screen before a file is chosen — the moment the user is deciding whether to hand over data.
dialog
DeleteDataset
Type-to-confirm. The same promise sentence is quoted here, so destroying data is described in exactly the words that were used to ask for it.
qa only
debug retention
Visible only to qa-role users. Keeps raw and intermediate artifacts for 7 days so a failing parse can be investigated.
tier 2 · deletionCloudflare Worker
route
DELETE /api/datasets/:id
Ownership is checked first: a dataset the session does not own is 404, so deletion cannot be used to probe for ids.
404
purge
purgeDataset()
Three stores, one call: both R2 prefixes listed and deleted page by page, the narrative KV prefix walked, then the D1 rows.
tier 2 · sweeperCloudflare Worker
qa route
POST …/qa/…/debug
404s for anyone without the qa role. Sets debug_until to 7 days out, or to an explicit until so the gate can put it in the past.
cron
17 3 * * *
Declared in both the dev and prod environments. Wiring verified with wrangler dev --test-scheduled.
guard
POST /internal/run-sweeper
The same function behind a §0.3 signature, so the gate can trigger a sweep deterministically instead of waiting for 03:17.
401
sweeper
sweepExpiredDebug
Selects datasets whose debug_until has passed, destroys their raw and intermediate artifacts, then clears the flag so the work is not repeated.
Why deletion order matters

Objects first, rows last. If a purge fails halfway, what remains is a dataset row whose objects are partly gone — visible, retryable, and 404-free only once the row itself is deleted. The reverse order would leave orphaned objects with nothing left to point at them, which no later sweep could find.

02

The sentence, and its three enforcement points

The plan's §3 commitment lives in the codebase as one exported constant, quoted wherever the product asks for a file or offers to destroy one — and asserted by the gate.

src/retention/promise.ts · quoted verbatim

“Your uploaded file is parsed and then deleted from our servers — we keep only the normalized demand history and the forecasts derived from it, visible only to your account. To analyse a new or corrected file, re-upload it; we cannot recover the original. Deleting a dataset removes everything derived from it.”

shown on the upload screen · quoted in the type-to-confirm deletion dialog · asserted by scripts/qa/m5.spec.ts
enforcement pointmilestonewhat is destroyedtrigger
parse-result callbackM2the uploaded workbook itselfthe running → awaiting_mapping transition
forecast-result callbackM3extracted.json, the intermediate artifactthe running → complete transition
purgeDataset()M5both R2 prefixes, narrative KV keys, D1 jobs and dataset rowthe user, through a type-to-confirm dialog
sweepExpiredDebug()M5raw and intermediate artifacts a qa user held backdaily cron, or the signed hook
R2 lifecycle ruleM0anything left under uploads/ after 9 daysthe storage layer, unconditionally
The one exception, and why it is time-boxed

A failing parse is impossible to investigate once its input has been destroyed. The debug flag keeps raw and intermediate artifacts — but only for qa-role users, only for 7 days, only per dataset, and inside the 9-day storage backstop that M0 put underneath everything. The sweeper is the primary mechanism; the lifecycle rule is the net. See finding R2.

03

The export a planner can hand to someone else

Every cell comes from a payload the browser already loaded, so exporting costs one client-side assembly and no server work — and the file contains exactly what the screen showed.

sheet 1 · summary11 rows
  • Datasetfile name
  • Months of historysummary.months
  • Series forecastsummary.series
  • Smooth / Erraticquadrants
  • Intermittent / Lumpyquadrants
  • Median champion MASE2 dp
  • Median seasonal-naive MASE2 dp
  • Exceptionsqueue length
  • Provenanceone sentence
sheet 2 · forecastone row per series
  • SKU · Name · Customeridentity
  • PatternSBC quadrant
  • Championwinning model
  • Backtest MASE2 dp, or the fallback reason
  • 6 period columnsrounded means
  • 6-mo totalrounded horizon total

A series with no fitted model prints fallback: <reason> where a score would be — never a number that was never measured.

sheet 3 · exceptionsthe queue, in order
  • SeverityHigh / Medium / Low
  • SKU · Name · Customeridentity
  • PatternSBC quadrant
  • Why it surfacedthe rule's own sentence
  • Expected quantityrounded horizon total

The same deterministic queue M4 renders, which is why the gate can assert the row count against what is on screen.

Loaded on demand, and only once

The vendored SheetJS build is imported as a URL and injected as a script tag the first time an export is requested, with the load promise memoised. Login, dashboard and the forecast screen never pay for a library they cannot use. See finding N3.

04

Production, on purpose only

Two changes make a production deploy a deliberate act: the config's top level stops being deployable, and prod ships through a manually dispatched workflow that provisions before it deploys.

environmentworker nameworkers_devbindingscron
top levelton-app-base-do-not-deployfalsenone — inert by design
env.devton-app-devtrueD1 · R2 · KV, ids committed17 3 * * *
env.prodton-appfalseD1 · R2 · KV, ids pending first provision17 3 * * *
per-PR previewton-app-pr-<n>truegenerated config, shared dev resources, no cron

deploy-prod.yml — manual dispatch only

Each step must succeed before the next runs; the last one is an assertion, not a deploy.
1

build & test both halves

bun run typecheck · bun run test · bun run build uv sync --locked · uv run pytest -k "not benchmark"

The 100-second forecast benchmark is excluded here; it has its own CI job.

2

provision prod resources — idempotent, drift-checked

bash scripts/provision.sh prod

Creates D1, R2 and KV if absent and fails loudly if the committed ids disagree with reality.

3

migrate the prod database

bunx wrangler d1 migrations apply ton-app-prod --remote --env prod

Schema before code, always.

4

deploy the Modal prod app

TON_APP_ENV=prod → ton-app-prod

Same file as dev; the app name resolves from the environment, and the shared HMAC secret is recreated with --force so rotation is a redeploy.

5

deploy the prod worker, then push its secrets

bunx wrangler deploy --env prod

Secrets are pushed from GitHub secrets on every deploy, never set by hand.

6

smoke: the custom domain serves the login page

curl the production hostname → expect the app, not an error page

The domain itself is Terraform state, not application state — owned by the infrastructure repo. This step asserts it works; it never creates it. See finding N5.

05

The gate, and its evidence

Two Playwright assertions for what a user does, one shell gate for what the storage layer actually contains afterwards — all against a real local stack in real Chromium.

m5.spec.ts · 2/2pass

The workbook is what the screen said

The downloaded file is parsed by the same vendored SheetJS that wrote it and must contain exactly Summary, Forecast, Exceptions in that order — with the Forecast row count equal to the payload's series count and the Exceptions row count equal to the queue total rendered on screen.

The second test deletes the dataset: the dialog must quote the retention promise, the app must navigate home, and a re-read of the dataset must return 404.

m5.shpass

And the storage layer agrees

A spec can only see what the API answers. The shell gate goes behind it, through the qa-only listing, and asserts the objects themselves — including that debug retention provably keeps extracted.json and that the sweeper provably removes it.

scripts/qa/m5.sh — the assertions it prints
PASS: every API read -> 404 after delete PASS: R2 prefix empty after delete PASS: debug retention keeps extracted.json after forecast PASS: expired debug dataset swept (raw + extracted gone) M5 QA gate PASS
quoted from the PR body QA (all against a real local stack in real Chromium): scripts/qa/m5.spec.ts 2/2 — the downloaded workbook parses to exactly Summary/Forecast/Exceptions with row counts matching the payload/screen, and deletion navigates home with every subsequent read 404. scripts/qa/m5.sh PASSES: post-delete API 404s, R2 prefix [] via the qa-only listing, debug retention provably keeps extracted.json, the sweeper provably removes it. M1 gate re-run as regression: PASS.
06

What landed, by area

27 files, +1,268 −66 — the smallest milestone by diff and the one that touches the most surfaces: UI copy, a Worker module, two workflows, the wrangler config and a vendored library.

retention4 files · 219
  • retention/DeleteDataset.tsx107
  • worker/retention.ts58
  • forecast/DebugRetentionToggle.tsx54
  • retention/promise.ts4
export3 files · 167
  • src/export/workbook.ts123
  • vendor/xlsx.full.min.js24
  • src/vendor-xlsx.d.ts20
gates2 files · 285
  • scripts/qa/m5.spec.ts193
  • scripts/qa/m5.sh92
production4 files · 117
  • .github/workflows/deploy-prod.yml66
  • wrangler.jsonc35
  • scripts/seed-client.sh15
  • scripts/provision.sh1
spa wiring5 files · 333
  • src/styles.css212
  • forecast/ForecastPage.tsx101
  • dashboard/DashboardPage.tsx34
  • api/client.ts12
  • upload/UploadPage.tsx7
worker & ci6 files · 96
  • worker/routes/qa.ts50
  • worker/routes/internal.ts14
  • worker/routes/datasets.ts10
  • .github/workflows/ci.yml9
  • .github/workflows/deploy-dev.yml9
  • worker/index.ts6
07

Findings & design notes

Two review items carried since M0 and closed here, plus five decisions that keep production and retention honest. Click any row to expand.

resolved open notes
08

Where this leaves the plan

M0
skeleton & topology
M1
auth + tenancy
M2
ingestion
M3
forecast engine
M4
screen + narratives
M5
export, retention, prod

The plan is complete, and the promise is code

Six milestones on, every claim the product makes has something enforcing it: the security topology was proved before any data existed, the raw file is destroyed at a state transition rather than on a timer, the forecasts are measured against a baseline that CI re-checks on every run, the screen cannot show a number the payload cannot back, and the retention sentence now has four enforcement points and a storage-layer backstop underneath them. What is left is provisioning, not building.

still to happen, outside this PR
  • First prod provision — then the real D1 and KV ids replace the sentinels.
  • The worker custom domain, owned by the infrastructure Terraform repo.
  • Client user seeding, out of band via scripts/seed-client.sh.
what a reviewer should check first
  • That purgeDataset deletes objects before rows — the retry-safe order.
  • That the qa listing route stays tenant-scoped as it gets used more.
  • That the debug window stays shorter than the bucket lifecycle rule.
  • That no future config change re-arms the top-level wrangler environment.