Data Flywheel Design

A data flywheel is the mechanism by which your product gets better every time it's used — and by which that improvement becomes a competitive advantage that's hard to replicate. Most teams have the data but no flywheel. They generate implicit signals from user behaviour and discard them. This skill designs the loop: what data to collect, how to label it, how to use it to improve the model, and how to measure whether the flywheel is spinning.

---

Context

What a data flywheel is:

More usage → more data → better model → better product → more usage.

The loop compounds over time. A competitor starting today can buy the same base model you use. They cannot buy your accumulated, labelled, product-specific training data.

The three types of flywheel data:

Type	What it is	Example
Explicit feedback	User deliberately rates or corrects the output	Thumbs up/down, edited drafts, star ratings
Implicit feedback	User behaviour reveals output quality without them saying so	Accepting vs. discarding an AI suggestion, re-querying, dwell time
Outcome data	What happened downstream of the AI output	Did the email sent via AI draft get a reply? Did the AI-recommended product sell?

Outcome data is the most valuable and the hardest to collect.

---

Step 1 — Map the current data generation

Audit: what AI outputs exist, what user actions are logged, what outcomes are logged, and what signals are being discarded.

Step 2 — Design the signal collection layer

Explicit feedback signals (binary rating, edit detection, freetext correction, re-query detection) and implicit behaviour signals (time-to-action, acceptance rate, downstream conversion, session abandonment).

Step 3 — Design the model improvement loop

Four loop types:

Loop 1 — Prompt improvement: Weekly review of negative feedback clusters → prompt update

Loop 2 — Few-shot example refresh: Monthly refresh of best-rated outputs as few-shot examples

Loop 3 — Fine-tuning: When 500+ labelled pairs available and prompt engineering has plateaued

Loop 4 — Outcome-driven RLHF proxy: Use downstream outcome data as training signal (advanced)

Step 4 — Measure flywheel velocity

Leading indicators: signals collected/week, signal quality rate, prompt update frequency.

Lagging indicators: output acceptance rate trend, re-query rate trend, task completion rate trend.

Step 5 — Identify the flywheel moat

Assess data volume moat, data quality moat, data specificity moat, and network effect moat.

Step 6 — Output the data flywheel design document

Include: data audit, signal collection design, improvement loops, velocity metrics, moat assessment, and 90-day sprint priorities.

Quality check before delivering

Current data audit is honest — including what's being discarded

At least one high-quality labelling signal is defined (edits or outcome data)

At least Loop Type 1 (prompt improvement) is defined with a cadence

Flywheel velocity metrics are specific

Moat assessment identifies when the data advantage becomes defensible

90-day sprint has an engineering owner for priority 1

Suggested next step: Start with edit capture. If your product shows AI-generated text that users can edit, instrument that edit event today — the diff is your highest-quality training signal and it costs almost nothing to collect.