From external signal to reliable data product — extraction, enrichment, validation, monitoring
Vojislav Vujić
Prototype for Better Collective — Research & Insights Source code on GitHub
Competitive signal arrives from sources you don't control and can't ask to stay still. The hard part
isn't the first extraction — it's the third month, when a selector silently returns nothing and nobody
notices until the numbers are already wrong. This console demonstrates the loop I'd build around that
problem: acquire, parse, enrich, validate, load, and watch. Press Break a source to see what
happens when the outside world changes shape.
This hosted demo runs on bundled fixtures — no live network calls, so nothing here depends on a third
party being up or on an API key in the browser. The parsing, selector-fallback, validation and
drift-detection logic running on this page is real. The live-network implementation — retry and backoff,
rate limiting, robots.txt handling, LLM enrichment, alert dispatch — is in the repository.
Pipeline
01
Acquire
02
Parse
03
Enrich
04
Validate
05
Load
Pipeline health
—
no run yet
Last heartbeat
—
awaiting first run
Records acquired
0
across 3 sources
Schema drift
0
selectors matching
Failed sources
0
all responding
Run the pipeline first, then break a source mid-flight.
Sources
Source
Mode
Records
Latency
Drift
Status
Event log
No events. Run the pipeline to begin.
Acquired records — raw payload parsed to schema, then enriched
Operator
Offer
Value
Market
Classification
Provenance
No records.
From prototype to team
Days 0–30
Map and measure
Inventory every source currently feeding Research & Insights, including the manual ones nobody has written down
Classify by acquisition pattern: documented API, undocumented API, static HTML, rendered HTML, file drop
Baseline the cost of each manual workflow in hours per week — that becomes the prioritisation ranking
Set the monitoring standard before building anything: what a heartbeat is, what a break is, who gets told
Days 30–60
Productionise the top of the list
Ship the highest-value pipelines against a shared extraction framework rather than one-off scripts
Provenance on every record: source, retrieval timestamp, parser version, confidence