Abstract — Competitive price intelligence exists to turn a noisy market into informed decisions: where we're losing the cheapest spot, which competitor is repositioning, and what to do about it. Building one normally depends on scraped competitor prices; instead I drive it with synthetic data shaped exactly like a real feed, so the engineering stays the focus. Over that feed runs the actual system — an in-process vector index that resolves messy listings to our catalog under explicit confidence, a deterministic anomaly brain that separates signal from noise, and a real-time spine that streams every change to the browser in under a second — all in one self-contained Go binary.
A competitive price-intelligence system has a chicken-and-egg problem before you write a line of the interesting code: it needs a market to watch. Real competitor prices mean scraping — a legally fraught, anti-bot arms race that proves nothing about my engineering. So I drew a hard line: the acquisition is synthetic and narrated; the intelligence — matching, uncertainty, anomaly detection, the real-time delivery — actually runs.
The whole thing ships as one Go binary: an embedded event bus, an embedded SQLite database, a server-rendered UI streaming live updates, and the synthetic market all in the same process. No external services, no JS build step, deployed by copying a single file to a VPS. Everything below runs inside that one process.
1. Synthetic data that behaves like a real market
The naive version of fake data is a random walk — prices jittering every second. It looks alive and means nothing. Real grocery prices are the opposite: sticky. A price holds for weeks, then moves once, for a reason — a competitor repositioning, a cost pass-through, a seasonal event. Most of the time, nothing meaningful happens.
So the generator models that. It builds an in-memory market from our catalog across five stores — Lidl as "us", plus Mercadona, Carrefour, Dia and Aldi — each anchored to a different baseline price factor. Every store-product pair gets a sticky baseline, and a collection pass only rarely disturbs it: by default a crawled item has a 2% chance of moving on any given run, and when it does it drifts 2–8% with a mild pull back toward its baseline so prices don't wander off forever.
Crucially, the generator does not model a ticker. It models collection runs. Each run crawls one source (one store), round-robin, the way a real scraper fleet schedules per-site crawls. That means cells that weren't re-collected this pass visibly age — which is what gives the board honest freshness and provenance instead of a uniform "everything updated just now" lie.
And the competitor side is messy on purpose, because that's the real problem. From a clean catalog product the generator synthesizes a competitor listing the way a real source would mangle it:
- titles are noisy and inconsistent — brand and name get reordered, accents stripped, sizes abbreviated (
"2L"/"2 litros"/"2 l"), with occasional filler like"oferta"or"envío gratis"; - the EAN is present and clean only ~55% of the time; ~7% of the time it's a corrupted barcode that won't exact-join (forcing a fuzzy fallback); the rest of the time there's no barcode at all;
- coverage is sparse and ragged — each competitor stocks only ~85% of our list, so a missing item is an empty matrix cell that means "not stocked here", distinct from a coverage gap or an undercut;
- each chain also carries a handful of exclusives — private-label or ranges we simply don't sell (
"Kombucha jengibre y limón 400ml","Tofu firme ecológico 250g") — that have no equivalent in our catalog and should land as honest non-matches; - and occasionally a listing's title mutates under a stable URL, simulating a site quietly swapping the product behind a link.
It publishes all of this to the in-process event bus exactly the way a real crawler would, on two subjects: our own store emits price.observed (we already know the product id), while every competitor emits snapshot.observed (a messy title + maybe-EAN that has to be resolved before its price can mean anything). That split is the whole point of the seam:
Swap the synthetic generator for real scraper workers and nothing downstream changes — they publish to the same two subjects. The fake part is a stand-in shaped like production, not a shortcut around it. The generator never touches the database; it only emits.
2. A clock I can fast-forward
A sticky market creates a tension: if nothing moves for weeks, a visitor lands on the page and sees… nothing. A demo of stillness is a demo of nothing.
The fix isn't to make the data lie — it's to move time. The whole system runs on a process-wide simulated clock that compresses real seconds into sim-seconds by a configurable factor (default 120: a sim-day passes in ~12 real minutes, so a market that moves "a few times a week" actually moves while you watch):
// Everything that is an OBSERVATION or FRESHNESS time reads from here — price
// observed_at, anomaly raised_at, the "·3m" / "2h ago" ages, the chart X axis.
// Set the factor to 1 and the exact same code is a real-time scraper — the
// acceleration is a knob, not a fiction baked into the logic.
func NowMs() int64 {
elapsed := time.Now().UnixMilli() - epochReal
return epochSim + int64(float64(elapsed)*Factor())
}
The discipline that makes this safe is the split between two notions of time. Sim-time drives anything that is an observation or a freshness age. Wall-clock time (time.Now directly) drives HTTP timeouts, SSE heartbeats and coalescing — those are real-time regardless of how fast the simulation runs.
Three details I'm glad I got right:
- No cold start. On a fresh boot it seeds ~90 sim-days of backdated, sticky history for every item, so the board, the charts, freshness and coverage all have real depth from the very first page load. Every seeded reading is flagged historical: ingest persists and matches it but raises no alerts and pushes no live morph — re-announcing the past on every boot would just flood the feed with stale "news."
- It resumes. On a returning boot the sim clock continues from the last stored observation instead of resetting to wall-now, and the live generator picks each competitor item back up at its last persisted price (so the first emit after a restart doesn't look like a price move and fire spurious alerts). That's what makes "online forever" coherent across restarts.
- The UI only ever shows relative time — ages, never an absolute sim-date — so sim-time can climb indefinitely without anything looking wrong.
3. A vector database, in-process, as a rebuildable projection
The load-bearing problem is entity resolution: is this competitor's messy title the same product as one of ours — and the same size? Get it wrong and every number downstream is a lie.
I used vecdb from the toolbelt library — a pure-Go, in-process vector index. No Pinecone, no sqlite-vec, no network hop. On boot I embed our whole catalog into a flat 256-dimension cosine index, and treat it as a rebuildable projection: it's a pure function of the catalog, so I never persist it — I rebuild it on startup and let it grow when a product is promoted. That fits the system's CQRS spine, where an append-only event log is the only source of truth and everything else is derived.
Matching is tiered, and the tiers exist to express confidence:
// Tier 1 — deterministic fast lane: a clean EAN that exact-joins our catalog.
if pid, ok := byEAN[ean]; ok {
return Result{ProductID: pid, Method: "ean", Confidence: 1.0, State: "auto"}
}
// Tier 2 — fuzzy lane: embed the title, take the 3 nearest products by cosine,
// then ADJUST that raw similarity by hard attributes (size, brand).
hits := index.Search(3, embed(competitorText)...)
conf := confirm(1-hits[0].Score, normalized, product(hits[0].ID))
switch {
case conf >= 0.80: state = "auto" // price flows onto the board
case conf >= 0.58: state = "review" // a human decides
default: state = "rejected" // a gap / whitespace — not a guess
}
Two things make this more than "nearest neighbour and hope".
The size trap dies in confirm. Two embeddings can look near-identical for "Coca-Cola 2L" and "Coca-Cola 500ml" — the brand and product words dominate. So raw cosine similarity is boosted or slashed by attributes parsed out of the title: if the sizes agree on a common basis the score gets +0.12; if they disagree it gets halved (almost certainly a different SKU); brand agreement nudges ±0.10/0.08. Size is the decisive guard, not a tiebreaker.
Uncertainty is a first-class outcome, not an error. A confident match auto-accepts and its price flows onto the board. A middling one lands in a review queue for a human. A weak one is rejected — which, for a competitor exclusive, is exactly right: it's whitespace, a product no one on our side sells, and that's intelligence too. And when a previously-trusted title drifts to a different product, its mapping is flagged suspect rather than silently re-pointed — the system would rather say "I might be wrong about this" than quietly compare two different things forever.
Only an auto mapping ever flows a competitor price onto the board. That single rule is why a competitor cell on the board is a confident match — never a guess dressed up as data.
4. The real-time spine — how a price reaches the screen
Real-time isn't a feature here; it's the default delivery mechanism for everything. The flow from a synthetic observation to a pixel is one straight line through the bus:
feed (simulator) --price.observed / snapshot.observed--> ingest (SOLE DB writer)
| append price_events (append-only log)
| upsert current_prices (projection)
| match snapshots → mapping state
| evaluate the anomaly brain
| republish price.changed / anomaly.raised
v
web (SSE handlers) <--subscribe NATS-- stream Datastar fat-morphs --> [Browser]
A few invariants hold this together:
ingestis the only thing that writes to the database. Everything else — the read models, the web layer — is read-only. One writer means no write races and a single place where alerts are evaluated.price_eventsis append-only and is the source of truth and full history.current_pricesis a projection rebuildable purely from that log. State is never updated in place.- The browser does no polling and no client-side fetching. Each page opens one long-lived SSE connection with a GET and keeps it open for the life of the page. Every server→browser update travels down that single stream as a fat morph — the full re-rendered region — which Datastar morphs into the DOM. New data reaches the screen in well under a second.
The interaction contract is deliberately inverted from the usual "POST returns HTML". When you change a board filter or confirm a mapping, the browser sends a POST command that returns no HTML body. The handler mutates server-side state and publishes that change to NATS; the page's already-open SSE stream observes it and pushes the re-rendered region back down. User actions and price ticks reach the screen through the exact same path — the bus.
The streaming engine itself does two unglamorous but important things. It coalesces bursts: the first event after a render arms a short timer and the re-render fires once when it elapses, so a flurry of ticks becomes one morph instead of fifty. And it offers an optional heartbeat re-render so quiet pages — the briefing, the relative "2m ago" ages — stay current even when the market isn't moving.
5. The product itself — KPIs, alerts, exceptions, products
All of the above is plumbing. The product is what it lets you notice. It's organized at two altitudes: category is the hero, the SKU is the drill-down — because a category manager decides at the category level and only zooms in to a single product when something demands it.
Home — the category matrix + a briefing. The landing view is a category × store standing matrix: for each category, how often we hold the cheapest spot, who leads it, how many SKUs each competitor undercuts us on, and the average price gap to each rival. Above it sits a briefing — a short synthesized digest ("we're cheapest on 64% of tracked products across 8 categories; biggest pressure: Dia undercuts us on 11 SKUs; latest signal: …"). Every number in that briefing is computed deterministically from the read models first; only the phrasing is generated. More on that seam in §6.
KPIs. The read-model layer (read-only, off the projection) computes the figures a buyer actually steers by:
- Price position — the share of products where we're the cheapest;
- Movers — the biggest percentage price changes over a window;
- A store price index — each store's average price over time, all indexed to a shared baseline of 100 at the window start, so a line below 100 is cheaper than the field began and its slope is the direction it's heading;
- Coverage — how much of the competitor catalog we've confidently resolved (auto / review / suspect / whitespace), and its mirror, coverage gaps: our products with thin competitor visibility — blind spots that are themselves competitive intelligence.
These are computed as queries and meant to be throttled off price.changed, not recomputed per event.
Alerts — the anomaly brain. A wall of N×M prices drives zero decisions. The product is the noticing, and it runs today as a deterministic engine over the price stream. It distinguishes a few kinds of extraordinary:
- Leadership lost — a competitor newly drops below our price on a SKU. It's edge-triggered: a rival that simply stays cheaper doesn't re-alert forever, only the moment we lose the position.
- Magnitude — a single move larger than a threshold (8% by default) against that item's own recent price.
- Coordinated cluster — three or more competitors undercutting us in the same category within ~90 seconds (with a cooldown so it doesn't re-fire). That reads as a repositioning, not noise — a genuinely higher-order signal than any single undercut.
- System alerts — a mapping gone suspect (drift) or a coverage gap. These are deliberately a different scope from market alerts: one says "the market moved," the other says "I might be wrong about something." Conflating them is how dashboards lose trust.
None of this needs a model — it's the boring, reliable core, and getting it right is what lets everything above it stay trustworthy. The full feed lives on its own page with range, search and pagination, all driven through the same SSE-morph contract.
Exceptions — where the system is honest about doubt. Everything the matcher wasn't sure about surfaces here, in three queues: review (middling-confidence matches awaiting a human), suspect (trusted mappings that drifted), and gaps/whitespace (competitor items that resolved to nothing of ours). A human can confirm a match (it becomes auto; the price starts flowing on the next run), reject it, or promote a whitespace item into a tracked watch-only product — something we don't sell but want to monitor. Promotion is the one place the catalog grows at runtime: it creates the product, teaches the matcher about it on the fly so future snapshots match it, and flows its latest price immediately. Every one of these actions is a POST command; the actual write still goes through ingest, the sole writer — the page only commands.
Products — the evidence. The SKU board is the full product × store matrix with per-cell freshness and provenance, where you go to verify a category-level claim. Each product drills into a detail view with a server-rendered SVG history chart — per-store price series over time, no client charting library, just SVG streamed down the same connection.
6. The AI layer 🚧 — under construction
🚧 This is the part I'm actively building. Everything above runs today. The two "AI" surfaces currently run on local, deterministic stand-ins behind clean interfaces — and that's deliberate: I wanted the pipeline, the uncertainty handling, the projections and the real-time delivery correct first, with the models as swappable leaves I drop in without touching anything else.
Where it stands and where it's going:
Matching — embeddings. Today's embeddings are lexical: signed feature hashing over word unigrams, word bigrams and character 3-grams into that 256-dim vector. They're genuinely good at brand/size/spelling overlap and keep the binary self-contained, but they only match shared tokens, not meaning — "refresco de cola" won't pull "Coca-Cola" the way a semantic model would. Next: swap in a small neural embedding model behind the same Embed interface; the vector index, the confidence gating, and the review queue don't change.
Normalization — a fine-tuned small model. Turning a messy title into {brand, variant, size, unit} is, today, regex plus a brand vocabulary. It's the right kind of task to fine-tune a small model on — not to call a big LLM for on every item:
- it's high-volume (every competitor item, every collection run) and bounded (a fixed output shape), so a small local model is cheap, fast, and has no per-call cost;
- it keeps competitor data out of a third party's hands;
- and the training data is easy to distill: use a capable LLM once to label a few thousand titles into structured fields, then fine-tune the small model on that. The big model is the teacher; the small model does the shift work.
Briefing — narration, not invention. The home briefing already computes every figure deterministically; a templated narrator turns those facts into prose today. It's split on purpose into computeBriefingFacts (numbers, SQL/Go only) and narrate (facts → prose) — and narrate is the only seam an external model would ever touch. The upgrade is to have a capable LLM narrate those facts — fed only the tiny, numbers-already-computed facts struct, strictly forbidden from inventing a figure. And then the real frontier: a decision loop that turns the briefing into a recommendation with a rationale and a guardrail. The "so what / now what," not just "here's what happened."
The reason it's structured this way is the point I most want to make: the seams were designed before the models exist, so "add the AI" is a swap, not a rewrite.
The result runs as a single Go binary — embedded bus, embedded database, server-rendered UI streaming live updates, synthetic market and all — deployed by copying one file to a VPS. It's live at pi.manulobato.com, resets nightly, and you're welcome to poke the review queue and break things.