tried & told

Almost everything written about a cosmetic product is either an advertisement or an aggregate score. The genuinely useful text — someone describing what a formula did to their skin over three weeks — is scattered across forums, comment threads, and personal blogs, and general web search is not built to surface it.

The question was whether a narrow vertical index, built for one kind of document, could find first-person accounts that a general engine buries.

Tried & Told is a search engine, not an interface over someone else's search API. The crawler, the index, the ranking, and the serving layer were all implemented.

The index is disk-backed: postings are written in segments with delta and varint compression, and a segment-based garbage collector reclaims space as documents are replaced. Positions are stored alongside document IDs so phrase queries can be resolved exactly rather than approximated by proximity heuristics.

Crawling is crash-resumable — frontier state and fetch progress are persisted, so a killed process restarts where it stopped rather than refetching a site. Duplicate detection runs in two layers: a Bloom filter to reject URLs and exact documents cheaply, and SimHash to catch near-duplicates, which are common when the same review is syndicated across affiliate sites.

Ranking combines BM25 over the lexical index with approximate nearest-neighbor retrieval over an HNSW graph, merged with reciprocal rank fusion.

Why a custom index rather than an existing engine? The document population is small and unusually homogeneous. Owning the postings format made it possible to store what this corpus actually needs — positions, per-document provenance — without paying for a general-purpose engine's operational surface.

Why fusion rather than one retriever? BM25 is precise when the query uses the vocabulary of the review; embeddings are better when it doesn't ("stings on broken skin"). Reciprocal rank fusion needs no score calibration between the two, which matters because their score distributions are not comparable.

Why Go for serving and Python for the pipeline? The API benefits from predictable concurrency and low allocation pressure; the offline embedding and evaluation work benefits from the Python ecosystem. The boundary between them is the index on disk.

The current index holds 835 first-person reviews crawled from 45 sites, covered by 163 automated tests spanning the postings format, compression round-trips, crawler resumption, and ranking behavior.

Compression and garbage collection were less difficult than expected; deciding what counts as a duplicate was harder. Syndicated reviews differ by a sentence of boilerplate, and a SimHash threshold tight enough to catch them occasionally collapsed two genuinely different accounts of the same product.

The part I'd do differently is evaluation. I built retrieval before I had a labeled set to judge it with, which meant early ranking changes were argued rather than measured.