One-off scrapes are demos. The hard part of data collection is the recurring version: the same sources, week after week, while storefronts redesign, marketplaces rotate templates, and anti-bot vendors earn their renewals. Pipelines that survive that churn share unglamorous design choices.
Collection fails at the matching step, not the fetching step
Fetching a page is the visible 10 percent. The durable problem is deciding that the product on page A this week is the same product that was on page B last month—different URL, renamed variant, merged listing. Catalog matching needs stable keys, fuzzy fallbacks, and a review queue for the ambiguous tail. Teams that skip matching get numbers that look precise and compare nothing.
Freshness is a budget, not a cron schedule
"Run it nightly" is not a freshness policy. Prices on promotional pages move hourly; assortment moves weekly; shelf positioning barely matters past daily. A production pipeline prices each source by how fast its data decays and how much a stale row costs, then schedules accordingly—and exposes collection lag in the same dashboard as the data itself, so a stale feed is visible before it becomes a wrong decision.
Bot walls reward boring engineering
Rotating everything forever is an arms race most budgets lose. The pipelines that hold up behave like polite infrastructure: steady rates, honest retry backoffs, cache-first rechecks, and source-specific adapters that fail loudly instead of degrading silently. When a wall changes, you want an alert naming the source and the selector class that broke—not a slow drift in coverage discovered during a quarterly review.
Delivery format decides whether the data gets used
Intelligence that lands as CSV attachments gets opened once. The same data delivered through dashboards, threshold alerts, and delivery APIs that plug into the buyer's existing tools becomes operational. The delivery surface is part of the scraping system, not a favor done afterward.
This is the problem space Crawlr, the Cherry Tech commerce-intelligence platform, is built around: recurring collection, product matching, and price, promotion, assortment, and digital-shelf monitoring delivered as a production service rather than a script somebody inherits.