Start with the receiving workflow
A useful crawl begins with the decision or system the records will support. A retailer comparing offers needs different observations from a team building a company directory. Describe the downstream use before expanding the source list.
Name the receiving team, expected record structure, and what would make a sample acceptable. An example file is often more useful than a request for ‘all the data’. Keep credentials and confidential documents out of a public enquiry; access arrangements belong in a separate discussion.
Define the collection boundary
List representative URLs and distinguish detail pages from search results, categories, and paginated lists. Explain whether a record represents a product, a variant, an offer, or another entity. Agree how identifiers connect observations across runs.
The source list is a feasibility question as well as a coverage requirement. Accessible pages, access conditions, and source constraints must be assessed before committing to volume or cadence. A requested frequency is not a promise that every source can support it.
- Target URLs and which pages are in scope
- Fields, data types, units and null-value meanings
- Record identity and variant handling
- Expected volume and refresh requirement
- Source URL and observation-time fields
- Access conditions and agreed exclusions
Agree acceptance and delivery
Review representative samples against the specification before scaling. Required-field checks, duplicate checks, variant matching, and volume reconciliation should reflect the intended workflow. A sample can demonstrate a field mapping; it cannot establish uninterrupted future coverage.
Confirm the format and handoff destination, who acknowledges delivery, and what happens after a partial run. Distinguish a missing record, an unavailable field, an access failure, and a delivery failure. Finally, agree maintenance responsibilities and how source or specification changes will be reviewed.
- Sample acceptance criteria and review owner
- CSV, JSON or another agreed delivery contract
- Partial-run and missing-field reporting
- Source-change review and maintenance responsibilities
- A process for changing the agreed scope
Illustrative data and templates explain the method. They do not establish source coverage, a customer outcome, or an agreed delivery commitment.
Explore the relevant services
Custom web crawling ↗ — Commission a web data collection service around your sources, fields, and delivery requirements. From a defined extraction to an ongoing data feed, agree the records your team needs and how they should arrive.