# A better brief for a managed web crawl

Turn a data request into an agreed specification: sources, fields, refresh, quality checks, and delivery.

Published: 2026-10-03 · Zeneral editorial team

Canonical: https://zeneral.ai/resources/specify-a-managed-web-crawl

## Start with the receiving workflow

A useful crawl begins with the decision or system the records will support. A retailer comparing offers needs different observations from a team building a company directory. Describe the downstream use before expanding the source list.

Name the receiving team, expected record structure, and what would make a sample acceptable. An example file is often more useful than a request for ‘all the data’. Keep credentials and confidential documents out of a public enquiry; access arrangements belong in a separate discussion.


## Define the collection boundary

List representative URLs and distinguish detail pages from search results, categories, and paginated lists. Explain whether a record represents a product, a variant, an offer, or another entity. Agree how identifiers connect observations across runs.

The source list is a feasibility question as well as a coverage requirement. Accessible pages, access conditions, and source constraints must be assessed before committing to volume or cadence. A requested frequency is not a promise that every source can support it.

- Target URLs and which pages are in scope
- Fields, data types, units and null-value meanings
- Record identity and variant handling
- Expected volume and refresh requirement
- Source URL and observation-time fields
- Access conditions and agreed exclusions

## Agree acceptance and delivery

Review representative samples against the specification before scaling. Required-field checks, duplicate checks, variant matching, and volume reconciliation should reflect the intended workflow. A sample can demonstrate a field mapping; it cannot establish uninterrupted future coverage.

Confirm the format and handoff destination, who acknowledges delivery, and what happens after a partial run. Distinguish a missing record, an unavailable field, an access failure, and a delivery failure. Finally, agree maintenance responsibilities and how source or specification changes will be reviewed.

- Sample acceptance criteria and review owner
- CSV, JSON or another agreed delivery contract
- Partial-run and missing-field reporting
- Source-change review and maintenance responsibilities
- A process for changing the agreed scope

Examples are illustrative. Scope and delivery are agreed for each engagement.

Explore services: https://zeneral.ai/ai
Discuss your requirement: https://zeneral.ai/contact