Node reference

URL Fetcher

Read one page, several pages, or everything reachable within a depth.

Reads configured URLs, or URLs supplied by an upstream Query, and emits raw_pages with HTML, links, and metadata plus a crawl summary.

Key options

Option	Meaning
`url` / `urls`	One start address, or several (deduplicated). A missing scheme defaults to `https`. Use one or the other.
`depth`	`0` reads only the given pages, `1` adds the pages they link to, `2` adds theirs. `"unlimited"` is bounded by `max_pages`.
`max_pages`	Maximum pages kept. Start with 8.
`mean_delay`	Seconds between requests. Use 1 or more.
`deep_crawl`	Which links to follow. See below.

Follow only the links you want

{
  "url": "https://example.com/members",
  "depth": 2,
  "max_pages": 8,
  "mean_delay": 1,
  "deep_crawl": {
    "filters": [
      { "type": "url_pattern", "patterns": ["*/members*", "*/member?id=*"] }
    ]
  }
}

Patterns are globs matched against the full URL. One must match the start page too. Without filters a depth of 2 can wander the whole site. See Link patterns.

Sites that need JavaScript

Need	Option
Wait for content	`wait_for` (`css:.result` or `js:...`)
Click, expand, paginate	`js_code`, `js_code_before_wait`, with `session_id`
Lazy or infinite content	`scan_full_page`, `scroll_delay`, `max_scroll_steps`
Virtualized lists	`virtual_scroll_config`
Cookie banners and overlays	`remove_consent_popups`, `remove_overlay_elements`

Other useful options

Trim before cleaning: css_selector, target_elements, excluded_tags, excluded_selector, word_count_threshold.
Links and images: exclude_external_links, exclude_social_media_links, exclude_domains, exclude_external_images.
Many start URLs: urls with dispatcher, rate_limiter, url_configs; or url_source: query_result with url_field to crawl URLs from an upstream Query.
Discovery: url_seeding (sitemaps), domain_mapping, adaptive_crawl (stop when a query is covered), prefetch (links only, fast).
Capture: screenshot, pdf, capture_mhtml, capture_network_requests.
Politeness: check_robots_txt.

Paging caveat

If a pager’s links carry tokens or need JavaScript, a plain fetch may only see page 1. Open the fetcher’s output and check which pages it actually read before trusting that it covered everything.

Output and errors

Inspect actual URLs and failure counts in the fetcher output: hitting a page limit, following the wrong pattern, or a site returning an error can produce incomplete coverage. Invalid configuration is rejected during validation; network, browser, and access failures are execution-time conditions. Do not infer full coverage from a completed status alone.

Use the generated schema for all defaults and bounds, and the JavaScript recipe for dynamic pages. Follow with Markdown Converter or Data Transformer; a raw-artifact branch can also feed Data Source directly.