Technical reference
Link patterns
Control which links the fetcher follows.
Patterns are globs (* matches anything) matched against the full URL.
| Pattern | Matches |
|---|---|
*/company?id=* | https://x.com/company?id=42 |
*/products/* | any page under /products/ |
*/list* | /list, /list?page=2, /listing |
Rules of thumb:
- Include a pattern for the list pages (so paging is followed) and one for the detail pages.
- One pattern must match the start page, or nothing is followed.
- Always filter. Depth 2 without filters can wander the whole site.