Build & operate
Validate and run
Check a workflow without running it, run all or part of it, and follow progress.
Validate
Section titled “Validate”Validate asks crawler-core to check the workflow without running it. If it rejects something, the node turns red and the panel says why. Validate before every first run.
- Run executes the whole canvas.
- To run one branch, press the ▶ on that branch’s first node.
- A saved workflow is saved first, then run as a background job. The run panel shows per-node progress; closing it loses nothing.
Start small
Section titled “Start small”Always begin with a low page limit (about 8) and a pause of 1 second or more between requests. When a small run is right, raise the limits.
Run history
Section titled “Run history”Every run is recorded in the workspace. Open any past run to see what it did.
Debug a partial or unhealthy run
Section titled “Debug a partial or unhealthy run”Follow the first failed or warning node, then inspect its input and output before changing downstream settings. Use run events for retries, rate limits, health changes, and pauses. Events contain operational information rather than crawled page content.
on_trouble controls warning/stalled behavior: continue reports it, pause holds for intervention, and stop fails the run. A completed graph can still have missing records; compare page counts and sample outputs with the source.
Pause, resume, and rerun
Section titled “Pause, resume, and rerun”Pause requests hold execution at a safe boundary; a running worker can show a pending control request before the run becomes paused. Resume keeps finished nodes and uses current saved node settings, but requires compatible graph ids and edges. Rerun creates a new run, reuses unchanged work where supported, and executes changed nodes and their descendants.
Use from_node_ids to force a branch to execute again. keep_items requests continuation of supported unfinished crawl items when settings and predecessors are unchanged. It is not a universal per-item checkpoint, especially inside For Each. See the HTTP API for exact controls.