HTTP API

Integrate through crawler-api, the product backend. crawler-core is the internal execution engine. Set BASE_URL to your product backend origin, without a trailing slash, and TOKEN to a valid session or personal access token. Keep tokens out of source control and client-side bundles.

Authentication

Send Authorization: Bearer <token>. Browser sessions use the HttpOnly session cookie. POST /api/auth/login accepts username and password and returns user, token, and expires_at; account approval and active-account checks apply.

Use the server’s /openapi.json for its exact deployed contracts, response schemas, and additional routes. No published SDK installation command is verified in this repository; the examples below use HTTP directly.

1. Create a workspace

Create a container for saved workflows

curl --fail-with-body "$BASE_URL/api/workspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"name":"First workflow"}'

Returns 201 with id, owner, name, created_at, and updated_at. Set WORKSPACE_ID to the returned id.

2. Save a canvas

This self-contained Text workflow tests the product API without requiring a crawl or model call. Node ids must be unique; settings belong in data on the product canvas.

Create a minimal saved workflow

curl --fail-with-body "$BASE_URL/api/workspaces/$WORKSPACE_ID/workflows" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"name":"Hello workflow","canvas":{"nodes":[{"id":"hello","type":"text","position":{"x":0,"y":0},"data":{"label":"Hello","text":"Hello from ez-crawl"}}],"edges":[]}}'

Returns 201 with id, workspace_id, name, version, canvas, created_at, and updated_at. Set WORKFLOW_ID to its id. A draft can be saved before it is valid; creation does not execute the graph.

3. Validate and run

Validate the saved graph

curl --fail-with-body -X POST \
  "$BASE_URL/api/workflows/$WORKFLOW_ID/validate" \
  -H "Authorization: Bearer $TOKEN"

Validation returns 200 with the engine execution plan, or 422 with validation errors.

Start a background run

curl --fail-with-body "$BASE_URL/api/workflows/$WORKFLOW_ID/runs" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"on_trouble":"continue"}'

Returns 202 with the run id, workflow_id, version, node_ids, and created_at (and optional metadata). This acknowledges a background job; it does not mean execution succeeded. Set RUN_ID to the returned id.

Inspect the run

curl --fail-with-body "$BASE_URL/api/runs/$RUN_ID" \
  -H "Authorization: Bearer $TOKEN"

The response relays the engine’s job status, per-node progress, and result. Poll until an ended or paused state; inspect errors before consuming outputs. Do not start another run merely to check the previous one.

Editing and controlling runs

Method and path	Body / behavior
GET /api/workflows/{id}	Read the latest canvas and version
PUT /api/workflows/{id}	{name, canvas, version}; stale versions return 409
GET /api/workflows/{id}/runs?details=true	History, with details for the newest 20 runs
POST /api/runs/{id}/cancel	Requests cancellation; returns 202
POST /api/runs/{id}/pause	Requests a pause; returns 202
POST /api/runs/{id}/resume	Optional {on_trouble}; preserves finished work, graph ids and edges must stay compatible
POST /api/runs/{id}/rerun	{from_node_ids, on_trouble, keep_items}; creates a new run
GET /api/runs/{id}/events?limit=100	{items, next_cursor}; pass the cursor on the next request

For run requests, on_trouble is continue, pause, or stop. node_ids optionally limits the saved graph to specified nodes and the edges between them; do not assume omitted dependencies will execute.

Read stored data

GET /api/data/stores lists authorized databases. POST /api/data/{store_id}/find accepts a collection, supported filter, sort, projection, skip, and limit:

Find request body

{"collection":"records","filter":{},"skip":0,"limit":20}

The response is {items, total, has_more}. limit is at most 100, skip at most 10000; counts are capped and long string previews may be truncated. Fetch a whole document through GET /api/data/{store_id}/documents/{collection}/{document_id}. See storage for scope and ownership.

Errors

Status	What to do
401	Check credentials, expiry, and account status
404	Verify the id and access; inaccessible resources can be hidden as not found
409	Reload the workflow version or check the run’s current state
422	Correct the request, graph, or unavailable resource using the returned detail
503	The engine or backing service is unavailable; check service health before retrying

Errors commonly include detail.message; relayed engine errors can have their own structured body. Preserve the full response when diagnosing a failure. See troubleshooting or use MCP for agent integrations.