Architecture and availability

Three cooperating applications

Request and execution boundaries

Visual editor / MCP client
           ↓
crawler-api — identity, permissions, saved canvas, resources, runs
           ↓
crawler-core — graph validation, background jobs, node execution
           ↓
MongoDB documents / S3-compatible blobs

Use crawler-api for product integrations. The frontend does not connect directly to crawler-core or receive storage connection secrets. At execution time, crawler-api resolves the workspace owner’s platform storage or authorized resource ids and passes connections to the engine.

Canvas and execution graph

A saved canvas has nodes and edges. Canvas nodes carry settings in data; the engine receives normalized node config. These are different payloads: do not submit an engine graph as a product canvas without adapting it. See API usage.

Connected nodes exchange named channels such as raw_pages, pages, llm_result, and query_result. Branches retain producer-specific outputs. Some nodes stream page results as they arrive; aggregation nodes such as Combine wait for all branches.

Execution history records the workflow version that ran. Saving a workflow increments its version; stale writes are rejected. See execution and debugging.

Availability boundaries

Capability	Repository status	Boundary
Agent-built, inspectable workflows	Implemented	Hosted MCP tools operate on saved canvases
Per-record inline subworkflows	Implemented	forEach; nested forEach is rejected
Combining branch records	Implemented	combine, grouped by a configured record key
Database and file browsing	Implemented	Product data/file routes enforce access before relaying to core
Model and MongoDB resources	Implemented	Chosen by authorized resource id
Plugin marketplace	Preview	Catalogue and settings preview; installation is unavailable
Published client SDK	Unverified	Use HTTP or MCP; no SDK installation command is asserted here
Independently versioned subworkflow registry	Unverified	Inline flows do not establish a registry or callable saved-flow interface

These statuses reflect source inspection, not a guarantee that a particular deployed version contains every change. For the actual server’s accepted fields use its node catalogue and OpenAPI specification.

Source of truth

The product’s architectural contracts live in the sibling protocol repository. Node models and runtime validation live in crawler-core/app/api/contracts.py and crawler-core/app/schemas.py; tools live in crawler-api/app/mcp_server.py. The generated parameter reference records declared field types, defaults, and bounds without duplicating them by hand.