What is ez-crawl?

ez-crawl is a visual data engineering platform with an agent interface. Give an AI agent a data goal and it can construct, validate, and execute a workflow. Open that same workflow on the canvas to inspect each step, change its settings, and reuse the process.

You can also build workflows manually. The agent and the editor work with the same saved graph.

From a question to a reusable process

1. Describe the sources, fields, and constraints: “Read these company pages and return names and contact details. Test eight pages first.”
2. Inspect the generated nodes and connections. Check URLs, extraction instructions, model selection, and storage before a larger run.
3. Refine the process using node settings or an agent request. Validate, execute, and inspect intermediate outputs.
4. Reuse the saved workflow, query stored records, or compose per-record processing with an inline subworkflow.

Start with your first workflow or connect an agent.

What a workflow looks like

Example: pages to structured records

URL Fetcher → Markdown Converter → LLM Processor → Table
                                        └──────→ MongoDB Store

This is one useful pattern, not a required sequence. Query stored data without crawling, transform stable markup without a model, combine records from multiple branches, or save files referenced by records.

Goal	Start with
Read websites, including JavaScript-rendered pages	URL Fetcher
Clean HTML or extract values deterministically	Data Transformer
Extract structured fields or answer questions	LLM Processor
Enrich each record with its own process	For Each
Merge records from multiple sources by a key	Combine
Persist records and files	Storage nodes
Read, filter, and revisit saved data	Query

What is available today

The repository implements the canvas, saved workflows, background runs, MCP tools, thirteen engine node types, MongoDB document storage, and S3-compatible file storage. Platform resources and supported user resources are resolved by the product backend.

The plugin marketplace is a preview; its integrations cannot be installed. This documentation does not establish a published SDK or a registry of independently versioned, callable subworkflows. See architecture and availability for the verified boundaries.

Keep the evidence

A successful run is not proof that extracted values are correct. Compare sample records with the source, inspect missing fields, and increase crawl limits only after the small run works.