Node reference

MongoDB and file storage

Persist structured records and referenced files with explicit storage nodes.

Use mongoStore for structured records and s3Store for files referenced by records or selected from HTML. Use Data Source to keep crawl artifacts such as raw pages, Markdown, and extraction outputs.

MongoDB Store

The mongoStore node reads upstream records and writes to the records collection. The backend supplies the storage connection. An optional derive list creates fields; key configures record matching and list-update behavior.

{
  "derive": [{"field": "domain", "from": "website", "normalize": "host"}],
  "key": {"field": "domain", "match": "one", "lists": "union"}
}

Normalization is exact by default or host for a hostname. Key matching supports one or all; list handling supports replace (default) or union. Choose a stable key before repeated collection runs. Without a key, do not assume repeated runs deduplicate records.

Inspect the node’s database_result for storage counts and ids. Missing upstream records or invalid key/derive rules must be resolved before relying on persistence.

File storage

Connect an extraction-producing node; s3Store requires extraction_result.data, including when selecting URLs from HTML. The node currently writes to platform file storage; user-owned S3 connections are rejected.

Configure exactly one of from_field, from_fields, or select. The first two read URLs from record fields; select uses {css, attribute} to find URLs in HTML. as selects the output reference field.

{
  "from_field": "logo_url",
  "as": "logo",
  "kind": "image",
  "max_files_per_item": 1,
  "max_bytes": 10000000
}

Parameter	Default and constraint
`kind`	`image`; also `pdf`, `video`, `file`
`max_files_per_item`	`1`; range `1–20`
`max_bytes`	`10000000`; range `1–100000000` per file
`allow_types`	Optional list of content-type patterns

The output carries stored-file references attached to records. Inspect s3_result.saved, skipped, and bytes; a URL does not guarantee a retrievable file. The backend injects blob_store; do not embed bucket credentials in node data.

Inspect and reuse

Browse stored documents and files under Resources, inspect a specific run’s output, or read the records collection with Query. See resources and lifecycle for ownership and retention boundaries and the generated schemas for declared field types.