Technical reference

Node configuration schemas

Generated field types, defaults, and declared constraints for the workflow engine.

Generated from crawler-core/app/api/contracts.py. Regenerate with python3 scripts/generate-reference.py from the documentation repository with sibling backend checkouts.

These are engine config fields, not complete canvas payloads. Use them inside a node’s data in the product API. Resource ids are resolved by crawler-api; store and blob_store are internal injected connections. Never place connection secrets in a saved canvas.

Required means no declared default; None means omitted/null by default. Runtime graph validation also enforces cross-field rules: consult the node guide and validate before running. Model names refer to the option sections below; Python Literal values are the allowed enum values.

urlFetcher

Field	Type / allowed values	Default	Constraints
`url`	`str \| None`	`None`	—
`urls`	`list[str] \| None`	`None`	min_length: 1
`url_source`	`Literal[‘config’, ‘query_result’, ‘records’]`	`'config'`	—
`url_field`	`str \| None`	`None`	—
`carry`	`list[str] \| None`	`None`	min_length: 1
`depth`	`Annotated[int, Field(ge=0)] \| Literal[‘unlimited’] \| None`	`0`	—
`max_pages`	`int`	`30`	gt: 0
`css_selector`	`str \| None`	`None`	—
`target_elements`	`list[str] \| str \| None`	`None`	—
`word_count_threshold`	`int \| None`	`None`	ge: 0
`excluded_tags`	`list[str] \| str \| None`	`None`	—
`exclude_external_links`	`bool \| None`	`None`	—
`exclude_social_media_links`	`bool \| None`	`None`	—
`exclude_domains`	`list[str] \| str \| None`	`None`	—
`exclude_social_media_domains`	`list[str] \| str \| None`	`None`	—
`exclude_external_images`	`bool \| None`	`None`	—
`process_iframes`	`bool \| None`	`None`	—
`remove_overlay_elements`	`bool \| None`	`None`	—
`remove_consent_popups`	`bool \| None`	`None`	—
`flatten_shadow_dom`	`bool \| None`	`None`	—
`remove_forms`	`bool \| None`	`None`	—
`keep_data_attributes`	`bool \| None`	`None`	—
`only_text`	`bool \| None`	`None`	—
`excluded_selector`	`str \| None`	`None`	—
`table_score_threshold`	`float \| None`	`None`	ge: 0
`js_code`	`str \| list[str] \| None`	`None`	—
`js_code_before_wait`	`str \| list[str] \| None`	`None`	—
`js_only`	`bool \| None`	`None`	—
`session_id`	`str \| None`	`None`	—
`kill_session`	`bool \| None`	`None`	—
`wait_for`	`str \| None`	`None`	—
`wait_until`	`Literal[‘commit’, ‘domcontentloaded’, ‘load’, ‘networkidle’] \| None`	`None`	—
`page_timeout`	`int \| None`	`None`	ge: 1000; le: 600000
`delay_before_return_html`	`float \| None`	`None`	ge: 0; le: 60
`mean_delay`	`float \| None`	`None`	ge: 0; le: 60
`max_range`	`float \| None`	`None`	ge: 0; le: 60
`cache_mode`	`Literal[‘enabled’, ‘bypass’, ‘disabled’, ‘read_only’, ‘write_only’] \| None`	`None`	—
`simulate_user`	`bool \| None`	`None`	—
`override_navigator`	`bool \| None`	`None`	—
`magic`	`bool \| None`	`None`	—
`virtual_scroll_config`	`VirtualScrollOptions \| None`	`None`	—
`scan_full_page`	`bool \| None`	`None`	—
`scroll_delay`	`float \| None`	`None`	ge: 0; le: 10.0
`max_scroll_steps`	`int \| None`	`None`	ge: 1; le: 500
`wait_for_images`	`bool \| None`	`None`	—
`viewport_width`	`int \| None`	`None`	ge: 320; le: 3840
`viewport_height`	`int \| None`	`None`	ge: 240; le: 2160
`device_scale_factor`	`float \| None`	`None`	ge: 0.5; le: 4.0
`locale`	`str \| None`	`None`	—
`timezone_id`	`str \| None`	`None`	—
`geolocation`	`GeolocationOptions \| None`	`None`	—
`text_mode`	`bool \| None`	`None`	—
`light_mode`	`bool \| None`	`None`	—
`avoid_ads`	`bool \| None`	`None`	—
`avoid_css`	`bool \| None`	`None`	—
`dispatcher`	`DispatcherOptions \| Literal[‘memory_adaptive’, ‘semaphore’] \| None`	`None`	—
`rate_limiter`	`RateLimiterOptions \| None`	`None`	—
`monitor`	`MonitorOptions \| bool \| None`	`None`	—
`stream`	`bool \| None`	`None`	—
`check_robots_txt`	`bool \| None`	`None`	—
`url_configs`	`list[UrlConfigOverride] \| None`	`None`	—
`url_seeding`	`UrlSeedingOptions \| None`	`None`	—
`domain_mapping`	`DomainMappingOptions \| None`	`None`	—
`deep_crawl`	`DeepCrawlOptions \| None`	`None`	—
`adaptive_crawl`	`AdaptiveCrawlOptions \| None`	`None`	—
`prefetch`	`bool \| None`	`None`	—
`screenshot`	`bool \| None`	`None`	—
`screenshot_full_page`	`bool \| None`	`None`	—
`force_viewport_screenshot`	`bool \| None`	`None`	—
`screenshot_wait_for`	`float \| None`	`None`	ge: 0; le: 60.0
`pdf`	`bool \| None`	`None`	—
`capture_mhtml`	`bool \| None`	`None`	—
`capture_network_requests`	`bool \| None`	`None`	—
`capture_console_messages`	`bool \| None`	`None`	—
`fetch_ssl_certificate`	`bool \| None`	`None`	—
`accept_downloads`	`bool \| None`	`None`	—
`verbose`	`bool \| None`	`None`	—
`concurrency`	`int \| None`	`None`	—
`delay_ms`	`int \| None`	`None`	—
`browser`	`bool \| str \| None`	`None`	—
`cookies`	`list[dict[str, Any]] \| dict[str, Any] \| None`	`None`	—
`user_agent`	`str \| None`	`None`	—
`auto_browser_fallback`	`bool \| None`	`None`	—
`retry_count`	`int \| None`	`None`	—
`retry_backoff_ms`	`int \| None`	`None`	—
`request_timeout_ms`	`int \| None`	`None`	—
`browser_wait_ms`	`int \| None`	`None`	—
`captcha_mode`	`str \| None`	`None`	—
`challenge_wait_seconds`	`float \| None`	`None`	—
`headless`	`bool \| None`	`None`	—
`ignore_https_errors`	`bool \| None`	`None`	—
`extra_headers`	`dict[str, str] \| None`	`None`	—
`allow_root_cross_domain_redirect`	`bool \| None`	`None`	—

markdownConverter

Field	Type / allowed values	Default	Constraints
`content_source`	`Literal[‘raw_html’, ‘cleaned_html’, ‘fit_html’] \| None`	`None`	—
`preserve_links`	`bool \| None`	`None`	—
`preserve_tables`	`bool \| None`	`None`	—
`preserve_code`	`bool \| None`	`None`	—
`ignore_images`	`bool \| None`	`None`	—
`escape_html`	`bool \| None`	`None`	—
`body_width`	`int \| None`	`None`	ge: 0
`skip_internal_links`	`bool \| None`	`None`	—
`include_sup_sub`	`bool \| None`	`None`	—
`citations`	`bool \| None`	`None`	—
`mark_code`	`bool \| None`	`None`	—
`handle_code_in_pre`	`bool \| None`	`None`	—
`options`	`MarkdownOptions \| None`	`None`	—
`content_filter`	`Literal[‘none’, ‘pruning’, ‘bm25’, ‘pruning_bm25’, ‘llm’] \| None`	`None`	—
`strict_filter`	`bool \| None`	`None`	—
`pruning_threshold`	`float \| None`	`None`	ge: 0; le: 1
`pruning_threshold_type`	`Literal[‘fixed’, ‘dynamic’] \| None`	`None`	—
`min_word_threshold`	`int \| None`	`None`	ge: 0
`user_query`	`str \| None`	`None`	—
`bm25_threshold`	`float \| None`	`None`	ge: 0; le: 100
`bm25_language`	`str \| None`	`None`	—
`bm25_case_sensitive`	`bool \| None`	`None`	—
`bm25_use_stemming`	`bool \| None`	`None`	—
`bm25_priority_tags`	`list[str] \| None`	`None`	—
`llm_filter_instruction`	`str \| None`	`None`	—
`llm_filter_provider`	`str \| None`	`None`	—
`llm_filter_model`	`str \| None`	`None`	—
`llm_filter_parallelism`	`int \| None`	`None`	ge: 1
`chunk_token_threshold`	`int \| None`	`None`	ge: 256

llmProcessor

Field	Type / allowed values	Default	Constraints
`enabled`	`bool`	`True`	—
`prompt`	`str`	`''`	—
`provider`	`Literal[‘ollama’, ‘openrouter’, ‘openai_compatible’, ‘openai’, ‘gemini’, ‘anthropic’] \| None`	`None`	—
`model`	`str \| None`	`None`	—
`base_url`	`str \| None`	`None`	—
`api_key`	`str \| None`	`None`	—
`endpoint`	`Literal[‘platform’, ‘own’] \| None`	`None`	—
`temperature`	`float \| None`	`None`	ge: 0; le: 2
`max_tokens`	`int \| None`	`None`	ge: 1; le: 32768
`output_format`	`Literal[‘json’, ‘text’] \| None`	`None`	—
`strict_json`	`bool \| None`	`None`	—
`schema`	`dict[str, Any] \| None`	`None`	—
`relevance_enabled`	`bool \| None`	`None`	—
`max_relevant_pages`	`int \| None`	`None`	gt: 0
`max_context_pages`	`int \| None`	`None`	gt: 0
`max_input_chars`	`int \| None`	`None`	gt: 0
`context_chunk_chars`	`int \| None`	`None`	ge: 500
`context_chunk_overlap_chars`	`int \| None`	`None`	ge: 0
`max_chunks_per_page`	`int \| None`	`None`	gt: 0
`extraction_strategy`	`Literal[‘css’, ‘xpath’, ‘regex’, ‘cosine’, ‘llm’] \| None`	`None`	—
`extraction_type`	`Literal[‘schema’, ‘block’]`	`'schema'`	—
`extraction_input_format`	`Literal[‘markdown’, ‘raw_markdown’, ‘fit_markdown’, ‘html’, ‘cleaned_html’, ‘raw_html’, ‘fit_html’]`	`'markdown'`	—
`extraction_schema`	`dict[str, Any] \| None`	`None`	—
`extraction_output_schema`	`dict[str, Any] \| None`	`None`	—
`extraction_instruction`	`str \| None`	`None`	—
`extraction_generate_schema`	`bool \| None`	`None`	—
`extraction_validate_generated_schema`	`bool \| None`	`None`	—
`extraction_schema_query`	`str \| None`	`None`	—
`extraction_schema_retries`	`int \| None`	`None`	ge: 0; le: 5
`extraction_max_schema_samples`	`int \| None`	`None`	ge: 1; le: 16
`extraction_strict_schema`	`bool \| None`	`None`	—
`extraction_llm_config`	`ExtractionLlmOptions \| None`	`None`	—
`extraction_extra_args`	`dict[str, Any] \| None`	`None`	—
`extraction_parallelism`	`int \| None`	`None`	ge: 1; le: 8
`extraction_adaptive`	`bool \| None`	`None`	—
`extraction_item_retries`	`int \| None`	`None`	ge: 0; le: 5
`extraction_batch_size`	`int \| None`	`None`	ge: 1; le: 20
`extraction_batch_max_chars`	`int \| None`	`None`	ge: 1000; le: 200000
`extraction_batch_max_wait_seconds`	`float \| None`	`None`	ge: 0; le: 600
`extraction_verbose`	`bool \| None`	`None`	—
`extraction_regex_patterns`	`list[str] \| str \| None`	`None`	—
`extraction_regex_custom`	`dict[str, str] \| None`	`None`	—
`extraction_generate_pattern`	`bool \| None`	`None`	—
`extraction_pattern_label`	`str \| None`	`None`	—
`extraction_pattern_query`	`str \| None`	`None`	—
`extraction_pattern_retries`	`int \| None`	`None`	ge: 0; le: 5
`extraction_apply_chunking`	`bool \| None`	`None`	—
`extraction_chunking_strategy`	`Literal[‘regex’, ‘sentence’, ‘topic’, ‘fixed_words’, ‘sliding_window’, ‘characters’] \| None`	`None`	—
`extraction_chunk_patterns`	`list[str] \| str \| None`	`None`	—
`extraction_chunk_language`	`str \| None`	`None`	—
`extraction_chunk_token_threshold`	`int \| None`	`None`	ge: 128
`extraction_chunk_size`	`int \| None`	`None`	ge: 1
`extraction_chunk_overlap`	`int \| None`	`None`	ge: 0
`extraction_chunk_step`	`int \| None`	`None`	ge: 1
`extraction_overlap_rate`	`float \| None`	`None`	ge: 0; le: 0.5
`extraction_topic_w`	`int \| None`	`None`	ge: 5
`extraction_topic_k`	`int \| None`	`None`	ge: 2
`extraction_max_chunks`	`int \| None`	`None`	ge: 1; le: 20000
`extraction_word_token_rate`	`float \| None`	`None`	gt: 0; le: 10
`extraction_tokenizer`	`str`	`'estimate'`	—
`extraction_semantic_filter`	`str \| None`	`None`	—
`extraction_word_count_threshold`	`int \| None`	`None`	ge: 1
`extraction_sim_threshold`	`float \| None`	`None`	ge: 0; le: 1
`extraction_max_dist`	`float \| None`	`None`	ge: 0; le: 2
`extraction_linkage_method`	`Literal[‘ward’, ‘complete’, ‘average’, ‘single’] \| None`	`None`	—
`extraction_top_k`	`int \| None`	`None`	ge: 1
`extraction_max_cosine_chunks`	`int \| None`	`None`	ge: 1; le: 2000
`extraction_embedding_model`	`str \| None`	`None`	—
`extraction_embedding_llm_config`	`ExtractionEmbeddingOptions \| None`	`None`	—
`extraction_allow_embedding_fallback`	`bool \| None`	`None`	—

dataSource

Storage is the platform MongoDB unless crawler-api sets a user store.

Field	Type / allowed values	Default	Constraints
`save_images`	`bool`	`False`	—
`store`	`StoreConnection \| None`	`None`	—
`blob_store`	`BlobStoreConnection \| None`	`None`	—

mongoStore

Documents go to the platform MongoDB unless crawler-api sets a user store.

Field	Type / allowed values	Default	Constraints
`key`	`RecordKeyConfig \| None`	`None`	—
`derive`	`list[DeriveConfig] \| None`	`None`	—
`store`	`StoreConnection \| None`	`None`	—

s3Store

Exactly one of from_field, from_fields and select names where the files are.

Field	Type / allowed values	Default	Constraints
`from_field`	`str \| None`	`None`	—
`from_fields`	`list[str] \| None`	`None`	—
`select`	`S3SelectConfig \| None`	`None`	—
`as`	`str \| None`	`None`	—
`kind`	`Literal[‘image’, ‘pdf’, ‘video’, ‘file’]`	`'image'`	—
`max_files_per_item`	`int`	`1`	ge: 1; le: 20
`max_bytes`	`int`	`10000000`	ge: 1; le: 100000000
`allow_types`	`list[str] \| None`	`None`	—
`blob_store`	`BlobStoreConnection \| None`	`None`	—

forEach

Runs the inline flow once per record of the connected node.

Field	Type / allowed values	Default	Constraints
`flow`	`dict[str, Any]`	`Required`	—
`entry`	`str`	`Required`	—
`bind`	`dict[str, str]`	`Required`	—
`concurrency`	`int`	`4`	ge: 1; le: 8
`max_items`	`int`	`500`	ge: 1; le: 5000
`on_error`	`Literal[‘continue’, ‘stop’]`	`'continue'`	—
`key`	`str \| None`	`None`	—
`item_timeout_seconds`	`int`	`300`	ge: 10; le: 3600

combine

Merges the records several branches made for the same thing into one record per key.

Field	Type / allowed values	Default	Constraints
`key`	`str`	`Required`	—
`fields`	`dict[str, Literal[‘first’, ‘last’, ‘union’, ‘join’]] \| None`	`None`	—
`separator`	`str`	`'\n\n'`	—

query

Field	Type / allowed values	Default	Constraints
`query`	`str \| None`	`None`	—
`prompt`	`str \| None`	`None`	—
`collection`	`Literal[‘raw_pages’, ‘pages’, ‘llm_results’, ‘records’] \| None`	`None`	—
`mongo_filter`	`dict[str, Any] \| None`	`None`	—
`limit`	`int \| None`	`None`	ge: 1; le: 100
`sort`	`list[tuple[str, Literal[1, -1]]] \| None`	`None`	min_length: 1; max_length: 2
`fields`	`list[str] \| None`	`None`	min_length: 1; max_length: 50
`provider`	`str \| None`	`None`	—
`model`	`str \| None`	`None`	—
`base_url`	`str \| None`	`None`	—
`api_key`	`str \| None`	`None`	—
`endpoint`	`Literal[‘platform’, ‘own’] \| None`	`None`	—
`store`	`StoreConnection \| None`	`None`	—
`source_execution_id`	`UUID \| None`	`None`	—
`source_node_ids`	`list[str] \| None`	`None`	min_length: 1

dataTransformer

Field	Type / allowed values	Default	Constraints
`transformer_type`	`Literal[‘cssSelector’, ‘htmlParser’, ‘jsonExtractor’, ‘custom’]`	`'cssSelector'`	—
`css_selector`	`str \| None`	`None`	—
`extract_pattern`	`str \| None`	`None`	—
`transformer_config`	`TransformerOptions \| None`	`None`	—

table

Field	Type / allowed values	Default	Constraints
`columns`	`list[str] \| None`	`None`	—
`rows`	`list[list[Any]] \| None`	`None`	—
`row_count`	`int \| None`	`None`	ge: 0

chart

Field	Type / allowed values	Default	Constraints
`chart_type`	`Literal[‘bar’, ‘line’, ‘pie’, ‘doughnut’, ‘scatter’]`	`'bar'`	—
`labels`	`list[str] \| None`	`None`	—
`datasets`	`list[ChartDataset] \| None`	`None`	—
`label_field`	`str \| None`	`None`	—
`value_field`	`str \| None`	`None`	—

text

Field	Type / allowed values	Default	Constraints
`text`	`str \| None`	`None`	—

VirtualScrollOptions

Field	Type / allowed values	Default	Constraints
`container_selector`	`str`	`'body'`	—
`scroll_count`	`int`	`10`	ge: 1; le: 200
`scroll_by`	`Literal[‘container_height’, ‘page_height’] \| Annotated[float, Field(ge=1, le=100000.0)]`	`'container_height'`	—
`wait_after_scroll`	`float`	`0.5`	ge: 0; le: 30.0

GeolocationOptions

Field	Type / allowed values	Default	Constraints
`latitude`	`float`	`Required`	ge: -90; le: 90
`longitude`	`float`	`Required`	ge: -180; le: 180
`accuracy`	`float`	`0`	ge: 0; le: 100000.0

DispatcherOptions

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘memory_adaptive’, ‘semaphore’]`	`'memory_adaptive'`	—
`memory_threshold_percent`	`float \| None`	`None`	gt: 0; le: 100
`check_interval`	`float \| None`	`None`	gt: 0
`max_session_permit`	`int \| None`	`None`	ge: 1; le: 200
`semaphore_count`	`int \| None`	`None`	ge: 1; le: 200
`memory_wait_timeout`	`float \| None`	`None`	gt: 0

RateLimiterOptions

Field	Type / allowed values	Default	Constraints
`enabled`	`bool \| None`	`None`	—
`base_delay`	`list[float] \| None`	`None`	min_length: 2; max_length: 2
`max_delay`	`float \| None`	`None`	ge: 0
`max_retries`	`int \| None`	`None`	ge: 0; le: 20
`rate_limit_codes`	`list[int] \| None`	`None`	min_length: 1

MonitorOptions

Field	Type / allowed values	Default	Constraints
`enabled`	`bool \| None`	`None`	—
`max_visible_rows`	`int \| None`	`None`	ge: 1; le: 500
`display_mode`	`Literal[‘detailed’, ‘aggregated’] \| None`	`None`	—

UrlConfigOverride

Field	Type / allowed values	Default	Constraints
`url_matcher`	`str \| list[str] \| None`	`None`	—
`match_mode`	`Literal[‘and’, ‘or’]`	`'or'`	—
`config`	`dict[str, Any]`	`{}`	—

UrlSeedingOptions

Field	Type / allowed values	Default	Constraints
`enabled`	`bool`	`True`	—
`include_root`	`bool`	`True`	—
`fill_links`	`bool`	`True`	—
`source`	`Literal[‘sitemap’, ‘cc’, ‘sitemap+cc’]`	`'sitemap'`	—
`pattern`	`str`	`'*'`	—
`extract_head`	`bool`	`False`	—
`live_check`	`bool`	`False`	—
`max_urls`	`int`	`-1`	—
`concurrency`	`int`	`10`	gt: 0
`hits_per_sec`	`float \| None`	`5.0`	ge: 0
`force`	`bool`	`False`	—
`verbose`	`bool`	`False`	—
`query`	`str \| None`	`None`	—
`scoring_method`	`Literal[‘bm25’] \| None`	`None`	—
`score_threshold`	`float \| None`	`None`	ge: 0
`filter_nonsense_urls`	`bool`	`True`	—
`cache_ttl_hours`	`float`	`24.0`	ge: 0
`validate_sitemap_lastmod`	`bool`	`True`	—
`common_crawl_index`	`str \| None`	`None`	—
`max_sitemaps`	`int`	`128`	gt: 0
`head_max_bytes`	`int`	`96000`	gt: 0
`sitemap_max_bytes`	`int`	`50000000`	gt: 0

DomainMappingOptions

Field	Type / allowed values	Default	Constraints
`enabled`	`bool`	`True`	—
`include_root`	`bool`	`True`	—
`fill_links`	`bool`	`True`	—
`source`	`str`	`'sitemap+cc+crt+probe'`	—
`max_urls`	`int`	`-1`	—
`concurrency`	`int`	`50`	gt: 0
`hits_per_sec`	`float \| None`	`10.0`	ge: 0
`force`	`bool`	`False`	—
`extract_head`	`bool`	`True`	—
`filter_nonsense_urls`	`bool`	`True`	—
`detect_soft_404`	`bool`	`True`	—
`query`	`str \| None`	`None`	—
`scoring_method`	`Literal[‘bm25’] \| None`	`'bm25'`	—
`score_threshold`	`float \| None`	`None`	ge: 0
`pattern`	`str`	`'*'`	—
`probe_paths`	`list[str] \| None`	`None`	—
`common_subdomains`	`list[str] \| None`	`None`	—
`use_browser_for_homepage`	`bool`	`False`	—
`verbose`	`bool`	`False`	—
`cache_ttl_hours`	`float`	`24.0`	ge: 0
`dns_timeout`	`float`	`3.0`	ge: 0
`http_timeout`	`float`	`10.0`	ge: 0
`common_crawl_index`	`str \| None`	`None`	—
`max_hosts`	`int`	`200`	gt: 0
`max_sitemaps`	`int`	`128`	gt: 0
`head_max_bytes`	`int`	`96000`	gt: 0

UrlScorer

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘keyword’]`	`'keyword'`	—
`keywords`	`list[str]`	`Required`	—
`weight`	`float`	`1.0`	—

DomainFilter

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘domain’]`	`Required`	—
`allowed_domains`	`list[str]`	`[]`	—
`blocked_domains`	`list[str]`	`[]`	—
`allow_subdomains`	`bool`	`True`	—

UrlPatternFilter

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘url_pattern’]`	`Required`	—
`patterns`	`list[str]`	`Required`	—
`use_regex`	`bool`	`False`	—
`reverse`	`bool`	`False`	—

ContentTypeFilter

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘content_type’]`	`Required`	—
`allowed_types`	`list[str]`	`[]`	—

SeoFilter

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘seo’]`	`Required`	—
`threshold`	`float`	`0.5`	—
`keywords`	`list[str]`	`[]`	—

ContentRelevanceFilter

Field	Type / allowed values	Default	Constraints
`type`	`Literal[‘content_relevance’]`	`Required`	—
`query`	`str`	`''`	—
`threshold`	`float`	`1.0`	—

DeepCrawlOptions

Field	Type / allowed values	Default	Constraints
`strategy`	`Literal[‘bfs’, ‘dfs’, ‘best_first’]`	`'bfs'`	—
`include_external`	`bool`	`False`	—
`concurrency`	`int \| None`	`None`	gt: 0
`score_threshold`	`float \| None`	`None`	—
`url_scorer`	`UrlScorer \| None`	`None`	—
`filters`	`list[DeepCrawlFilter]`	`[]`	—
`resume_state`	`dict[str, Any] \| None`	`None`	—

AdaptiveCrawlOptions

Field	Type / allowed values	Default	Constraints
`enabled`	`bool`	`True`	—
`query`	`str \| None`	`None`	—
`resume_state`	`dict[str, Any] \| None`	`None`	—
`strategy`	`Literal[‘statistical’, ‘embedding’]`	`'statistical'`	—
`confidence_threshold`	`float`	`0.7`	ge: 0; le: 1
`max_pages`	`int`	`20`	gt: 0
`top_k_links`	`int`	`3`	gt: 0
`min_gain_threshold`	`float`	`0.1`	ge: 0; le: 1
`min_pages_before_stop`	`int`	`2`	ge: 1
`saturation_window`	`int`	`3`	ge: 1
`save_state`	`bool`	`False`	—
`embedding_model`	`str`	`'sentence-transformers/all-MiniLM-L6-v2'`	—
`embedding_llm_config`	`dict[str, str] \| None`	`None`	—
`query_llm_config`	`dict[str, str] \| None`	`None`	—
`n_query_variations`	`int`	`10`	ge: 1
`embedding_coverage_radius`	`float`	`0.2`	ge: 0; le: 1
`embedding_k_exp`	`float`	`3.0`	gt: 0
`embedding_min_relative_improvement`	`float`	`0.1`	ge: 0; le: 1
`embedding_validation_min_score`	`float`	`0.3`	ge: 0; le: 1
`embedding_min_confidence_threshold`	`float`	`0.1`	ge: 0; le: 1
`embedding_overlap_threshold`	`float`	`0.85`	ge: 0; le: 1
`embedding_quality_min_confidence`	`float`	`0.7`	ge: 0; le: 1
`embedding_quality_max_confidence`	`float`	`0.95`	ge: 0; le: 1

MarkdownOptions

Field	Type / allowed values	Default	Constraints
`ignore_links`	`bool \| None`	`None`	—
`ignore_images`	`bool \| None`	`None`	—
`escape_html`	`bool \| None`	`None`	—
`body_width`	`int \| None`	`None`	ge: 0
`skip_internal_links`	`bool \| None`	`None`	—
`include_sup_sub`	`bool \| None`	`None`	—
`preserve_tables`	`bool \| None`	`None`	—
`preserve_code`	`bool \| None`	`None`	—
`citations`	`bool \| None`	`None`	—
`mark_code`	`bool \| None`	`None`	—
`handle_code_in_pre`	`bool \| None`	`None`	—

ExtractionLlmOptions

Field	Type / allowed values	Default	Constraints
`provider`	`str \| None`	`None`	—
`model`	`str \| None`	`None`	—
`backend`	`Literal[‘auto’, ‘native’, ‘litellm’]`	`'auto'`	—

ExtractionEmbeddingOptions

Field	Type / allowed values	Default	Constraints
`provider`	`str \| None`	`None`	—
`model`	`str \| None`	`None`	—

DeriveConfig

Field	Type / allowed values	Default	Constraints
`field`	`str`	`Required`	—
`from`	`str \| None`	`None`	—
`normalize`	`Literal[‘host’, ‘exact’]`	`'exact'`	—

RecordKeyConfig

Field	Type / allowed values	Default	Constraints
`field`	`str`	`Required`	—
`from`	`str \| None`	`None`	—
`normalize`	`Literal[‘host’, ‘exact’]`	`'exact'`	—
`match`	`Literal[‘one’, ‘all’]`	`'one'`	—
`lists`	`Literal[‘replace’, ‘union’]`	`'replace'`	—

S3SelectConfig

Field	Type / allowed values	Default	Constraints
`css`	`str`	`Required`	—
`attribute`	`str`	`Required`	—

TransformerOptions

Field	Type / allowed values	Default	Constraints
`source`	`Literal[‘html’, ‘cleaned_html’, ‘fit_html’, ‘llm_result’, ‘query_result’] \| None`	`None`	—
`remove_tags`	`list[str] \| None`	`None`	—
`operation`	`Literal[‘identity’, ‘clean_html’]`	`'identity'`	—
`pattern_type`	`Literal[‘jsonpath’, ‘regex’]`	`'jsonpath'`	—

ChartDataset

Field	Type / allowed values	Default	Constraints
`label`	`str`	`''`	—
`data`	`list[float]`	`Required`	—

NodeInputs

Field	Type / allowed values	Default	Constraints
`raw_pages`	`list[dict[str, Any]] \| None`	`None`	min_length: 1
`pages`	`list[dict[str, Any]] \| None`	`None`	min_length: 1
`llm_result`	`Any`	`None`	—
`query_result`	`dict[str, Any] \| None`	`None`	—
`transformed_data`	`list[dict[str, Any]] \| None`	`None`	—
`final_answer`	`Any`	`None`	—