Web Scraper
AI & KnowledgeExtract structured data from static or JavaScript-rendered web pages
Overview
The Web Scraper fetches a single page over HTTP or through a secured headless browser, then extracts text, HTML, and selector-based fields for downstream workflow steps. Use static mode for simple pages and enable JavaScript rendering for client-rendered sites.
Configuration
| Property | Type | Default | Description |
|---|---|---|---|
urlrequiredvars | string | — | Page URL to fetch. Supports workflow variables. |
method | options | GET | HTTP method. JavaScript rendering supports GET only. |
renderJs | boolean | false | Render the page in an isolated Chromium browser before extraction. |
waitForSelector | string | — | Optional CSS selector to wait for when renderJs is enabled. |
timeout | number | 30000 | Maximum render time in milliseconds (1,000–120,000). |
selectors | json | — | Map of output field names to CSS selectors or structured selector objects. |
Output Variables
Reference these variables in downstream nodes using the {{node.field}} syntax.
{{node-id.url}}stringRequested URL
{{node-id.status_code}}numberHTTP status code
{{node-id.title}}stringDocument title
{{node-id.text}}stringTruncated plain text preview
{{node-id.html}}stringTruncated HTML preview
{{node-id.data}}objectSelector extraction results
{{node-id.final_url}}stringFinal URL after redirects or rendering
{{node-id.rendered_js}}booleanWhether JavaScript rendering was used
Examples
Product price scrape
Static HTML extraction with one selector map.
{
"url": "https://example.com/product/123",
"method": "GET",
"renderJs": false,
"selectors": {
"title": "h1",
"price": { "selector": ".price", "type": "text" }
}
}Tips & Best Practices
- •Use Smart Scan in the editor to suggest selectors with sample values.
- •Run the Test tab before publishing to verify HTTP status and extracted fields.
- •Keep renderJs disabled unless the target page depends on client-side rendering.
Related Nodes
Try Web Scraper in Your Workflow
Create a free account and start building AI-powered workflows with the Web Scraper node.
Open Workflow Editor