Web Crawler

Fetch a public web page and extract its content.

Web Crawler fetches a public URL and returns its content as both plain text and Markdown. It is how a workflow reads a page that isn't yours.

How it works

• Input resolution — takes a url and an optional instruction as direct configuration.

• Processing — issues an HTTP GET to the URL, parses the HTML response to extract the main textual content, and formats it as both plain text and Markdown.

• Execution — blocking. Waits for the page to be fetched, extracted, and parsed.

• Response — success returns extractedContent and markdown; failure returns an error message and status code.

Worked example: monitoring competitor pricing

You need to check a product's price on a competitor's public site regularly, rather than having someone look every Monday.

The problem. Browsing a page and copying a number out of it is inefficient, easy to get wrong, and nobody remembers to do it.

The approach. A Web Crawler skill fetches the page, and the instruction field narrows what comes back — "Extract the pricing table for the Professional plan" rather than the whole page including the navigation and the cookie banner.

Configuration:

{
  "url": "https://www.example.com/news/article-123",
  "instruction": "Extract the main article text and headline."
}

Output:

{
  "extractedContent": "Headline: Latest News. This is the main body of the article...",
  "markdown": "# Latest News\n\nThis is the main body of the article...",
  "statusCode": 200,
  "error": null
}

Two formats, different jobs. markdown keeps the structure — headings, lists, emphasis — which usually helps a model understand what it's reading. extractedContent is flat text, better when you're matching or extracting rather than reasoning. Pick per use, and don't pass both to a model just because both are there.

Configuration reference

Field	Type	Required	Description
url	string	✅	Absolute URL of the page to crawl.
instruction	string	—	Guides extraction, e.g. "Extract only the main article text".
name	string	—	Display name for this skill instance.
description	string	—	Longer description shown in the designer.

Output

The output port (crawledContent) always conforms to:

Field	Type	Always	Description
extractedContent	string	✅	Plain text extracted from the page.
markdown	string	✅	Markdown-formatted content from the page.
statusCode	number	✅	HTTP-like status for the operation.
error	string | null	—	Error message on failure; null on success.

Errors

Status	Meaning
200	Successfully extracted data from the page.
400	Bad request — malformed url, or a missing/invalid instruction.
500	Internal error — failed to parse or extract from the HTTP response.

The skill reads public pages. It has no session, so anything behind a login, a paywall, or a bot check will not come back — and the failure may look like a successful fetch of a login page rather than an error. Check what you actually got, not just the status code.

Testing in isolation

• Path — /skill-runtime/workflows/nodes/WebCrawler/execute

• Method — POST

---

To add this skill to an agent, see Adding a Skill to the Agent. For search rather than fetching a known URL, see the MCP catalog under MCP Tools.