Web research

Our crawler, across your workspace.

Paste a public URL in the web reader, ask about that URL in a connected chat, call the MCP tool or use the CLI. These surfaces use the same PersistMemory crawler. Public reading and extraction make no AI model or paid crawler call.

Open web reader Open isolated browser Monitor API documentation

Read a website

Open Web scraper in the dashboard and paste the public HTTP(S) URL.
Choose one page or a crawl of up to 25 page attempts on the same site, with up to four links of depth. Enable Check the latest content to validate the website now.
Under Reading options, choose automatic rendering, HTTP only, the isolated browser, or Scrapling stealth mode. Optional CSS selectors include or exclude content; enter one selector per line, up to eight per list.
Review the title, source, reading method, timestamps, structured data and any unfinished links. Tables, sections and publisher-supplied JSON-LD are available in the extraction JSON. Expand the text, download it or save it to an editable Space. Saving starts memory extraction and may use AI.

pm web read https://example.com/docs --fresh
pm web read https://example.com/docs --render always --include main --exclude .sidebar

Search can also read its results: enable “Read the search results” to collect up to 10 pages across sources within 90 seconds. Search snippets remain separate from downloaded content. The reader follows pagination and entity-detail links, reports partial coverage, and never claims a complete ranking.

The API is POST /api/v1/web/read with { url, maxPages?, maxDepth?, goal?, concurrency?, maxDurationMs?, maxTextChars?, fresh?, render?, includeSelectors?, excludeSelectors? }. The authenticated account owns its cache. MCP and connected chat use read_public_page with the same options. In chat, supply the exact URL in your current message.

Save media originals

Choose Save media to bucket in the web reader or call archive_public_site in chat or MCP. The crawler discovers image sources, posters, video, audio, PDFs and document links. Each file has a stored, failed, skipped or unsupported outcome. Stored means the validated original bytes reached your configured private bucket; no OCR, transcription or memory extraction is implied.

The default is 20 attempted files, 25 MiB per file, 100 MiB total and two minutes per run. The saved archive JSON contains page text, source links and file keys. Keep its archive key to reopen it or continue unfinished downloads. Previously stored files are preserved. A streaming manifest or embedded player link is not a downloadable video original.

pm web archive https://example.com --render stealth --pages 3 --files 20
pm web archive-get <archive-key>
pm web archive https://example.com --resume <archive-key>
pm web archive-download <archive-key> --file-key <file-key>

API: POST /api/v1/web/archive requires read and write access. Use POST /api/v1/web/archives/read to inspect a receipt and POST /api/v1/web/archives/download for an expiring original or JSON download link. These read endpoints require the owning account. Raw files remain in the configured S3, R2 or local store.

Freshness and change detection

Ordinary reading may reuse content for five minutes. Fresh reading contacts the website; a conditional request can reuse content only after the site confirms it is unchanged. fetchedAt records when the content was obtained; checkedAt records validation, and revalidated identifies a conditional confirmation.

API watches always request fresh validation when created or checked manually or on schedule. They compare the first page's normalized extracted content with the reviewed baseline. Main content is preferred, but unusual sites can still include navigation. One reading covers at most 12,000 characters; a truncation notice means changes beyond the limit are not monitored.

Interact with a website

Open the isolated browser from the web reader. Start a session or paste an existing session ID from chat, MCP or the CLI. Inspect its screenshot, visible text and controls. Choose an observed control to click, fill, select an option, change its checked state, or press a supported key. Scrolling and same-origin embedded forms are supported. Navigation, extraction and screenshots are also available. Each action uses the observed version. If a dispatched action fails, its receipt is marked uncertain and further mutations are blocked while the temporary session remains inspectable. This does not confirm website submission.

The MCP tools are start_browser, inspect_browser, act_in_browser and close_browser. Browser mutations require write access. Connected chat proposes an action before executing it under the existing confirmation flow. A proposal is not a completed website action.

pm web browser start https://example.com
pm web browser get <session-id>
pm web browser click <session-id> --version <observed-version> --element <observed-element-id>
pm web browser screenshot <session-id> --version <observed-version>
pm web browser close <session-id>

Passwords and other sensitive fields belong in the explicit manual dashboard input, never chat or CLI arguments. Browser state is temporary and account scoped. Closing or expiry removes its session state. Screenshots can be inspected in the dashboard; they are not inserted as large image payloads in model tool text.

Access and limits

Public reading rejects private network addresses, credential-bearing URLs and unsupported ports. Optional Scrapling stealth rendering attempts supported Cloudflare challenges automatically within a fixed time limit. It does not wait for human input. A remaining challenge is reported as blocked; access is not guaranteed. Blocked or empty pages are reported as unavailable evidence; a failed watch check preserves its successful baseline.

A direct URL uses our crawler first. Topic search is a separate discovery capability: the deployment's existing TinyFish, Firecrawl and Brave search adapters may consume their provider quotas. A supplied URL task does not silently invoke those adapters unless the person explicitly requests web search. Search results and website text remain untrusted source material.

Configure the isolated browser

The operator deploys the browser service separately from the API and worker. Configure both CRAWLER_RENDERER_URL and CRAWLER_RENDERER_TOKEN for the API, MCP and worker; the URL identifies your own service and the token authenticates its control plane. Do not expose this token in client code or a chat.

The optional browser image build enables pinned Scrapling dependencies with CRAWLER_INSTALL_STEALTH=true. Set CRAWLER_STEALTH_PYTHON to its installed interpreter inside the browser service. Without the browser service, public HTTP extraction remains available. Explicit browser sessions and rendering requests report that rendering is not configured. Deployment and browser infrastructure can incur hosting costs even though reading uses no paid crawler provider.