Read any page

Read pages, product prices, multi-page listings and forms on sites that have no dedicated Bowmark functions, and know what to do when a page cannot be read.

When get_library has nothing for a site, read static or pre-rendered public pages with bowmark.read. For an interactive search form, login, CAPTCHA, or app that only renders after JavaScript runs, start with web_form_fields or the browser agent.

Read a page

const page = await bowmark.read.page("https://example.com/blog/post-1");
return { title: page.title, preview: page.content.slice(0, 200) };

Read several at once with read.pages, which returns one result per URL in the same order:

const pages = await bowmark.read.pages(["https://example.com/a", "https://example.com/b"]);

Call it through run()

read.page and read.pages work inside a script (the agent's run tool, run(), or POST /v1/run). A session() or a plain typed-client call refuses them with rung_undeclared.

read.page reads what a signed-out visitor sees. For a page behind your own login, pass a saved connection: read.page(url, { connection }). See Signed-in sites.

Options

await bowmark.read.page(url, { strategy: "auto", format: "markdown", maxChars: 200000 });
OptionDefaultWhat it does
format"markdown"Also "text", "cleanHtml" or "html". Markdown keeps every link's URL. "cleanHtml" keeps the page's tags and drops the navigation and footer, so a price stays with the item it belongs to. Use it when the markdown leaves a price or a link unclear
strategy"auto""fetch": fast, never waits on a slow page. "browser": always wait for the page's scripts to run
maxChars200,000Caps content. Raise it when truncated is true
timeoutMs45,000Budget for the whole read. Capped at 55,000
connectionnoneRead signed in with a saved login
headersfalsetrue adds the response headers. It does not send request headers
imagesfalsetrue adds every image the page names
pageMarkersfalseOn a PDF, starts each page's text with <!-- page N -->

A heavy page can take 30 to 55 seconds. For a hard latency budget, like a live chat reply, pass strategy: "fetch": escalationReason then tells you when the page needed more.

A fast read can also come back ok with figures that the page fills in by script after it loads, so they are blank. Check the figures you need. If they are blank, read again with strategy: "browser", which waits for the page's scripts to run. A browser read is slower, and it can still miss values that load later.

What it returns

FieldWhat it tells you
okfalse when the read failed or got a wall, a login page or an empty shell
titleThe page title or first heading, or null
contentThe page in the requested format
format / charsThe format and its length
truncatedtrue when content was cut at maxChars
warningsWhy the result may be thin: a wall, a redirect, a budget
url / requestedUrlWhere the read ended, and the URL you passed
status / errorThe HTTP status, and why the read failed. A 404 means the page is not there, so for a product it is no match, not a failed read
needsLogintrue when the page is a sign-in form
escalationReasonWhy the page needed a fuller load than this call gave it
wall{ vendor, cleared } for a recognised bot wall, else null
jsonParsed JSON, when the URL served JSON
headersResponse headers, with { headers: true }

Check ok and warnings before you trust content. PDFs return their text.

When a warning says navigation or chrome was removed, it can include the quote, booking, or contact link you need. For a navigation-heavy page, retry with { format: "html" } before concluding that link is absent.

What it does not do

  • No POST or caller-supplied headers. read.page always sends a GET and cannot send your Authorization header, cookies or request body to the site. For a search or form, use web_form_fields or the browser agent.
  • One scroll. It scrolls once for lazy content, then reads. It does not click "Load more". See Listings.
  • No structured fields. You parse content yourself. For typed JSON from a page, use the browser agent's outputSchema.
  • No iframes. Read the iframe's own src, or use web_form_fields for a form widget.
  • No charts drawn on a canvas (Power BI, Tableau). Use the browser agent, or find the JSON the chart loads.
  • No signing in. Without a connection, a bank or marketplace URL may return its public marketing page as ok: true. Check the content before you treat it as account data.

Redirects are followed. The final address is in url.

Find the right page

const { urls } = await bowmark.read.urls("https://example.com", { pathPrefix: "/press/" });
const hits = await bowmark.search.web("acme quarterly earnings");

read.urls lists a site's pages from its sitemap and links, as { url, source, depth, lastmod }. search.web searches the open web and returns { source, rank, title, url, snippet, published }.

Start small with read.urls: depth: 1, maxUrls: 300 is a good default for a normal site. depth is capped at 3, maxUrls at 5,000, and maxPages at 100. A large crawl can return a partial run with incomplete.cutOff.by: "memory"; keep the URLs already returned, lower the scope, and split the site by pathPrefix instead of assuming the normal result shape.

A product's price and stock

bowmark.products.getAvailability(url) returns one product page's price and stock without any parsing:

const p = await bowmark.products.getAvailability("https://example-store.com/products/widget");
// { ok, title, price: { amount, currency } | null, availability: "InStock" | "OutOfStock" | … | null,
//   source: "ld+json" | "microdata" | "provider" | "none", url, requestedUrl, status, error, warnings }
// price.amount is in minor units: 1999 means 19.99.

It works on pages that publish product markup and on stores Bowmark covers. When it returns ok: false with source: "none", read the page with read.page and take the price from content. To find a product by name, use bowmark.products.search({ query }).

Read a listing across pages

Markdown keeps links as [text](url), so follow the page's own "Next" link:

const all = [];
const seen = new Set();
let url = "https://example.com/products";

while (url) {
  const page = await bowmark.read.page(url);
  if (!page.ok) break;

  const rows = page.content.match(/Product: (.*?)\n/g) || [];
  const fresh = rows.filter((r) => !seen.has(r));
  if (fresh.length === 0) break; // the site served the same page again
  fresh.forEach((r) => seen.add(r));
  all.push(...fresh);

  url = null;
  for (const [, text, href] of page.content.matchAll(/\[([^\]]*)\]\(([^)]+)\)/g)) {
    if (text.toLowerCase().includes("next") || text === ">") {
      url = new URL(href, page.url).toString();
      break;
    }
  }
}
return all;
  • Check that page 2 is really page 2. Some sites ignore a page parameter and serve page 1 again with no error. Compare stable IDs from the rows, not the title. The seen check above stops the loop when that happens.
  • Put page parameters in the URL (?page=2). read.page has no page or offset option.
  • No next link? The page probably loads more as you scroll. Read it with { format: "html" } and look for embedded JSON (__NEXT_DATA__) or the URL its "load more" calls, then read that URL. json holds it parsed.

Read and fill forms

bowmark.web_form_fields reads the fields a form asks for and fills them in. It finds forms that only appear after a button is clicked, including ones inside an iframe.

const form = await run(`return await bowmark.web_form_fields.getFields("https://example.com/quote")`);
// form.forms: [{ action, method, frameUrl, fields: [{ name, label, type, required, options? }] }]
// also fieldCount, openedWith, multiStep, stepLabel, needsLogin, readFailure, warnings

const step = await run(`return await bowmark.web_form_fields.fillForm("https://example.com/quote", {
  "Postcode": "4217", // keyed by a field's name, or a word or two of its label
}, { advance: true })`);
// step.filled, step.notFound, step.advanced, step.submitted, step.resultContent, step.warnings

Call web_form_fields through run() as shown. A plain client call or session() call is refused with rung_undeclared.

getFields returns { url, title, forms, fieldCount, warnings, needsLogin, readFailure, openedWith, multiStep, stepLabel }. fillForm returns { url, filled, notFound, advanced, submitted, autocompleteSelected, resultContent, warnings, needsLogin, readFailure }. autocompleteSelected records the suggestion chosen for an address or similar lookup. Check it when a site requires a choice from its own dropdown, not just typed text.

Third argumentWhat happens
{} or noneFills the fields. Nothing is sent
{ advance: true }Clicks "Next" or "Continue"
{ submit: true }Clicks the button that sends the form

Pass { egress: "default" }, { egress: "us-datacenter" }, or { egress: "static-residential" } as the options argument when needed. Keep the default unless the same site consistently refuses it; static-residential is the option to try then.

It solves common CAPTCHAs. One it cannot solve returns needs_input with a watchUrl for a person. Forms behind a login report needsLogin: true: use the browser agent for those.

The form you want is missing

getFields returns every form it found. Pick the one whose fields match. If a page's only form is a newsletter signup, the form you want may load later. Ask again:

await bowmark.web_form_fields.getFields(url, { strategy: "browser" });  // wait for the page
await bowmark.web_form_fields.getFields(url, { open: "Get a quote" }); // click to open it

open takes the button's visible text or a CSS selector. openedWith says what was clicked.

Warning starts withTry
nothing on the page matched the "open" hintCopy the exact button text, or use a CSS selector
the "open" hint … matched something on the page, but it could not be clickedA more specific selector. If a banner must be dismissed first, use the browser agent
clicked what matched …, but no form appearedThe form may be on another page. Read where the button links
the "open" hint … was never triedPass a larger timeoutMs (default 30,000)
the page answered HTTP 403, a wall, or a shellRetry once with a browser strategy. If it remains unreadable, use the browser agent

Forms found are still in forms whatever the warning says.

Fill a wizard

One fillForm call is one step. Pass advance: true on each step that ends in "Next", and submit: true on the last. If advance finds no next button, that step is the last one.

  • Pass the same open you used for getFields, so both see the same form.
  • notFound names any key that matched nothing.
  • resultContent is plain text (up to 4,000 characters) from the page the click led to: a price, a match, a validation message. Search it for the value you expect.

It fills real inputs, selects and text areas. A page of clickable cards with no form gives it nothing to fill. Read it with read.page, or click through it with the browser agent.

If a multi-step form needs interaction the form tool cannot complete, use the browser agent for that step. It is billed per task, so start with the form tool when the page has ordinary fields.

The page could not be read

readFailure is set when the page was never read (a dead domain, a 404, a 500):

const form = await bowmark.web_form_fields.getFields(url);
if (form.readFailure) {
  // readFailure.httpStatus: what the site answered, or null if it never answered
  return { ok: false, reason: form.readFailure.detail };
}

A 404 or unreachable domain will not change on retry. A 5xx or 429 might.

When a page cannot be read

Read the result before you retry:

What you seeWhat to do
429, 5xx, or retry: "later"Retry once after the wait
needsLogin: trueDon't retry the same URL. Use a saved connection, or find a public page with the same data (a product page instead of a search listing)
ok: false with wall: { cleared: false }Read it once more. A site can refuse one read and pass the next. If that read is walled too, use the browser agent once, or another source
status: 0See Is it the site, or is it Bowmark?
A CAPTCHAread.page solves common ones before it answers

If the browser agent also returns blocked:, Bowmark cannot read that site today. Look for the same data on the maker's site, another retailer, or a site get_library covers. Or ask for the site.

A failed or walled read is still billed for what it used. See Pricing.