Read any page
Read pages, product prices, multi-page listings and forms on sites that have no dedicated Bowmark functions, and know what to do when a page cannot be read.
When get_library has nothing for a site, read static or pre-rendered public pages with
bowmark.read. For an interactive search form, login, CAPTCHA, or app that only renders after
JavaScript runs, start with web_form_fields or the browser agent.
Read a page
const page = await bowmark.read.page("https://example.com/blog/post-1");
return { title: page.title, preview: page.content.slice(0, 200) };Read several at once with read.pages, which returns one result per URL in the same order:
const pages = await bowmark.read.pages(["https://example.com/a", "https://example.com/b"]);Call it through run()
read.page and read.pages work inside a script (the agent's run tool, run(), or
POST /v1/run). A session() or a plain typed-client call refuses them with
rung_undeclared.
read.page reads what a signed-out visitor sees. For a page behind your own login, pass a
saved connection: read.page(url, { connection }). See
Signed-in sites.
Options
await bowmark.read.page(url, { strategy: "auto", format: "markdown", maxChars: 200000 });| Option | Default | What it does |
|---|---|---|
format | "markdown" | Also "text", "cleanHtml" or "html". Markdown keeps every link's URL. "cleanHtml" keeps the page's tags and drops the navigation and footer, so a price stays with the item it belongs to. Use it when the markdown leaves a price or a link unclear |
strategy | "auto" | "fetch": fast, never waits on a slow page. "browser": always wait for the page's scripts to run |
maxChars | 200,000 | Caps content. Raise it when truncated is true |
timeoutMs | 45,000 | Budget for the whole read. Capped at 55,000 |
connection | none | Read signed in with a saved login |
headers | false | true adds the response headers. It does not send request headers |
images | false | true adds every image the page names |
pageMarkers | false | On a PDF, starts each page's text with <!-- page N --> |
A heavy page can take 30 to 55 seconds. For a hard latency budget, like a live chat reply, pass
strategy: "fetch": escalationReason then tells you when the page needed more.
A fast read can also come back ok with figures that the page fills in by script after it loads,
so they are blank. Check the figures you need. If they are blank, read again with
strategy: "browser", which waits for the page's scripts to run. A browser read is slower, and it
can still miss values that load later.
What it returns
| Field | What it tells you |
|---|---|
ok | false when the read failed or got a wall, a login page or an empty shell |
title | The page title or first heading, or null |
content | The page in the requested format |
format / chars | The format and its length |
truncated | true when content was cut at maxChars |
warnings | Why the result may be thin: a wall, a redirect, a budget |
url / requestedUrl | Where the read ended, and the URL you passed |
status / error | The HTTP status, and why the read failed. A 404 means the page is not there, so for a product it is no match, not a failed read |
needsLogin | true when the page is a sign-in form |
escalationReason | Why the page needed a fuller load than this call gave it |
wall | { vendor, cleared } for a recognised bot wall, else null |
json | Parsed JSON, when the URL served JSON |
headers | Response headers, with { headers: true } |
Check ok and warnings before you trust content. PDFs return their text.
When a warning says navigation or chrome was removed, it can include the quote, booking, or
contact link you need. For a navigation-heavy page, retry with { format: "html" } before
concluding that link is absent.
What it does not do
- No POST or caller-supplied headers.
read.pagealways sends a GET and cannot send yourAuthorizationheader, cookies or request body to the site. For a search or form, useweb_form_fieldsor the browser agent. - One scroll. It scrolls once for lazy content, then reads. It does not click "Load more". See Listings.
- No structured fields. You parse
contentyourself. For typed JSON from a page, use the browser agent'soutputSchema. - No iframes. Read the iframe's own
src, or useweb_form_fieldsfor a form widget. - No charts drawn on a canvas (Power BI, Tableau). Use the browser agent, or find the JSON the chart loads.
- No signing in. Without a connection, a bank or marketplace URL may return its public
marketing page as
ok: true. Check the content before you treat it as account data.
Redirects are followed. The final address is in url.
Find the right page
const { urls } = await bowmark.read.urls("https://example.com", { pathPrefix: "/press/" });
const hits = await bowmark.search.web("acme quarterly earnings");read.urls lists a site's pages from its sitemap and links, as { url, source, depth, lastmod }.
search.web searches the open web and returns { source, rank, title, url, snippet, published }.
Start small with read.urls: depth: 1, maxUrls: 300 is a good default for a normal site.
depth is capped at 3, maxUrls at 5,000, and maxPages at 100. A large crawl can return a
partial run with incomplete.cutOff.by: "memory"; keep the URLs already returned, lower the
scope, and split the site by pathPrefix instead of assuming the normal result shape.
A product's price and stock
bowmark.products.getAvailability(url) returns one product page's price and stock without any
parsing:
const p = await bowmark.products.getAvailability("https://example-store.com/products/widget");
// { ok, title, price: { amount, currency } | null, availability: "InStock" | "OutOfStock" | … | null,
// source: "ld+json" | "microdata" | "provider" | "none", url, requestedUrl, status, error, warnings }
// price.amount is in minor units: 1999 means 19.99.It works on pages that publish product markup and on stores Bowmark covers. When it returns
ok: false with source: "none", read the page with read.page and take the price from
content. To find a product by name, use bowmark.products.search({ query }).
Read a listing across pages
Markdown keeps links as [text](url), so follow the page's own "Next" link:
const all = [];
const seen = new Set();
let url = "https://example.com/products";
while (url) {
const page = await bowmark.read.page(url);
if (!page.ok) break;
const rows = page.content.match(/Product: (.*?)\n/g) || [];
const fresh = rows.filter((r) => !seen.has(r));
if (fresh.length === 0) break; // the site served the same page again
fresh.forEach((r) => seen.add(r));
all.push(...fresh);
url = null;
for (const [, text, href] of page.content.matchAll(/\[([^\]]*)\]\(([^)]+)\)/g)) {
if (text.toLowerCase().includes("next") || text === ">") {
url = new URL(href, page.url).toString();
break;
}
}
}
return all;- Check that page 2 is really page 2. Some sites ignore a page parameter and serve page 1
again with no error. Compare stable IDs from the rows, not the title. The
seencheck above stops the loop when that happens. - Put page parameters in the URL (
?page=2).read.pagehas no page or offset option. - No next link? The page probably loads more as you scroll. Read it with
{ format: "html" }and look for embedded JSON (__NEXT_DATA__) or the URL its "load more" calls, then read that URL.jsonholds it parsed.
Read and fill forms
bowmark.web_form_fields reads the fields a form asks for and fills them in. It finds forms
that only appear after a button is clicked, including ones inside an iframe.
const form = await run(`return await bowmark.web_form_fields.getFields("https://example.com/quote")`);
// form.forms: [{ action, method, frameUrl, fields: [{ name, label, type, required, options? }] }]
// also fieldCount, openedWith, multiStep, stepLabel, needsLogin, readFailure, warnings
const step = await run(`return await bowmark.web_form_fields.fillForm("https://example.com/quote", {
"Postcode": "4217", // keyed by a field's name, or a word or two of its label
}, { advance: true })`);
// step.filled, step.notFound, step.advanced, step.submitted, step.resultContent, step.warningsCall web_form_fields through run() as shown. A plain client call or session() call is
refused with rung_undeclared.
getFields returns { url, title, forms, fieldCount, warnings, needsLogin, readFailure, openedWith, multiStep, stepLabel }. fillForm returns { url, filled, notFound, advanced, submitted, autocompleteSelected, resultContent, warnings, needsLogin, readFailure }.
autocompleteSelected records the suggestion chosen for an address or similar lookup. Check it
when a site requires a choice from its own dropdown, not just typed text.
| Third argument | What happens |
|---|---|
{} or none | Fills the fields. Nothing is sent |
{ advance: true } | Clicks "Next" or "Continue" |
{ submit: true } | Clicks the button that sends the form |
Pass { egress: "default" }, { egress: "us-datacenter" }, or
{ egress: "static-residential" } as the options argument when needed. Keep the default
unless the same site consistently refuses it; static-residential is the option to try then.
It solves common CAPTCHAs. One it cannot solve returns needs_input with a watchUrl for a
person. Forms behind a login report needsLogin: true: use the browser agent for those.
The form you want is missing
getFields returns every form it found. Pick the one whose fields match. If a page's only form
is a newsletter signup, the form you want may load later. Ask again:
await bowmark.web_form_fields.getFields(url, { strategy: "browser" }); // wait for the page
await bowmark.web_form_fields.getFields(url, { open: "Get a quote" }); // click to open itopen takes the button's visible text or a CSS selector. openedWith says what was clicked.
| Warning starts with | Try |
|---|---|
nothing on the page matched the "open" hint | Copy the exact button text, or use a CSS selector |
the "open" hint … matched something on the page, but it could not be clicked | A more specific selector. If a banner must be dismissed first, use the browser agent |
clicked what matched …, but no form appeared | The form may be on another page. Read where the button links |
the "open" hint … was never tried | Pass a larger timeoutMs (default 30,000) |
the page answered HTTP 403, a wall, or a shell | Retry once with a browser strategy. If it remains unreadable, use the browser agent |
Forms found are still in forms whatever the warning says.
Fill a wizard
One fillForm call is one step. Pass advance: true on each step that ends in "Next", and
submit: true on the last. If advance finds no next button, that step is the last one.
- Pass the same
openyou used forgetFields, so both see the same form. notFoundnames any key that matched nothing.resultContentis plain text (up to 4,000 characters) from the page the click led to: a price, a match, a validation message. Search it for the value you expect.
It fills real inputs, selects and text areas. A page of clickable cards with no form gives it
nothing to fill. Read it with read.page, or click through it with the browser agent.
If a multi-step form needs interaction the form tool cannot complete, use the browser agent for that step. It is billed per task, so start with the form tool when the page has ordinary fields.
The page could not be read
readFailure is set when the page was never read (a dead domain, a 404, a 500):
const form = await bowmark.web_form_fields.getFields(url);
if (form.readFailure) {
// readFailure.httpStatus: what the site answered, or null if it never answered
return { ok: false, reason: form.readFailure.detail };
}A 404 or unreachable domain will not change on retry. A 5xx or 429 might.
When a page cannot be read
Read the result before you retry:
| What you see | What to do |
|---|---|
429, 5xx, or retry: "later" | Retry once after the wait |
needsLogin: true | Don't retry the same URL. Use a saved connection, or find a public page with the same data (a product page instead of a search listing) |
ok: false with wall: { cleared: false } | Read it once more. A site can refuse one read and pass the next. If that read is walled too, use the browser agent once, or another source |
status: 0 | See Is it the site, or is it Bowmark? |
| A CAPTCHA | read.page solves common ones before it answers |
If the browser agent also returns blocked:, Bowmark cannot read that site today. Look for the
same data on the maker's site, another retailer, or a site get_library covers. Or
ask for the site.
A failed or walled read is still billed for what it used. See Pricing.
Signed-in sites
Let your AI agent log in to websites as your user. The user signs in once on the real site, and later calls act as their account without the agent ever seeing a password.
Browser agent
Hand a task that needs clicking, a form or a sign-in to a hosted browser agent, then poll it from later runs until it finishes.