# Read any page (/docs/read-any-page)


When `get_library` has nothing for a site, read static or pre-rendered public pages with
`bowmark.read`. For an interactive search form, login, CAPTCHA, or app that only renders after
JavaScript runs, start with `web_form_fields` or the [browser agent](/docs/browser-agent).

## Read a page

```js
const page = await bowmark.read.page("https://example.com/blog/post-1");
return { title: page.title, preview: page.content.slice(0, 200) };
```

Read several at once with `read.pages`, which returns one result per URL in the same order:

```js
const pages = await bowmark.read.pages(["https://example.com/a", "https://example.com/b"]);
```

  `read.page` and `read.pages` work inside a script (the agent's `run` tool, `run()`, or
  `POST /v1/run`). A `session()` or a plain typed-client call refuses them with
  `rung_undeclared`.

`read.page` reads what a signed-out visitor sees. For a page behind your own login, pass a
saved connection: `read.page(url, { connection })`. See
[Signed-in sites](/docs/signed-in-sites#sites-without-their-own-functions).

### Options

```js
await bowmark.read.page(url, { strategy: "auto", format: "markdown", maxChars: 200000 });
```

| Option        | Default      | What it does                                                                                                                                                                                                                                                |
| ------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `format`      | `"markdown"` | Also `"text"`, `"cleanHtml"` or `"html"`. Markdown keeps every link's URL. `"cleanHtml"` keeps the page's tags and drops the navigation and footer, so a price stays with the item it belongs to. Use it when the markdown leaves a price or a link unclear |
| `strategy`    | `"auto"`     | `"fetch"`: fast, never waits on a slow page. `"browser"`: always wait for the page's scripts to run                                                                                                                                                         |
| `maxChars`    | 200,000      | Caps `content`. Raise it when `truncated` is `true`                                                                                                                                                                                                         |
| `timeoutMs`   | 45,000       | Budget for the whole read. Capped at 55,000                                                                                                                                                                                                                 |
| `connection`  | none         | Read signed in with a saved login                                                                                                                                                                                                                           |
| `headers`     | `false`      | `true` adds the response headers. It does not send request headers                                                                                                                                                                                          |
| `images`      | `false`      | `true` adds every image the page names                                                                                                                                                                                                                      |
| `pageMarkers` | `false`      | On a PDF, starts each page's text with `<!-- page N -->`                                                                                                                                                                                                    |

A heavy page can take 30 to 55 seconds. For a hard latency budget, like a live chat reply, pass
`strategy: "fetch"`: `escalationReason` then tells you when the page needed more.

A fast read can also come back `ok` with figures that the page fills in by script after it loads,
so they are blank. Check the figures you need. If they are blank, read again with
`strategy: "browser"`, which waits for the page's scripts to run. A browser read is slower, and it
can still miss values that load later.

### What it returns

| Field                  | What it tells you                                                                                                                 |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `ok`                   | `false` when the read failed or got a wall, a login page or an empty shell                                                        |
| `title`                | The page title or first heading, or `null`                                                                                        |
| `content`              | The page in the requested format                                                                                                  |
| `format` / `chars`     | The format and its length                                                                                                         |
| `truncated`            | `true` when `content` was cut at `maxChars`                                                                                       |
| `warnings`             | Why the result may be thin: a wall, a redirect, a budget                                                                          |
| `url` / `requestedUrl` | Where the read ended, and the URL you passed                                                                                      |
| `status` / `error`     | The HTTP status, and why the read failed. A `404` means the page is not there, so for a product it is no match, not a failed read |
| `needsLogin`           | `true` when the page is a sign-in form                                                                                            |
| `escalationReason`     | Why the page needed a fuller load than this call gave it                                                                          |
| `wall`                 | `{ vendor, cleared }` for a recognised bot wall, else `null`                                                                      |
| `json`                 | Parsed JSON, when the URL served JSON                                                                                             |
| `headers`              | Response headers, with `{ headers: true }`                                                                                        |

Check `ok` and `warnings` before you trust `content`. PDFs return their text.

When a warning says navigation or chrome was removed, it can include the quote, booking, or
contact link you need. For a navigation-heavy page, retry with `{ format: "html" }` before
concluding that link is absent.

### What it does not do

* **No POST or caller-supplied headers.** `read.page` always sends a GET and cannot send your
  `Authorization` header, cookies or request body to the site. For a search or form, use
  [`web_form_fields`](#read-and-fill-forms) or the [browser agent](/docs/browser-agent).
* **One scroll.** It scrolls once for lazy content, then reads. It does not click "Load more".
  See [Listings](#read-a-listing-across-pages).
* **No structured fields.** You parse `content` yourself. For typed JSON from a page, use the
  browser agent's `outputSchema`.
* **No iframes.** Read the iframe's own `src`, or use `web_form_fields` for a form widget.
* **No charts drawn on a canvas** (Power BI, Tableau). Use the browser agent, or find the JSON
  the chart loads.
* **No signing in.** Without a connection, a bank or marketplace URL may return its public
  marketing page as `ok: true`. Check the content before you treat it as account data.

Redirects are followed. The final address is in `url`.

## Find the right page

```js
const { urls } = await bowmark.read.urls("https://example.com", { pathPrefix: "/press/" });
const hits = await bowmark.search.web("acme quarterly earnings");
```

`read.urls` lists a site's pages from its sitemap and links, as `{ url, source, depth, lastmod }`.
`search.web` searches the open web and returns `{ source, rank, title, url, snippet, published }`.

Start small with `read.urls`: `depth: 1, maxUrls: 300` is a good default for a normal site.
`depth` is capped at 3, `maxUrls` at 5,000, and `maxPages` at 100. A large crawl can return a
partial run with `incomplete.cutOff.by: "memory"`; keep the URLs already returned, lower the
scope, and split the site by `pathPrefix` instead of assuming the normal result shape.

## A product's price and stock

`bowmark.products.getAvailability(url)` returns one product page's price and stock without any
parsing:

```js
const p = await bowmark.products.getAvailability("https://example-store.com/products/widget");
// { ok, title, price: { amount, currency } | null, availability: "InStock" | "OutOfStock" | … | null,
//   source: "ld+json" | "microdata" | "provider" | "none", url, requestedUrl, status, error, warnings }
// price.amount is in minor units: 1999 means 19.99.
```

It works on pages that publish product markup and on stores Bowmark covers. When it returns
`ok: false` with `source: "none"`, read the page with `read.page` and take the price from
`content`. To find a product by name, use `bowmark.products.search({ query })`.

## Read a listing across pages

Markdown keeps links as `[text](url)`, so follow the page's own "Next" link:

```js
const all = [];
const seen = new Set();
let url = "https://example.com/products";

while (url) {
  const page = await bowmark.read.page(url);
  if (!page.ok) break;

  const rows = page.content.match(/Product: (.*?)\n/g) || [];
  const fresh = rows.filter((r) => !seen.has(r));
  if (fresh.length === 0) break; // the site served the same page again
  fresh.forEach((r) => seen.add(r));
  all.push(...fresh);

  url = null;
  for (const [, text, href] of page.content.matchAll(/\[([^\]]*)\]\(([^)]+)\)/g)) {
    if (text.toLowerCase().includes("next") || text === ">") {
      url = new URL(href, page.url).toString();
      break;
    }
  }
}
return all;
```

* **Check that page 2 is really page 2.** Some sites ignore a page parameter and serve page 1
  again with no error. Compare stable IDs from the rows, not the title. The `seen` check above
  stops the loop when that happens.
* **Put page parameters in the URL** (`?page=2`). `read.page` has no page or offset option.
* **No next link?** The page probably loads more as you scroll. Read it with
  `{ format: "html" }` and look for embedded JSON (`__NEXT_DATA__`) or the URL its "load more"
  calls, then read that URL. `json` holds it parsed.

## Read and fill forms

`bowmark.web_form_fields` reads the fields a form asks for and fills them in. It finds forms
that only appear after a button is clicked, including ones inside an iframe.

```js
const form = await run(`return await bowmark.web_form_fields.getFields("https://example.com/quote")`);
// form.forms: [{ action, method, frameUrl, fields: [{ name, label, type, required, options? }] }]
// also fieldCount, openedWith, multiStep, stepLabel, needsLogin, readFailure, warnings

const step = await run(`return await bowmark.web_form_fields.fillForm("https://example.com/quote", {
  "Postcode": "4217", // keyed by a field's name, or a word or two of its label
}, { advance: true })`);
// step.filled, step.notFound, step.advanced, step.submitted, step.resultContent, step.warnings
```

Call `web_form_fields` through `run()` as shown. A plain client call or `session()` call is
refused with `rung_undeclared`.

`getFields` returns `{ url, title, forms, fieldCount, warnings, needsLogin, readFailure,
openedWith, multiStep, stepLabel }`. `fillForm` returns `{ url, filled, notFound, advanced,
submitted, autocompleteSelected, resultContent, warnings, needsLogin, readFailure }`.
`autocompleteSelected` records the suggestion chosen for an address or similar lookup. Check it
when a site requires a choice from its own dropdown, not just typed text.

| Third argument      | What happens                          |
| ------------------- | ------------------------------------- |
| `{}` or none        | Fills the fields. Nothing is sent     |
| `{ advance: true }` | Clicks "Next" or "Continue"           |
| `{ submit: true }`  | Clicks the button that sends the form |

Pass `{ egress: "default" }`, `{ egress: "us-datacenter" }`, or
`{ egress: "static-residential" }` as the options argument when needed. Keep the default
unless the same site consistently refuses it; `static-residential` is the option to try then.

It solves common CAPTCHAs. One it cannot solve returns `needs_input` with a `watchUrl` for a
person. Forms behind a login report `needsLogin: true`: use the browser agent for those.

### The form you want is missing

`getFields` returns every form it found. Pick the one whose fields match. If a page's only form
is a newsletter signup, the form you want may load later. Ask again:

```js
await bowmark.web_form_fields.getFields(url, { strategy: "browser" });  // wait for the page
await bowmark.web_form_fields.getFields(url, { open: "Get a quote" }); // click to open it
```

`open` takes the button's visible text or a CSS selector. `openedWith` says what was clicked.

| Warning starts with                                                            | Try                                                                                                        |
| ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `nothing on the page matched the "open" hint`                                  | Copy the exact button text, or use a CSS selector                                                          |
| `the "open" hint … matched something on the page, but it could not be clicked` | A more specific selector. If a banner must be dismissed first, use the browser agent                       |
| `clicked what matched …, but no form appeared`                                 | The form may be on another page. Read where the button links                                               |
| `the "open" hint … was never tried`                                            | Pass a larger `timeoutMs` (default 30,000)                                                                 |
| `the page answered HTTP 403`, a wall, or a shell                               | Retry once with a browser strategy. If it remains unreadable, use the [browser agent](/docs/browser-agent) |

Forms found are still in `forms` whatever the warning says.

### Fill a wizard

One `fillForm` call is one step. Pass `advance: true` on each step that ends in "Next", and
`submit: true` on the last. If `advance` finds no next button, that step is the last one.

* Pass the same `open` you used for `getFields`, so both see the same form.
* `notFound` names any key that matched nothing.
* `resultContent` is plain text (up to 4,000 characters) from the page the click led to: a
  price, a match, a validation message. Search it for the value you expect.

It fills real inputs, selects and text areas. A page of clickable cards with no form gives it
nothing to fill. Read it with `read.page`, or click through it with the browser agent.

If a multi-step form needs interaction the form tool cannot complete, use the
[browser agent](/docs/browser-agent) for that step. It is billed per task, so start with the
form tool when the page has ordinary fields.

### The page could not be read

`readFailure` is set when the page was never read (a dead domain, a 404, a 500):

```js
const form = await bowmark.web_form_fields.getFields(url);
if (form.readFailure) {
  // readFailure.httpStatus: what the site answered, or null if it never answered
  return { ok: false, reason: form.readFailure.detail };
}
```

A 404 or unreachable domain will not change on retry. A 5xx or 429 might.

## When a page cannot be read

Read the result before you retry:

| What you see                                | What to do                                                                                                                                                        |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `429`, `5xx`, or `retry: "later"`           | Retry once after the wait                                                                                                                                         |
| `needsLogin: true`                          | Don't retry the same URL. Use a saved connection, or find a public page with the same data (a product page instead of a search listing)                           |
| `ok: false` with `wall: { cleared: false }` | Read it once more. A site can refuse one read and pass the next. If that read is walled too, use the [browser agent](/docs/browser-agent) once, or another source |
| `status: 0`                                 | See [Is it the site, or is it Bowmark?](/docs/responses#is-it-the-site-or-is-it-bowmark)                                                                          |
| A CAPTCHA                                   | `read.page` solves common ones before it answers                                                                                                                  |

If the browser agent also returns `blocked:`, Bowmark cannot read that site today. Look for the
same data on the maker's site, another retailer, or a site `get_library` covers. Or
[ask for the site](/sites).

A failed or walled read is still billed for what it used. See [Pricing](/docs/pricing).
