Evals

Real recorded runs — the prompt as a person typed it, the script the model wrote, and the answer that came back off the live sites.

Every run below actually happened. The prompt is what a person typed, the calls are what the model wrote against the library, and the answer is what came back off the live sites. Nothing is hand-written and no prompt names a Bowmark function — the model gets one lookup and has to find its own way there.

They all ran on GPT-5.6 Luna, the cheapest tier OpenAI publishes, at $0.20 per million input tokens. That is deliberate. The hard part of these tasks is the website, not the model: the agent is calling a typed function, so it never has to be clever enough to drive a browser, read a rendered page, or recover from a layout change.

Every prompt was run five times. The pass count on each row is the real one — ten of these went five for five, and one did not; that row says so rather than rounding it up. The run shown is the fastest from the most recent batch, and the timing is real wall-clock for the whole exchange: the lookup, the script, every live site it touched, and the written answer.

A call that errored is marked. They are left in because the run genuinely made them, and a page claiming every step succeeded would be a different page from this one.

11 of 11 prompts · 72 recorded runs, all passing

Taking the kids to Six Flags Magic Mountain — what are the current pass prices?

model: gpt-5.6-luna
effort: medium
took: 14.5s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "Six Flags Magic Mountain current pass prices season pass membership tickets"

run8.7s

const r = await bowmark.theme_park_tickets.search("Six Flags Magic Mountain");
return r;

What came back

Current Six Flags Magic Mountain prices:

Slash Sale Ticket: $45 — regularly $110
Park Ticket: $49 — regularly $110
Fright Fest Bundle: $74 — admission plus haunted mazes
2027 Gold Pass: $139 per person — regularly $199

Prices are in USD and may have date or promotional restrictions. The live listing currently shows fall-event promotions, so check the purchase page for eligible dates and terms.

What's my iPhone 15 Pro 256GB worth as a trade-in if it's in good condition?

model: gpt-5.6-luna
effort: medium
took: 17.4s
passed: 3/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "current iPhone trade-in value for iPhone 15 Pro 256GB in good condition"

run11.0s

const r = await bowmark.phone_trade_in.estimate({ model: "iPhone 15 Pro", storage: "256GB", condition: "good" });
return r;

What came back

Your iPhone 15 Pro 256GB in good condition is currently worth approximately $370–$427 USD as a trade-in, depending on the buyer.

Highest offer: $426.99 from SellMyCellPhone
Other strong offers:
WebTradeIn: $423.99
BuyBackTronics: $405.00
ItsWorthMore, GadgetGone, BuyBackBoss: $395.00
Apple Trade In estimate: up to $370.00 — listed as a ceiling estimate and not condition-specific

Offers generally include free shipping, but payment methods and processing times vary. These values assume the phone is fully functional, unlocked or accepted by the buyer, and has no significant damage.

Renting a car out of DEN next Friday, returning Sunday — who's cheapest for the whole rental?

model: gpt-5.6-luna
effort: medium
took: 22s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "car rental from DEN next Friday returning Sunday cheapest total rental"

run12.4s

const { cars, warnings } = await bowmark.cars.search({ pickup: "DEN", pickupDate: "2026-09-25", dropoffDate: "2026-09-27" }, 50);
for (const w of warnings) log(w);
return { offers: cars.slice(0, 10), warnings };

What came back

For Friday, Sept. 25 through Sunday, Sept. 27 at DEN—the cheapest quote I found is:

$169 total ($85/day)
Ford Escape or similar, intermediate SUV
Turo, sold and fulfilled by Turo
Seats 5, 3 bags, automatic
Free cancellation
Pickup listed at Denver Intl; exact pickup instructions require contacting the host
Mileage is not unlimited
View the $169 offer

This is the lowest whole-rental total returned; taxes, fees, and final availability can change at checkout.

Flexible on dates — one way SFO to Tokyo in November. What are the cheapest days to go?

model: gpt-5.6-luna
effort: medium
took: 25.7s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "flights flexible dates one way SFO to Tokyo November 2026 cheapest days"

run18.3s

const graph = await bowmark.providers.google_flights.getPriceGraph({
  from: "SFO", to: "TYO", depart: "2026-11-08"
});
const november = graph.points
  .filter(p => p.price != null && p.date.startsWith("2026-11"))
  .sort((a,b) => a.price - b.price);
return {
  cheapestDays: november.slice(0, 10),
  window: [graph.rangeStart, graph.rangeEnd],
  allNovemberDays: november
};

What came back

For a one-way SFO → Tokyo flight in November 2026, the cheapest departure dates currently showing are:

November 23–26: $395 one-way
November 29: $395
November 2–6: $409
November 8–12, 16, 27–28, and 30: $409

The absolute cheapest options are therefore November 23–26 or November 29 at about $395. Prices are live and can change; this is based on the current Google Flights price graph for SFO → Tokyo (TYO).

Find the newest YouTube video about self-driving cars, get its transcript, and summarize it in five bullet points.

model: gpt-5.6-luna
effort: medium
took: 26.4s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "YouTube newest video self-driving cars transcript"

run18.0s

const videos = await bowmark.providers.youtube.search({query: 'self-driving cars', uploadedWithin: 'month'});
const ranked = videos.filter(v => v.publishedAgeSeconds != null).sort((a,b) => a.publishedAgeSeconds - b.publishedAgeSeconds);
const newest = ranked[0];
if (!newest) return {error: 'No matching videos found'};
const transcript = await bowmark.providers.youtube.getTranscript({video: newest.url});
return {video: newest, transcript};

What came back

Newest matching video found:

“How Do Self-Driving Cars See? 🤖 Inside the AI Behind Autonomous Cars” Channel: 60 Seconds Into Future — published about 9 hours ago Watch on YouTube

Five-point transcript summary:

The video explains how self-driving cars use sensor fusion to understand their surroundings, especially in poor weather.
Cameras provide visual information such as colors, road signs, and general scene context, but can be impaired by fog or darkness.
Radar penetrates fog and estimates objects’ speed and distance using reflected radio waves.
Lidar sends laser pulses to create a detailed 3D map, helping identify the shape of objects such as pedestrians.
The vehicle’s AI combines camera, radar, and lidar data into a unified real-time map and uses it to detect hazards and brake safely—though the video’s claim that this makes the system “completely infallible” is an overstatement.

I need to ship a 2 lb box from 94103 to 10001. What's the cheapest way to send it and how long will it take?

model: gpt-5.6-luna
effort: medium
took: 26.8s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "shipping rate quote 2 lb package 94103 to 10001 delivery time"

run15.8s

const { rates, warnings } = await bowmark.shipping.estimate({ fromZip: "94103", toZip: "10001", weightOz: 32 });
return {
  rates: rates.map(r => ({ carrier: r.source, service: r.serviceName, price: r.price.amount / 100, currency: r.price.currency, transitDays: r.transitDays, deliveryEstimate: r.deliveryEstimate ?? null })),
  warnings
};

What came back

For a regular 2 lb package from 94103 to 10001, the cheapest practical option is:

UPS Ground Saver via Pirate Ship: $9.96
Estimated delivery: Thursday, September 24 if shipped today
Transit time: about 7 calendar days

A faster option:

USPS Ground Advantage via Pirate Ship: $10.05
Estimated delivery: Tuesday, September 22
Transit time: about 5 calendar days

One special case: USPS Media Mail is $5.13, but it’s only valid for eligible media such as books, sound recordings, and certain educational materials—not ordinary merchandise. The quotes assumed no dimensions were provided; the discounted quote used a default 10 × 8 × 4 in. box, so the final price may change with the actual box size.

I don't want to get sucked into YouTube's feed today. Take the newest video from each of @mkbhd, @veritasium and @Computerphile, pull each transcript, and build me one web page with all three write-ups on it. Give me a link I can open.

model: gpt-5.6-luna
effort: medium
took: 32.5s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.2s

query: "YouTube channels newest video transcripts and create a web page link"

run20.4s

const refs = await Promise.all([
  bowmark.providers.youtube.findChannel({ query: "@mkbhd" }),
  bowmark.providers.youtube.findChannel({ query: "@veritasium" }),
  bowmark.providers.youtube.findChannel({ query: "@Computerphile" })
]);
if (refs.some(x => !x)) return { error: "Could not resolve one or more channels", refs };
const pages = await Promise.all(refs.map(r => bowmark.providers.youtube.listChannelVideos({ channel: r.channelId })));
const newest = pages.map((p, i) => ({
  channel: ["MKBHD", "Veritasium", "Computerphile"][i],
  ref: refs[i],
  video: p.videos && p.videos[0]
}));
if (newest.some(x => !x.video)) return { error: "Could not find a newest video", newest };
const transcripts = await Promise.all(newest.map(x => bowmark.providers.youtube.getTranscript({ video: x.video.videoId })));
const esc = s => String(s ?? "").replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/\"/g,"&quot;");
const blocks = newest.map((x, i) => {
  const t = transcripts[i];
  const body = t.segments && t.segments.length
    ? t.segments.map(s => `<p><span class="time">${esc(s.timestamp)}</span> ${esc(s.text)}</p>`).join("\n")
    : `<p><em>No caption transcript was available for this video.</em></p>`;
  return `<article><h2>${esc(x.channel)}</h2><h3><a href="https://www.youtube.com/watch?v=${esc(x.video.videoId)}" target="_blank" rel="noreferrer">${esc(x.video.title)}</a></h3><p class="meta">${esc(x.video.published || "")} · ${esc(x.video.length || "")}</p><section>${body}</section></article>`;
}).join("\n");
const html = `<!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Three videos, no feed</title><style>body{font:16px/1.55 system-ui,sans-serif;max-width:900px;margin:0 auto;padding:32px 20px;color:#202124;background:#fafafa}h1{font-size:2rem;margin-bottom:.2em}header{margin-bottom:32px}article{background:white;border:1px solid #ddd;border-radius:12px;padding:24px;margin:24px 0;box-shadow:0 2px 8px #0000000a}h2{margin:0;color:#666;font-size:1rem;text-transform:uppercase;letter-spacing:.08em}h3{font-size:1.35rem;margin:.35em 0}.meta{color:#666;font-size:.9rem}.time{font:12px ui-monospace,monospace;color:#777;margin-right:.5em}section p{margin:.65em 0}</style></head><body><header><h1>Three videos, no feed</h1><p>Newest videos from MKBHD, Veritasium, and Computerphile, with YouTube's available captions.</p></header>${blocks}</body></html>`;
const saved = await bowmark.files.save({ name: "three-videos-no-feed.html", text: html, contentType: "text/html; charset=utf-8" });
const pub = await bowmark.files.makePublic(saved.id);
return { url: pub.publicUrl, filesUrl: saved.url, videos: newest.map((x,i)=>({channel:x.channel,title:x.video.title,url:"https://www.youtube.com/watch?v="+x.video.videoId,transcriptSegments:transcripts[i].segments.length})), htmlBytes: html.length };

What came back

Done: open the page with all three transcripts

It includes the newest videos found from:

MKBHD — “iPhone 18 Pro Review: All About that Chip”
Veritasium — “We sent a camera to space to film the solar eclipse”
Computerphile — “The AI Language We Can't Read: Neuralese ft. Rob Miles”

Three cheapest direct SFO to JFK flights on Oct 16, please.

model: gpt-5.6-luna
effort: medium
took: 34.5s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "flights cheapest direct SFO to JFK October 16 2026"

run25.2s

const { flights, warnings } = await bowmark.flights.search({ from: "SFO", to: "JFK", depart: "2026-10-16", stops: "nonstop" });
const priced = flights.filter(f => f.price != null).sort((a,b) => a.price - b.price).slice(0,3);
return { flights: priced, warnings, totalMatches: flights.length };

What came back

Three cheapest nonstop, one-way SFO → JFK flights for October 16, 2026:

JetBlue — $252

Depart 8:44 PM, arrive 5:20 AM (+1 day) — 5h 36m View flight

Alaska — $282

Depart 10:23 PM, arrive 7:04 AM (+1 day) — 5h 41m View flight

JetBlue — $300

Depart 2:18 PM, arrive 10:59 PM — 5h 41m View flight

Prices and availability can change.

Is the lululemon Align High-Rise Pant 25" in stock in black, size 6? How much is it?

model: gpt-5.6-luna
effort: medium
took: 35.4s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.2s

query: "lululemon Align High-Rise Pant 25 black size 6 stock price"

run14.7s

const r = await bowmark.providers.lululemon.search({query: "Align High-Rise Pant 25\"", limit: 8}); return {products:r.products, matched:r.matched, warnings:r.warnings};

run9.7s

const p = await bowmark.providers.lululemon.getProduct({productId:"prod11900026"}); const hits=p.colorways.filter(c=>/black/i.test(c.color) && c.variants.some(v=>v.options.size==="6")); return {title:p.title,url:p.url,inStock:p.inStock,matches:hits.map(c=>({color:c.color,colorId:c.colorId,price:c.price,salePrice:c.salePrice,currentPrice:c.sale.currentPrice,originalPrice:c.sale.originalPrice,onSale:c.sale.onSale,promo:c.promoMessage,size6:c.variants.find(v=>v.options.size==="6")})),colorways:p.colorways.map(c=>({color:c.color,has6:c.variants.some(v=>v.options.size==="6"),price:c.price,salePrice:c.salePrice}))};

What came back

Yes. The lululemon Align No-Line™ High-Rise Pant 25" is currently in stock in Black, size 6.

Price: $108 USD
SKU: `us_158101538`
View the product

Where can I get AirPods Pro 3 the cheapest right now? Is it in stock?

model: gpt-5.6-luna
effort: medium
took: 53.6s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library0.1s

query: "find the cheapest current price and stock for AirPods Pro 3"

run45.0s

const ACCESSORY = /case|cover|protector|skin|strap|clip|ear ?tips|charger|mount|stand|holder|pouch|bundle|applecare|refurb|renewed|restored|pre-?owned/i;
const { results, warnings } = await bowmark.retail.search({ query: "AirPods Pro 3" });
const real = results.filter((o) => /airpods pro 3/i.test(o.title) && !ACCESSORY.test(o.title));
const priced = real.filter((o) => o.price !== null).sort((a,b) => a.price - b.price);
return { offers: priced, cheapestInStock: priced.find((o) => o.inStock) ?? null, warnings };

What came back

The cheapest current in-stock option I found is:

Walmart — $199.00 — In stock (view listing)
Target — $199.99 — In stock (view listing)
Best Buy — $199.99 — In stock (view listing)

Best deal: Walmart at $199, marked down from $249. Prices and stock can change at checkout.

Find me a hotel in Chicago for two nights next Friday and Saturday. I care about the total I actually pay, not the nightly rate.

model: gpt-5.6-luna
effort: medium
took: 57.8s
passed: 5/5
recorded: 2026-09-17

What it ran

get_library9.8s

query: "Chicago hotel search total price for two nights next Friday and Saturday"

run36.0s

const r = await bowmark.hotels.search({ location: "Chicago", checkIn: "2026-09-18", checkOut: "2026-09-20", adults: 2, rooms: 1 }, 10, { timeoutMs: 50000 });
for (const w of r.warnings) log(w);
return { hotels: r.hotels.map(h => ({ name: h.name, total: h.price, currency: h.currency, perNight: h.nightPrice, seller: h.seller, sellers: h.sellerCount, stars: h.stars, score: h.score, reviews: h.reviewCount, type: h.propertyType, neighborhood: h.neighborhood, distance: h.distance, url: h.url })) };

What came back

I interpreted “next Friday and Saturday” as check-in Friday, September 18, 2026, and check-out Sunday, September 20, 2026—two nights for two adults in one room.

Best value

New Relax Inn — $229 total for 2 nights

$102/night equivalent
Guest score: 7.4/10 from 185 reviews
Type: Motel
About 12.1 miles from Chicago
Cheapest seller: Booking.com
View booking options

Cheapest option

Plaza Hotel by OYO Calumet Park Alsip I-57 — $163 total

$76/night equivalent
Guest score: 4.9/10 from 34 reviews
About 14.9 miles away
Sold by Booking.com

Other reasonable options

MainStay Suites Chicago – Hoffman Estates — $247 total, score 7.6, 3-star, 29.9 miles away
Extended Stay America Suites – Chicago – Itasca — $286 total, score 8.0, 22 miles away
Des Plaines Motel by OYO — $209 total, score 6.0, 18.1 miles away

These are the live whole-stay totals returned by the hotel search, rather than nightly rates. The cheapest results are in Chicago suburbs rather than downtown; prices and availability can change before checkout, so verify the final tax/fee breakdown on the seller’s page.

Running them yourself

The prompts and the checks that grade them ship in the repo, in apps/demos/browser-use. node eval.mjs runs the lot on the cheap model; pass --model for a bigger one.

Three rules the checks enforce, because they are what make a pass mean anything:

A check never trusts the prose. It pulls the links and prices out of the answer and requires each one to appear in what Bowmark actually returned on that run. An answer that invents a number fails, even when the number happens to be right.
Some checks require a specific function. The month-of-fares prompt fails if the agent ran thirty searches instead of reading the price graph, even when the answer is correct. Getting there the slow way is the thing being measured.
Only passing runs are published here. Failures stay in the repo and on the board, which is where they are useful.

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

What it ran

What came back

Running them yourself

On this page