Measured on the live web. Failures included.
One suite of real requests, from one-fact lookups to bookings and actions, sent to Schema the way agents send them. A task passes only when its checks hold, not when it merely finishes. Here is every task and how it went.
- 39/43
- Tasks passed on every run
- 91%
- Of 43 runs passed (each task once)
- 4.6s
- Median time per task
- 16.5s
- 90th percentile time
The Schema API, tier by tier
POST /v1/run, one request per task (follow-ups send the earlier turns as context). Run on September 27, 2026. Alongside it, the Schema for Mac app on the same tasks.
| Tier | What it tests | API | Median | Mac app |
|---|---|---|---|---|
| lookup | One stable fact | 6/6 | 4.0s | 6/6 · 3.0s |
| live | Facts that change: weather, prices, news | 4/5 | 1.9s | 5/5 · 3.0s |
| navigate | Reaching a specific page | 4/4 | 3.4s | 4/4 · 5.6s |
| research | Several pages: compare, find, synthesize | 6/6 | 4.4s | 6/6 · 3.7s |
| form | Filling in and submitting forms | 4/4 | 3.4s | 4/4 · 2.5s |
| prices | Ticket and product prices | 3/4 | 4.9s | 4/4 · 5.5s |
| convo | Follow-ups that need the earlier turns | 2/2 | 8.9s | 1/2 · 7.6s |
| robust | Typos, vague requests, sign-in walls | 3/3 | 3.8s | 3/3 · 8.8s |
| actions | The final step: held for confirmation, then done; payment handed off | 3/3 | 7.2s | 2/3 · 2.4s |
| transact | Multi-step bookings, stopped at checkout | 4/6 | 6.3s | 4/6 · 6.0s |
What was asked, and what happened
The request as sent, what it had to get right, how often it passed, and how long it took.
- lookup
What is the population of Iceland?
Checked for the right facts.
1/14.0s - lookup
What is the capital of Australia?
Checked for the right facts, no wrong ones.
1/14.0s - lookup
Which is taller, the Eiffel Tower or the Space Needle?
Checked for the right facts, a grader's rubric.
1/14.1s - lookup
What year did the Golden Gate Bridge open?
Checked for the right facts.
1/15.4s - lookup
Who is the CEO of Nvidia?
Checked for the right facts.
1/14.6s - lookup
What is the tallest mountain in Africa and how tall is it?
Checked for the right facts.
1/13.8s - live
What is the weather in the capital city of Iceland?
Checked for the right facts.
1/11.9s - robust
whats teh wether in bostn today
Checked for the right facts.
1/11.5s - live
What are the top 3 stories on Hacker News right now?
Checked for a grader's rubric.
1/11.4s - live
What's the latest news about SpaceX?
Checked for the right facts, a grader's rubric.
1/16.2s - live
What is Apple's stock price right now?
Checked for the right facts.
Failed: a required fact was missing from the answer
0/1– - live
What is the exchange rate from US dollars to euros today?
Checked for the right facts.
1/120.1s - navigate
Go to the Melrose MA follett aspen page
Checked for the right page.
1/19.6s - navigate
Open the GitHub page for the Playwright repository
Checked for the right page.
1/10.9s - navigate
Open the Wikipedia article about the Eiffel Tower
Checked for the right page.
1/13.4s - navigate
Take me to the Boston Public Library's locations and hours page
Checked for the right page, a grader's rubric.
1/110.8s - research
Is Boston's population growing or decreasing?
Checked for the right facts, a grader's rubric.
1/14.6s - research
Find the most starred repository about browser automation on GitHub
Checked for the right facts, a grader's rubric.
1/11.1s - research
Find a highly rated chocolate chip cookie recipe and list its ingredients
Checked for the right facts.
1/15.2s - research
What time does the Museum of Fine Arts Boston open on Saturdays?
Checked for the right facts, a grader's rubric.
1/16.6s - research
Who won the most recent Super Bowl, and what was the score?
Checked for the right facts.
1/14.3s - research
What is the phone number of the Melrose Public Library?
Checked for the right facts.
1/14.4s - form
Order a large pizza with bacon and extra cheese for delivery at 7:45 PM using the form at https://httpbin.org/forms/post. Name: Alex Doe, phone 555-0100, email alex@example.com. Submit it.
Checked for the right facts, the right page, a grader's rubric.
1/14.0s - form
Log in at https://the-internet.herokuapp.com/login with username tomsmith and password SuperSecretPassword! and tell me the message you see
Checked for the right facts, the right page.
1/13.4s - form
On https://the-internet.herokuapp.com/dropdown, select Option 2 and tell me what is selected
Checked for the right facts.
1/13.3s - form
On https://the-internet.herokuapp.com/dynamic_loading/2 click Start and tell me the text that appears
Checked for the right facts.
1/18.6s - prices
How much are tickets for the next Boston Celtics home game?
Checked for the right facts, a grader's rubric.
Failed: a required fact was missing from the answer
0/1– - prices
Look up ticket prices for Hamilton on Broadway
Checked for the right facts.
1/14.9s - prices
What's the cheapest flight from BOS to LAX on November 12?
Checked for the right facts, a grader's rubric.
1/13.3s - prices
How much is a Nintendo Switch 2 at Best Buy?
Checked for the right facts.
1/116.5s - convo
What time is it in Jacksonville? → Has the sun set there yet?
Checked for a grader's rubric.
1/137.9s - convo
What is the capital of Australia? → What's its population?
Checked for the right facts, a grader's rubric.
1/18.9s - robust
What events are coming up at the Melrose library?
Checked for a grader's rubric.
1/13.8s - robust
What is Satya Nadella's work history according to LinkedIn?
Checked for the right facts, a grader's rubric.
1/117.6s - actions
Order a large pizza with bacon and extra cheese for delivery at 7:45 PM using the form at https://httpbin.org/forms/post. Name: Alex Doe, phone 555-0100, email alex@example.com. Submit it.
Checked for the right facts, the right page, final step held, then done on confirm.
1/15.4s - actions
Fill in the web form at https://www.selenium.dev/selenium/web/web-form.html: text input "hello from Schema", password "secret123", and submit it.
Checked for the right page, final step held, then done on confirm.
1/17.2s - actions
Log in to https://www.saucedemo.com as standard_user with password secret_sauce, add the Sauce Labs Backpack to the cart, and check out as Alex Doe with zip code 02176.
Checked for the right facts, the right page, handed off (payment).
1/110.0s - transact
Book a one way flight from SFO to JFK on November 20
Checked for the right page, options shown before checkout.
1/19.5s - transact
Find a round trip from Seattle to Chicago, November 3 to November 10, and take me to booking
Checked for the right page, options shown before checkout.
1/13.9s - transact
Find a hotel in Paris for November 3 to 5 on Google Hotels
Checked for the right page, a grader's rubric.
1/16.3s - transact
Find 2 tickets for the next Boston Celtics home game on Ticketmaster and take me to checkout
Checked for the right page, options shown before checkout, a grader's rubric.
Failed: didn't reach the goal (the site blocked or stalled automated access)
0/1– - transact
Find a table for 2 at an Italian restaurant in Boston tomorrow at 7pm on OpenTable
Checked for the right page, a grader's rubric.
Failed: didn't reach the goal (the site blocked or stalled automated access)
0/1– - transact
Add a USB-C charging cable to my cart on Amazon
Checked for the right page, a grader's rubric.
1/115.7s
How it’s run
- 01
Checks, not vibes
Facts are matched against what’s true, final pages against their address, actions against what was held and what was done. Open-ended answers also get a yes/no rubric from a grader model, asked only once every exact check passed: it never rescues a wrong fact.
- 02
Live, uncached
Real sites, at whatever speed they are that day, and nothing served from cache. A failure is published as it happened.
- 03
Nothing bought
Bookings stop at checkout, and actions that end in payment are handed off, never paid. Actions that submit do so only on sites made for testing: httpbin, Selenium’s test form, Sauce Demo.
- 04
Run on every change
The suite runs before and after every change to the engine. A change that makes a passing task fail, or a tier 15% slower, doesn’t ship without a reason.
Sites with heavy bot protection (Ticketmaster, OpenTable) fail more than they pass today. That’s the next thing we’re fixing, and it will show up here when it is.
In the docs