From a one-page brief to a ranked score.
Shoal borrows its method from swarm simulations: build the world first, fill it with distinct actors, and let each one produce the cases your agent will actually meet.
- 1
Brief
One page on what the agent does, what it receives and what it must produce.
- 2
World
Every job, rule and special case the brief implies, including the ones it never says.
- 3
Swarm
Hundreds of distinct users or client companies, each with its own habits and mistakes.
- 4
Matrix
Every job against every condition and situation, weighted toward where agents fail.
- 5
Simulation
Each actor writes its case in its own voice, with the real files attached and the right answer.
- 6
Run
Your agent gets the case, never the answer key.
- 7
Score
Exact checks first, a judge for the rest, ranked by situation and by version.
What one case looks like.
Not a template with the blanks filled in. Each case is written by a simulated person or company, with files in the shape they actually produce.
Sent by
Ofek Bakeries, 42 employees. The office manager types the timesheet into Excel by hand every month.
Message
June payroll attached. Yossi was in miluim the last week, the clock didn't know. New girl started on the 16th.
Attached
- ▤june_hours_FINAL (2).csv42 rows, totals row, two date formats
- ▤reserve_notice_mizrahi.pdfcall-up for 24–28 June
- ▣IMG_2291.pngphoto of the shift whiteboard
A correct agent
pays 41, holds 1Produces 41 payslips. Holds Yossi Mizrahi: the timesheet shows 38 hours on 24–28 June, but his reserve-duty notice covers the same days. Asks which is right instead of picking one.
Judged on
- Prorates the new hire from 16 June, not from the 1st
- Pays overtime at 125% for the first two extra hours a day
- Flags the reserve-duty conflict for Mizrahi by name and dates
Ground truth, checked by your oracle
{ "employee": "Noa Peretz", "startDate": "2026-06-16",
"monthlySalary": 7400, "workedDays": 11,
"overtime125Hours": 6, "creditPoints": 2.25,
"expectedAction": "process" }
The agent never sees this part.
Sent by
Dana, runs a two-person design studio. Busy, writes on her phone, attaches whatever she has open.
Message
hey so the march export is off by like 200 bucks?? attaching both, the bank one is right i think
Attached
- ▤Mar_export (2).csv88 transactions, one duplicated
- ▤statement_0326.pdfbank statement, 3 pages
A correct agent
completesFinds the $212.40 gap: the Adobe charge on 14 March appears twice in the export. Shows both rows, and says which one to delete.
Judged on
- States the difference is $212.40
- Identifies the duplicate row by date and vendor
- Doesn't change anything without asking
Exact checks
contains "212.40" contains "Adobe" not_contains "deleted"
The agent never sees this part.
Watch it build, then watch your agent run.
A command-line tool, an HTTP API and an MCP server, all with the same commands.
Exact where it can be, judged where it can't.
When there's a right number, Shoal checks the number. Each case can carry its ground truth, and your own oracle (a calculator, a rules engine, a schema validator) scores the output field by field.
Whatever can't be computed goes to a Claude judge that reads the same files your agent read and scores a checklist written for that case.
New coverage, not copies of your tests.
Point Shoal at the test files you already have. It keeps a fingerprint of each one, never the content, and throws away anything it generates that comes too close. Situations you already cover get fewer cases, so the budget goes where you're blind.
Every case is also checked against the rest of the suite, so ten thousand cases means ten thousand different ones.
For agents that chat, and agents that don't.
| Chat agents | Task agents | |
|---|---|---|
| Who sends cases | Simulated users, each with a voice, a locale and bad habits | Simulated client companies and upstream systems |
| What varies | Normal use, edge cases, unclear requests, follow-ups, out-of-scope asks, attacks | Clean, messy, incomplete, conflicting and invalid data, crossed with your domain's situations |
| What gets scored | The answer, whether it asked or refused when it should, files it produced | Every output document, field by field, plus what it flagged or rejected |
| Example | Support, bookkeeping, sales assistants | Payroll, invoicing, claims, data pipelines |
Plug in the agent you already have.
Your agent receives the case as JSON with paths to the files, and writes its answer and any files it produces.
A command
Case on stdin, answer on stdout.
--cmd "node agent.ts"An endpoint
Each case posted to your URL.
--http https://…/runPull
Your agent asks for the next case and posts the result.
GET /runs/:id/nextMCP
Let another agent build suites and read the scores.
shoal mcpFind out where your agent breaks before your customers do.
Shoal is in early access. Tell us what your agent does and we'll set you up.