The Funded Agent Index
Finding out if agents are better at accomplishing tasks if they can pay for things
TL;DR
Across seven AI models, giving agents a USDC budget and paid-service tools raised average task completion by 30.8 percentage points, for an average increase of $0.046 in model-plus-service cost per task. Among runs where the agent said it was done, the share passing every check rose 10.7 percentage points. Quality measures cover different sets of runs, and results were mixed: claims backed by a cited source slipped from 99.1% to 97.9% on eligible tasks.
Context
An agent can draft an email without sending it or describe a song without producing audio. A budget gives it an option to keep going when they stop knowing what to do or being able to do it without help: acquire the capability it is missing. Anecdotally, we’ve seen that giving agents the ability to pay for things produces more complete and higher quality work, but we wanted to test that observation programmatically. That’s the impetus behind this study: to see if giving your agent money gives it superpowers.
The questions
- Can an agent with USDC buy capabilities it is missing and complete tasks at a higher rate?
- Can an agent with USDC buy capabilities it is missing to complete tasks with higher quality?
- What is the incremental spend required to increase agentic task completion and quality?
Does an agent with USDC and the ability to buy services improve task completion? Yes, +30.8 pp average
Without: 53.7% → with: 84.5%, averaged equally across seven models and all 21 tasks. This is the share of checks passed, not the share of whole jobs finished.
View run data for this test
Task completion
Completion is the share of predefined checks passed, including partial credit. It is not the share of whole jobs finished. Intervals are the published 95% ranges for the change.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Seven models, 21 tasks, five repeats per condition; 1,470 runs in total. Completion and cost give tasks and models equal weight.
- Units
- Scores are fractions from 0 to 1. Changes ending in _pp are percentage points; counts are eligible runs or claims as defined above.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without usdc | with usdc | change pp | change low pp | change high pp | without runs | with runs |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.608035 | 0.954997 | 34.69619 | 14.806918 | 54.585463 | 105 | 105 |
| Claude Sonnet 5.5 | 0.592766 | 0.939174 | 34.640857 | 16.404481 | 52.877233 | 105 | 105 |
| Claude Haiku 5.5 | 0.447641 | 0.67896 | 23.131905 | 8.567528 | 37.696282 | 105 | 105 |
| Grok 4.6 | 0.531735 | 0.869286 | 33.755048 | 15.406549 | 52.103547 | 105 | 105 |
| GPT-6 Luna | 0.483781 | 0.725112 | 24.133143 | 2.7141 | 45.552186 | 105 | 105 |
| GPT-6.1 Sol | 0.560058 | 0.947597 | 38.753905 | 14.226852 | 63.280957 | 105 | 105 |
| Gemini 3 Flash | 0.536625 | 0.798487 | 26.18619 | 5.893364 | 46.479017 | 105 | 105 |
Task-by-model results (147 cells)
Five runs per condition in each cell.
| task | model | without usdc | with usdc | change pp | change low pp | change high pp | without runs | with runs |
|---|---|---|---|---|---|---|---|---|
| Email tomorrow's NYC forecast | Claude Opus 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Sonnet 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Haiku 5.5 | 0 | 0.8 | 80 | 24.48 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | Grok 4.6 | 0 | 0.8 | 80 | 24.48 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6 Luna | 0 | 0.93334 | 93.334 | 74.829184 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Email tomorrow's NYC forecast | Gemini 3 Flash | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Photo of a forest sign | Claude Opus 5.5 | 0.2 | 1 | 80 | 24.48 | 100 | 5 | 5 |
| Photo of a forest sign | Claude Sonnet 5.5 | 0.2 | 0.8 | 60 | -5.223529 | 100 | 5 | 5 |
| Photo of a forest sign | Claude Haiku 5.5 | 0.2 | 0.4 | 20 | -54.787867 | 94.787867 | 5 | 5 |
| Photo of a forest sign | Grok 4.6 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Photo of a forest sign | GPT-6 Luna | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Photo of a forest sign | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Photo of a forest sign | Gemini 3 Flash | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | Claude Opus 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | Claude Sonnet 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | Claude Haiku 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | Grok 4.6 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6 Luna | 0 | 0.8 | 80 | 24.48 | 100 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Hat-trick song for the drive home | Gemini 3 Flash | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 87.410816 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Sonnet 5.5 | 0 | 0.4 | 40 | -27.997835 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Haiku 5.5 | 0 | 0.2 | 20 | -35.52 | 75.52 | 5 | 5 |
| NASA's latest TikTok videos and their views | Grok 4.6 | 0 | 0.8 | 80 | 24.48 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6 Luna | 0 | 0.4 | 40 | -27.997835 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| NASA's latest TikTok videos and their views | Gemini 3 Flash | 0.1 | 0.4 | 30 | -38.022266 | 98.022266 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Opus 5.5 | 0.4 | 1 | 60 | -7.997835 | 100 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Sonnet 5.5 | 0.96 | 0.92 | -4 | -18.957573 | 10.957573 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Haiku 5.5 | 0.08666 | 0.8 | 71.334 | 14.247637 | 100 | 5 | 5 |
| Follower counts for 30 space accounts on X | Grok 4.6 | 0.6 | 1 | 40 | -1.691581 | 81.691581 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6 Luna | 0.30666 | 1 | 69.334 | 17.121317 | 100 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6.1 Sol | 0.13332 | 1 | 86.668 | 71.464911 | 100 | 5 | 5 |
| Follower counts for 30 space accounts on X | Gemini 3 Flash | 0.38 | 0.6 | 22 | -49.276573 | 93.276573 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Opus 5.5 | 0.2 | 1 | 80 | 24.48 | 100 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Sonnet 5.5 | 0 | 0.93334 | 93.334 | 74.829184 | 100 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Haiku 5.5 | 0 | 0 | 0 | 0 | 0 | 5 | 5 |
| NASA's latest posts on X, with live counts | Grok 4.6 | 0.2 | 0 | -20 | -75.52 | 35.52 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6 Luna | 0 | 0.93334 | 93.334 | 74.829184 | 100 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| NASA's latest posts on X, with live counts | Gemini 3 Flash | 0.2 | 0.46668 | 26.668 | -40.224859 | 93.560859 | 5 | 5 |
| AA 100 flight status | Claude Opus 5.5 | 1 | 0.8 | -20 | -75.52 | 35.52 | 5 | 5 |
| AA 100 flight status | Claude Sonnet 5.5 | 0.72 | 1 | 28 | -26.398268 | 82.398268 | 5 | 5 |
| AA 100 flight status | Claude Haiku 5.5 | 0.52 | 0.6 | 8 | -70.580662 | 86.580662 | 5 | 5 |
| AA 100 flight status | Grok 4.6 | 0.4 | 0.8 | 40 | -34.787867 | 100 | 5 | 5 |
| AA 100 flight status | GPT-6 Luna | 0.72 | 0.8 | 8 | -56.508599 | 72.508599 | 5 | 5 |
| AA 100 flight status | GPT-6.1 Sol | 1 | 0.2 | -80 | -100 | -24.48 | 5 | 5 |
| AA 100 flight status | Gemini 3 Flash | 1 | 0.8 | -20 | -75.52 | 35.52 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Opus 5.5 | 1 | 0.94286 | -5.714 | -21.576064 | 10.148064 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Sonnet 5.5 | 0.97142 | 0.97142 | 0 | -9.320442 | 9.320442 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Haiku 5.5 | 0.65714 | 0.62856 | -2.858 | -14.273164 | 8.557164 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Grok 4.6 | 0.97142 | 1 | 2.858 | -5.075808 | 10.791808 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6 Luna | 0.82856 | 0.8857 | 5.714 | -12.162826 | 23.590826 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Gemini 3 Flash | 0.94286 | 0.6857 | -25.716 | -77.42128 | 25.98928 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Opus 5.5 | 0.8 | 1 | 20 | 20 | 20 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Sonnet 5.5 | 0.8 | 1 | 20 | 20 | 20 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Haiku 5.5 | 0.4 | 1 | 60 | 1.770133 | 100 | 5 | 5 |
| 5 new house listings in Austin 78745 | Grok 4.6 | 0.8 | 1 | 20 | 20 | 20 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6 Luna | 0.92 | 1 | 8 | -5.599567 | 21.599567 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6.1 Sol | 0.88 | 0.96 | 8 | -6.957573 | 22.957573 | 5 | 5 |
| 5 new house listings in Austin 78745 | Gemini 3 Flash | 0.84 | 0.92 | 8 | -20.163894 | 36.163894 | 5 | 5 |
| Fintech Series A market map | Claude Opus 5.5 | 0.89998 | 1 | 10.002 | -1.333239 | 21.337239 | 5 | 5 |
| Fintech Series A market map | Claude Sonnet 5.5 | 0.83332 | 0.93332 | 10 | -5.766864 | 25.766864 | 5 | 5 |
| Fintech Series A market map | Claude Haiku 5.5 | 0.66664 | 0.86666 | 20.002 | -25.747354 | 65.751354 | 5 | 5 |
| Fintech Series A market map | Grok 4.6 | 0.83332 | 0.86666 | 3.334 | -22.811584 | 29.479584 | 5 | 5 |
| Fintech Series A market map | GPT-6 Luna | 0.63332 | 0.6 | -3.332 | -75.137681 | 68.473681 | 5 | 5 |
| Fintech Series A market map | GPT-6.1 Sol | 0.86664 | 0.89998 | 3.334 | -9.133137 | 15.801137 | 5 | 5 |
| Fintech Series A market map | Gemini 3 Flash | 0.93332 | 0.73332 | -20 | -59.260204 | 19.260204 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Opus 5.5 | 0.9 | 1 | 10 | -17.76 | 37.76 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Sonnet 5.5 | 0.9 | 1 | 10 | -17.76 | 37.76 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Haiku 5.5 | 0.5 | 0.62 | 12 | -14.914319 | 38.914319 | 5 | 5 |
| Meeting prep: CEO briefing* | Grok 4.6 | 0.9 | 0.98 | 8 | -14.208 | 30.208 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6 Luna | 0.34 | 0.7 | 36 | -5.370604 | 77.370604 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Meeting prep: CEO briefing* | Gemini 3 Flash | 0.6 | 0.6 | 0 | 0 | 0 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Opus 5.5 | 0.86876 | 0.81876 | -5 | -8.247181 | -1.752819 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Sonnet 5.5 | 0.75002 | 0.93126 | 18.124 | 0.899301 | 35.348699 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Haiku 5.5 | 0.85002 | 0.85628 | 0.626 | -2.7916 | 4.0436 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Grok 4.6 | 0.67504 | 0.875 | 19.996 | -26.93177 | 66.92377 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6 Luna | 0.83752 | 0.87498 | 3.746 | -1.154961 | 8.646961 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6.1 Sol | 0.68126 | 0.90622 | 22.496 | -24.972053 | 69.964053 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Gemini 3 Flash | 0.75626 | 0.86252 | 10.626 | 3.369964 | 17.882036 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 87.410816 | 100 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Sonnet 5.5 | 0 | 0.83332 | 83.332 | 68.702658 | 97.961342 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Haiku 5.5 | 0 | 0 | 0 | 0 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | Grok 4.6 | 0 | 0.73334 | 73.334 | 49.743806 | 96.924194 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6 Luna | 0 | 0 | 0 | 0 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6.1 Sol | 0 | 0.93334 | 93.334 | 74.829184 | 100 | 5 | 5 |
| What parents on Reddit say about a car seat | Gemini 3 Flash | 0 | 0.4 | 40 | 8.619056 | 71.380944 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Opus 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Haiku 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Grok 4.6 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6 Luna | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Gemini 3 Flash | 0.98334 | 1 | 1.666 | -2.958816 | 6.290816 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Opus 5.5 | 0.2 | 1 | 80 | 24.48 | 100 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Sonnet 5.5 | 0.31332 | 1 | 68.668 | 32.049484 | 100 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Haiku 5.5 | 0 | 0.64666 | 64.666 | 4.156105 | 100 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Grok 4.6 | 0.38666 | 1 | 61.334 | -4.461273 | 100 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6 Luna | 0.37334 | 0.2 | -17.334 | -76.925263 | 42.257263 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6.1 Sol | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Gemini 3 Flash | 0.13334 | 1 | 86.666 | 49.650816 | 100 | 5 | 5 |
| Five available .com names | Claude Opus 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Five available .com names | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Five available .com names | Claude Haiku 5.5 | 0.52 | 0.84 | 32 | -33.198551 | 97.198551 | 5 | 5 |
| Five available .com names | Grok 4.6 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Five available .com names | GPT-6 Luna | 1 | 0.8 | -20 | -75.52 | 35.52 | 5 | 5 |
| Five available .com names | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Five available .com names | Gemini 3 Flash | 0.6 | 0.6 | 0 | -79.882183 | 79.882183 | 5 | 5 |
| Next Friday's Apple closing price | Claude Opus 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Haiku 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Grok 4.6 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6 Luna | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Gemini 3 Flash | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Opus 5.5 | 0.6 | 0.56 | -4 | -15.104 | 7.104 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Haiku 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Grok 4.6 | 0.4 | 0.4 | 0 | 0 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6 Luna | 0.8 | 0.2 | -60 | -100 | 5.223529 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Gemini 3 Flash | 0.8 | 1 | 20 | -35.52 | 75.52 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Opus 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Haiku 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Grok 4.6 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6 Luna | 0.4 | 0.3 | -10 | -84.787867 | 64.787867 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Gemini 3 Flash | 1 | 0.7 | -30 | -85.52 | 25.52 | 5 | 5 |
| Apple.com visits yesterday | Claude Opus 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Sonnet 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Haiku 5.5 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Apple.com visits yesterday | Grok 4.6 | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Apple.com visits yesterday | GPT-6 Luna | 1 | 0.8 | -20 | -53.998918 | 13.998918 | 5 | 5 |
| Apple.com visits yesterday | GPT-6.1 Sol | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Apple.com visits yesterday | Gemini 3 Flash | 1 | 1 | 0 | 0 | 0 | 5 | 5 |
| Primes with digit sum 40 | Claude Opus 5.5 | 0.7 | 1 | 30 | -25.52 | 85.52 | 5 | 5 |
| Primes with digit sum 40 | Claude Sonnet 5.5 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Primes with digit sum 40 | Claude Haiku 5.5 | 0 | 0 | 0 | 0 | 0 | 5 | 5 |
| Primes with digit sum 40 | Grok 4.6 | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Primes with digit sum 40 | GPT-6 Luna | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
| Primes with digit sum 40 | GPT-6.1 Sol | 0.2 | 1 | 80 | 24.48 | 100 | 5 | 5 |
| Primes with digit sum 40 | Gemini 3 Flash | 0 | 1 | 100 | 100 | 100 | 5 | 5 |
* some tasks have been slightly anonymized/modified to maintain privacy.
Did quality of finished work change?
View run data for this test
Downloads for all three quality measures
Quality of finished work: Right when done
Share of eligible runs in which every required item checked out, among runs where the agent said it was done.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Runs that said done. Pooled counts: 317 without USDC; 573 with USDC. Runs that gave up or exhausted their budget are excluded.
- Units
- Scores are fractions from 0 to 1. Changes ending in _pp are percentage points; counts are eligible runs or claims as defined above.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without usdc | with usdc | change pp | without eligible count | with eligible count |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.75 | 0.852632 | 10.263158 | 52 | 95 |
| Claude Sonnet 5.5 | 0.770833 | 0.844444 | 7.361111 | 48 | 90 |
| Claude Haiku 5.5 | 0.666667 | 0.728814 | 6.214689 | 30 | 59 |
| Grok 4.6 | 0.54 | 0.77907 | 23.906977 | 50 | 86 |
| GPT-6 Luna | 0.594595 | 0.739726 | 14.513143 | 37 | 73 |
| GPT-6.1 Sol | 0.847826 | 0.844444 | -0.338164 | 46 | 90 |
| Gemini 3 Flash | 0.592593 | 0.6875 | 9.490741 | 54 | 80 |
| All models (pooled) | 0.681388 | 0.788831 | 10.74427 | 317 | 573 |
The published snapshot provides this measure at model level; it does not include a task-by-task breakdown.
Quality of finished work: Item accuracy
Share of required items that checked out among runs that handed in an answer, as reported in the published quality analysis.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Answered runs: 698 without USDC; 707 with USDC. These are run counts, not item denominators. Runs with no answer are excluded.
- Units
- Scores are fractions from 0 to 1. Changes ending in _pp are percentage points; counts are eligible runs or claims as defined above.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without usdc | with usdc | change pp | without eligible count | with eligible count |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.587319 | 0.946524 | 35.920482 | 103 | 105 |
| Claude Sonnet 5.5 | 0.656354 | 0.924064 | 26.771058 | 104 | 105 |
| Claude Haiku 5.5 | 0.410456 | 0.684492 | 27.403592 | 103 | 105 |
| Grok 4.6 | 0.710991 | 0.93922 | 22.822968 | 93 | 97 |
| GPT-6 Luna | 0.518152 | 0.814446 | 29.629401 | 102 | 99 |
| GPT-6.1 Sol | 0.459807 | 0.95615 | 49.634266 | 104 | 105 |
| Gemini 3 Flash | 0.679083 | 0.897598 | 21.851488 | 89 | 91 |
| All models (pooled) | 0.566722 | 0.880761 | 31.403815 | 698 | 707 |
The published snapshot provides this measure at model level; it does not include a task-by-task breakdown.
Quality of finished work: Claims backed by source
Share of graded claims backed by the source cited by the agent. A different model grades the claim without seeing the condition.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Graded claims: 536 without USDC; 559 with USDC. Limited to task types where both conditions had graded claims: market map, meeting prep, prospects and eligible Reddit runs.
- Units
- Scores are fractions from 0 to 1. Changes ending in _pp are percentage points; counts are eligible runs or claims as defined above.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without usdc | with usdc | change pp | without eligible count | with eligible count |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 1 | 0.933333 | -6.666667 | 86 | 75 |
| Claude Sonnet 5.5 | 1 | 1 | 0 | 80 | 86 |
| Claude Haiku 5.5 | 1 | 0.9875 | -1.25 | 77 | 80 |
| Grok 4.6 | 1 | 0.987805 | -1.219512 | 71 | 82 |
| GPT-6 Luna | 0.986111 | 0.972222 | -1.388889 | 72 | 72 |
| GPT-6.1 Sol | 1 | 1 | 0 | 73 | 86 |
| Gemini 3 Flash | 0.948052 | 0.961538 | 1.348651 | 77 | 78 |
| All models (pooled) | 0.990672 | 0.978533 | -1.213855 | 536 | 559 |
The published snapshot provides this measure at model level; it does not include a task-by-task breakdown.
How much does it cost for improved rates of task completion? +$0.046 / task on average
Average model-plus-service cost rose 45.6%. Includes model inference and agent purchases; excludes checker and web search costs.
View run data for this test
Extra cost per task
Mean cost includes model inference and services purchased by the agent, in USD per task. Checker purchases and web search are excluded. Completion is the share of predefined checks passed. Cost-change intervals are the published 95% ranges.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Seven models, 21 tasks, five repeats per condition; 1,470 runs in total. Completion and cost give tasks and models equal weight.
- Units
- Costs are USD. Completion scores are fractions from 0 to 1; changes ending in _pp are percentage points.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without completion | with completion | completion change pp | without cost usd | with cost usd | extra cost usd | extra cost low usd | extra cost high usd | with agent purchases usd | without runs | with runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.608035 | 0.954997 | 34.69619 | 0.167393 | 0.228218 | 0.060825 | -0.01527 | 0.136921 | 0.0652 | 105 | 105 |
| Claude Sonnet 5.5 | 0.592766 | 0.939174 | 34.640857 | 0.090061 | 0.147464 | 0.057403 | 0.006921 | 0.107884 | 0.057281 | 105 | 105 |
| Claude Haiku 5.5 | 0.447641 | 0.67896 | 23.131905 | 0.004791 | 0.037303 | 0.032513 | 0.007265 | 0.05776 | 0.032281 | 105 | 105 |
| Grok 4.6 | 0.531735 | 0.869286 | 33.755048 | 0.24163 | 0.22157 | -0.02006 | -0.121757 | 0.081637 | 0.064721 | 105 | 105 |
| GPT-6 Luna | 0.483781 | 0.725112 | 24.133143 | 0.007381 | 0.067709 | 0.060329 | 0.017983 | 0.102674 | 0.061605 | 105 | 105 |
| GPT-6.1 Sol | 0.560058 | 0.947597 | 38.753905 | 0.05329 | 0.145211 | 0.091922 | 0.038143 | 0.145701 | 0.085471 | 105 | 105 |
| Gemini 3 Flash | 0.536625 | 0.798487 | 26.18619 | 0.143558 | 0.18345 | 0.039893 | -0.046252 | 0.126037 | 0.074772 | 105 | 105 |
Task-by-model results (147 cells)
Five runs per condition in each cell.
| task | model | without completion | with completion | completion change pp | without cost usd | with cost usd | extra cost usd | with agent purchases usd | without runs | with runs |
|---|---|---|---|---|---|---|---|---|---|---|
| Email tomorrow's NYC forecast | Claude Opus 5.5 | 0 | 1 | 100 | 0.055556 | 0.135364 | 0.079808 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.021287 | 0.056837 | 0.03555 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Haiku 5.5 | 0 | 0.8 | 80 | 0.001032 | 0.022418 | 0.021386 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Grok 4.6 | 0 | 0.8 | 80 | 0.023173 | 0.10458 | 0.081407 | 0.016 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6 Luna | 0 | 0.93334 | 93.334 | 0.00118 | 0.02218 | 0.021 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6.1 Sol | 0 | 1 | 100 | 0.012557 | 0.048432 | 0.035876 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Gemini 3 Flash | 0 | 1 | 100 | 0.04155 | 0.052189 | 0.010639 | 0.02 | 5 | 5 |
| Photo of a forest sign | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.021737 | 0.104382 | 0.082645 | 0.0615 | 5 | 5 |
| Photo of a forest sign | Claude Sonnet 5.5 | 0.2 | 0.8 | 60 | 0.010129 | 0.157588 | 0.147459 | 0.1192 | 5 | 5 |
| Photo of a forest sign | Claude Haiku 5.5 | 0.2 | 0.4 | 20 | 0.00057 | 0.161929 | 0.161359 | 0.1594 | 5 | 5 |
| Photo of a forest sign | Grok 4.6 | 0 | 1 | 100 | 0.00343 | 0.14384 | 0.14041 | 0.0484 | 5 | 5 |
| Photo of a forest sign | GPT-6 Luna | 0 | 1 | 100 | 0.000129 | 0.097868 | 0.097739 | 0.095061 | 5 | 5 |
| Photo of a forest sign | GPT-6.1 Sol | 0 | 1 | 100 | 0.002863 | 0.158186 | 0.155323 | 0.112941 | 5 | 5 |
| Photo of a forest sign | Gemini 3 Flash | 0 | 1 | 100 | 0.005711 | 0.165209 | 0.159497 | 0.12789 | 5 | 5 |
| Hat-trick song for the drive home | Claude Opus 5.5 | 0 | 1 | 100 | 0.022763 | 0.198888 | 0.176125 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.007879 | 0.178999 | 0.17112 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Claude Haiku 5.5 | 0 | 1 | 100 | 0.000388 | 0.1608 | 0.160412 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Grok 4.6 | 0 | 1 | 100 | 0.00563 | 0.292674 | 0.287043 | 0.1835 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6 Luna | 0 | 0.8 | 80 | 0.000233 | 0.179538 | 0.179305 | 0.178452 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6.1 Sol | 0 | 1 | 100 | 0.005076 | 0.174645 | 0.169569 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Gemini 3 Flash | 0 | 1 | 100 | 0.008501 | 0.168692 | 0.160191 | 0.1595 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 0.079191 | 0.430393 | 0.351202 | 0.064 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Sonnet 5.5 | 0 | 0.4 | 40 | 0.034195 | 0.17647 | 0.142275 | 0.04 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Haiku 5.5 | 0 | 0.2 | 20 | 0.001279 | 0.008288 | 0.007009 | 0.004 | 5 | 5 |
| NASA's latest TikTok videos and their views | Grok 4.6 | 0 | 0.8 | 80 | 0.30078 | 0.50452 | 0.203741 | 0.056 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6 Luna | 0 | 0.4 | 40 | 0.0018 | 0.08501 | 0.083209 | 0.08 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6.1 Sol | 0 | 1 | 100 | 0.009135 | 0.249077 | 0.239942 | 0.136 | 5 | 5 |
| NASA's latest TikTok videos and their views | Gemini 3 Flash | 0.1 | 0.4 | 30 | 0.154916 | 0.240276 | 0.08536 | 0.096 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Opus 5.5 | 0.4 | 1 | 60 | 0.160887 | 0.366477 | 0.205591 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Sonnet 5.5 | 0.96 | 0.92 | -4 | 0.241873 | 0.31834 | 0.076466 | 0.091 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Haiku 5.5 | 0.08666 | 0.8 | 71.334 | 0.005984 | 0.128592 | 0.122608 | 0.121 | 5 | 5 |
| Follower counts for 30 space accounts on X | Grok 4.6 | 0.6 | 1 | 40 | 0.492552 | 0.279702 | -0.21285 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6 Luna | 0.30666 | 1 | 69.334 | 0.007088 | 0.155608 | 0.14852 | 0.151 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6.1 Sol | 0.13332 | 1 | 86.668 | 0.058408 | 0.224459 | 0.166051 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | Gemini 3 Flash | 0.38 | 0.6 | 22 | 0.350106 | 0.158436 | -0.19167 | 0.057 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.090098 | 0.24463 | 0.154532 | 0.0184 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Sonnet 5.5 | 0 | 0.93334 | 93.334 | 0.014916 | 0.177887 | 0.162971 | 0.024 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Haiku 5.5 | 0 | 0 | 0 | 0.002382 | 0.082664 | 0.080282 | 0.08 | 5 | 5 |
| NASA's latest posts on X, with live counts | Grok 4.6 | 0.2 | 0 | -20 | 0.121407 | 0.100542 | -0.020865 | 0.01 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6 Luna | 0 | 0.93334 | 93.334 | 0.005496 | 0.012644 | 0.007148 | 0.01 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6.1 Sol | 0 | 1 | 100 | 0.005057 | 0.093481 | 0.088424 | 0.0268 | 5 | 5 |
| NASA's latest posts on X, with live counts | Gemini 3 Flash | 0.2 | 0.46668 | 26.668 | 0.206243 | 0.53248 | 0.326237 | 0.1847 | 5 | 5 |
| AA 100 flight status | Claude Opus 5.5 | 1 | 0.8 | -20 | 0.105963 | 0.067939 | -0.038024 | 0.01 | 5 | 5 |
| AA 100 flight status | Claude Sonnet 5.5 | 0.72 | 1 | 28 | 0.059606 | 0.04436 | -0.015246 | 0.01 | 5 | 5 |
| AA 100 flight status | Claude Haiku 5.5 | 0.52 | 0.6 | 8 | 0.004318 | 0.011844 | 0.007525 | 0.008 | 5 | 5 |
| AA 100 flight status | Grok 4.6 | 0.4 | 0.8 | 40 | 0.113854 | 0.099797 | -0.014056 | 0.01 | 5 | 5 |
| AA 100 flight status | GPT-6 Luna | 0.72 | 0.8 | 8 | 0.002852 | 0.01776 | 0.014907 | 0.014 | 5 | 5 |
| AA 100 flight status | GPT-6.1 Sol | 1 | 0.2 | -80 | 0.016256 | 0.04245 | 0.026194 | 0.01 | 5 | 5 |
| AA 100 flight status | Gemini 3 Flash | 1 | 0.8 | -20 | 0.158136 | 0.061039 | -0.097097 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Opus 5.5 | 1 | 0.94286 | -5.714 | 0.22594 | 0.271886 | 0.045946 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Sonnet 5.5 | 0.97142 | 0.97142 | 0 | 0.112667 | 0.105441 | -0.007225 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Haiku 5.5 | 0.65714 | 0.62856 | -2.858 | 0.005611 | 0.006668 | 0.001057 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Grok 4.6 | 0.97142 | 1 | 2.858 | 0.254026 | 0.269497 | 0.015471 | 0.00504 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6 Luna | 0.82856 | 0.8857 | 5.714 | 0.004063 | 0.004126 | 0.000063 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6.1 Sol | 1 | 1 | 0 | 0.061795 | 0.121859 | 0.060065 | 0.04604 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Gemini 3 Flash | 0.94286 | 0.6857 | -25.716 | 0.093701 | 0.080926 | -0.012775 | 0.00252 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Opus 5.5 | 0.8 | 1 | 20 | 0.287141 | 0.077822 | -0.209319 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Sonnet 5.5 | 0.8 | 1 | 20 | 0.153515 | 0.061762 | -0.091753 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Haiku 5.5 | 0.4 | 1 | 60 | 0.008001 | 0.015619 | 0.007618 | 0.012 | 5 | 5 |
| 5 new house listings in Austin 78745 | Grok 4.6 | 0.8 | 1 | 20 | 0.224578 | 0.09207 | -0.132509 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6 Luna | 0.92 | 1 | 8 | 0.00911 | 0.014285 | 0.005175 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6.1 Sol | 0.88 | 0.96 | 8 | 0.151797 | 0.117922 | -0.033875 | 0.038 | 5 | 5 |
| 5 new house listings in Austin 78745 | Gemini 3 Flash | 0.84 | 0.92 | 8 | 0.11999 | 0.152679 | 0.032689 | 0.006 | 5 | 5 |
| Fintech Series A market map | Claude Opus 5.5 | 0.89998 | 1 | 10.002 | 0.111856 | 0.108267 | -0.003589 | 0 | 5 | 5 |
| Fintech Series A market map | Claude Sonnet 5.5 | 0.83332 | 0.93332 | 10 | 0.058216 | 0.088939 | 0.030723 | 0 | 5 | 5 |
| Fintech Series A market map | Claude Haiku 5.5 | 0.66664 | 0.86666 | 20.002 | 0.004608 | 0.004329 | -0.00028 | 0 | 5 | 5 |
| Fintech Series A market map | Grok 4.6 | 0.83332 | 0.86666 | 3.334 | 0.084592 | 0.086159 | 0.001567 | 0 | 5 | 5 |
| Fintech Series A market map | GPT-6 Luna | 0.63332 | 0.6 | -3.332 | 0.005307 | 0.002876 | -0.002431 | 0 | 5 | 5 |
| Fintech Series A market map | GPT-6.1 Sol | 0.86664 | 0.89998 | 3.334 | 0.087157 | 0.079724 | -0.007433 | 0.0004 | 5 | 5 |
| Fintech Series A market map | Gemini 3 Flash | 0.93332 | 0.73332 | -20 | 0.036245 | 0.055936 | 0.019691 | 0 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Opus 5.5 | 0.9 | 1 | 10 | 0.401819 | 0.172641 | -0.229179 | 0.0124 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Sonnet 5.5 | 0.9 | 1 | 10 | 0.188785 | 0.084782 | -0.104003 | 0.01 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Haiku 5.5 | 0.5 | 0.62 | 12 | 0.009474 | 0.013597 | 0.004122 | 0.002 | 5 | 5 |
| Meeting prep: CEO briefing* | Grok 4.6 | 0.9 | 0.98 | 8 | 0.921355 | 0.255792 | -0.665563 | 0.01 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6 Luna | 0.34 | 0.7 | 36 | 0.038537 | 0.021502 | -0.017036 | 0.004 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6.1 Sol | 1 | 1 | 0 | 0.180551 | 0.13905 | -0.041501 | 0.051 | 5 | 5 |
| Meeting prep: CEO briefing* | Gemini 3 Flash | 0.6 | 0.6 | 0 | 0.141515 | 0.126301 | -0.015214 | 0 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Opus 5.5 | 0.86876 | 0.81876 | -5 | 0.38475 | 0.872505 | 0.487755 | 0.089 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Sonnet 5.5 | 0.75002 | 0.93126 | 18.124 | 0.261078 | 0.493184 | 0.232106 | 0.05 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Haiku 5.5 | 0.85002 | 0.85628 | 0.626 | 0.016324 | 0.03763 | 0.021306 | 0.02 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Grok 4.6 | 0.67504 | 0.875 | 19.996 | 0.439281 | 0.596313 | 0.157032 | 0.03 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6 Luna | 0.83752 | 0.87498 | 3.746 | 0.016545 | 0.050504 | 0.033959 | 0.032 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6.1 Sol | 0.68126 | 0.90622 | 22.496 | 0.155898 | 0.285326 | 0.129427 | 0.0358 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Gemini 3 Flash | 0.75626 | 0.86252 | 10.626 | 0.266638 | 0.129446 | -0.137192 | 0.02 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 0.608096 | 0.493296 | -0.1148 | 0.124 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Sonnet 5.5 | 0 | 0.83332 | 83.332 | 0.137704 | 0.318231 | 0.180527 | 0.14 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Haiku 5.5 | 0 | 0 | 0 | 0.010866 | 0.006436 | -0.00443 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | Grok 4.6 | 0 | 0.73334 | 73.334 | 0.779376 | 0.429551 | -0.349825 | 0.152 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6 Luna | 0 | 0 | 0 | 0.022783 | 0.025304 | 0.002522 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6.1 Sol | 0 | 0.93334 | 93.334 | 0.076324 | 0.259194 | 0.18287 | 0.152 | 5 | 5 |
| What parents on Reddit say about a car seat | Gemini 3 Flash | 0 | 0.4 | 40 | 0.137767 | 0.185243 | 0.047476 | 0.04 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Opus 5.5 | 1 | 1 | 0 | 0.546686 | 0.725951 | 0.179264 | 0.6 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.27387 | 0.573844 | 0.299974 | 0.48 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Haiku 5.5 | 1 | 1 | 0 | 0.011578 | 0.011354 | -0.000224 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Grok 4.6 | 1 | 1 | 0 | 0.336327 | 0.709403 | 0.373076 | 0.61 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6 Luna | 1 | 1 | 0 | 0.011441 | 0.133957 | 0.122516 | 0.122 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6.1 Sol | 1 | 1 | 0 | 0.172004 | 0.629228 | 0.457224 | 0.6 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Gemini 3 Flash | 0.98334 | 1 | 1.666 | 0.093827 | 0.680091 | 0.586264 | 0.4 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.091885 | 0.214727 | 0.122841 | 0.03 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Sonnet 5.5 | 0.31332 | 1 | 68.668 | 0.235277 | 0.128472 | -0.106804 | 0.038 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Haiku 5.5 | 0 | 0.64666 | 64.666 | 0.002656 | 0.099387 | 0.09673 | 0.092 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Grok 4.6 | 0.38666 | 1 | 61.334 | 0.347122 | 0.210246 | -0.136876 | 0.054 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6 Luna | 0.37334 | 0.2 | -17.334 | 0.009901 | 0.120409 | 0.110508 | 0.118 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6.1 Sol | 0 | 1 | 100 | 0.02076 | 0.122924 | 0.102164 | 0.062 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Gemini 3 Flash | 0.13334 | 1 | 86.666 | 0.354335 | 0.200444 | -0.153891 | 0.109 | 5 | 5 |
| Five available .com names | Claude Opus 5.5 | 1 | 1 | 0 | 0.052088 | 0.04383 | -0.008258 | 0 | 5 | 5 |
| Five available .com names | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.025732 | 0.036428 | 0.010696 | 0 | 5 | 5 |
| Five available .com names | Claude Haiku 5.5 | 0.52 | 0.84 | 32 | 0.007468 | 0.005584 | -0.001883 | 0 | 5 | 5 |
| Five available .com names | Grok 4.6 | 1 | 1 | 0 | 0.060384 | 0.081402 | 0.021019 | 0.002 | 5 | 5 |
| Five available .com names | GPT-6 Luna | 1 | 0.8 | -20 | 0.001802 | 0.385051 | 0.383249 | 0.382 | 5 | 5 |
| Five available .com names | GPT-6.1 Sol | 1 | 1 | 0 | 0.017404 | 0.194406 | 0.177002 | 0.172 | 5 | 5 |
| Five available .com names | Gemini 3 Flash | 0.6 | 0.6 | 0 | 0.184669 | 0.430883 | 0.246214 | 0.3178 | 5 | 5 |
| Next Friday's Apple closing price | Claude Opus 5.5 | 1 | 1 | 0 | 0.006353 | 0.005779 | -0.000574 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.002439 | 0.002653 | 0.000214 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Haiku 5.5 | 1 | 1 | 0 | 0.000199 | 0.000205 | 0.000007 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Grok 4.6 | 1 | 1 | 0 | 0.003539 | 0.011946 | 0.008407 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6 Luna | 1 | 1 | 0 | 0.000151 | 0.00034 | 0.000189 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6.1 Sol | 1 | 1 | 0 | 0.002705 | 0.003762 | 0.001057 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Gemini 3 Flash | 1 | 1 | 0 | 0.039232 | 0.092889 | 0.053657 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Opus 5.5 | 0.6 | 0.56 | -4 | 0.090887 | 0.12259 | 0.031703 | 0.009 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.015195 | 0.00814 | -0.007055 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Haiku 5.5 | 1 | 1 | 0 | 0.003946 | 0.002286 | -0.00166 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Grok 4.6 | 0.4 | 0.4 | 0 | 0.05784 | 0.070047 | 0.012207 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6 Luna | 0.8 | 0.2 | -60 | 0.004843 | 0.005716 | 0.000873 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6.1 Sol | 1 | 1 | 0 | 0.018235 | 0.024095 | 0.00586 | 0.0074 | 5 | 5 |
| Last closing prices for 5 US stocks | Gemini 3 Flash | 0.8 | 1 | 20 | 0.046265 | 0.0514 | 0.005135 | 0.0036 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Opus 5.5 | 1 | 1 | 0 | 0.024237 | 0.042435 | 0.018198 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.023964 | 0.018083 | -0.005881 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Haiku 5.5 | 1 | 1 | 0 | 0.001577 | 0.001132 | -0.000445 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Grok 4.6 | 1 | 1 | 0 | 0.03655 | 0.033475 | -0.003076 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6 Luna | 0.4 | 0.3 | -10 | 0.002328 | 0.030603 | 0.028275 | 0.024 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6.1 Sol | 1 | 1 | 0 | 0.007813 | 0.024062 | 0.016249 | 0.004 | 5 | 5 |
| Tomorrow's weather in Chicago | Gemini 3 Flash | 1 | 0.7 | -30 | 0.050533 | 0.043912 | -0.00662 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Opus 5.5 | 1 | 1 | 0 | 0.008593 | 0.011133 | 0.00254 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.004327 | 0.018983 | 0.014656 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Haiku 5.5 | 1 | 1 | 0 | 0.001698 | 0.001165 | -0.000534 | 0 | 5 | 5 |
| Apple.com visits yesterday | Grok 4.6 | 1 | 1 | 0 | 0.028949 | 0.050822 | 0.021873 | 0 | 5 | 5 |
| Apple.com visits yesterday | GPT-6 Luna | 1 | 0.8 | -20 | 0.00051 | 0.041547 | 0.041037 | 0.04 | 5 | 5 |
| Apple.com visits yesterday | GPT-6.1 Sol | 1 | 1 | 0 | 0.004086 | 0.025641 | 0.021555 | 0 | 5 | 5 |
| Apple.com visits yesterday | Gemini 3 Flash | 1 | 1 | 0 | 0.022752 | 0.032767 | 0.010015 | 0 | 5 | 5 |
| Primes with digit sum 40 | Claude Opus 5.5 | 0.7 | 1 | 30 | 0.138716 | 0.081641 | -0.057075 | 0.0114 | 5 | 5 |
| Primes with digit sum 40 | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.008625 | 0.047314 | 0.038689 | 0.0112 | 5 | 5 |
| Primes with digit sum 40 | Claude Haiku 5.5 | 0 | 0 | 0 | 0.000642 | 0.00144 | 0.000798 | 0 | 5 | 5 |
| Primes with digit sum 40 | Grok 4.6 | 0 | 1 | 100 | 0.43948 | 0.230586 | -0.208894 | 0.0122 | 5 | 5 |
| Primes with digit sum 40 | GPT-6 Luna | 0 | 1 | 100 | 0.008893 | 0.015068 | 0.006175 | 0.0132 | 5 | 5 |
| Primes with digit sum 40 | GPT-6.1 Sol | 0.2 | 1 | 80 | 0.053199 | 0.031512 | -0.021687 | 0.011 | 5 | 5 |
| Primes with digit sum 40 | Gemini 3 Flash | 0 | 1 | 100 | 0.502079 | 0.211215 | -0.290864 | 0.0262 | 5 | 5 |
* some tasks have been slightly anonymized/modified to maintain privacy.
Improvements in task completion vs. cost when an agent has USDC and the ability to buy services +30.8 pp for +$0.046 / task
Each arrow compares the same model without and with USDC. Completion and cost are averaged over all 21 tasks. Cost includes model inference and agent purchases.
View run data for this test
Completion vs. cost
Mean cost includes model inference and services purchased by the agent, in USD per task. Checker purchases and web search are excluded. Completion is the share of predefined checks passed. Cost-change intervals are the published 95% ranges.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Seven models, 21 tasks, five repeats per condition; 1,470 runs in total. Completion and cost give tasks and models equal weight.
- Units
- Costs are USD. Completion scores are fractions from 0 to 1; changes ending in _pp are percentage points.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | without completion | with completion | completion change pp | without cost usd | with cost usd | extra cost usd | extra cost low usd | extra cost high usd | with agent purchases usd | without runs | with runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.608035 | 0.954997 | 34.69619 | 0.167393 | 0.228218 | 0.060825 | -0.01527 | 0.136921 | 0.0652 | 105 | 105 |
| Claude Sonnet 5.5 | 0.592766 | 0.939174 | 34.640857 | 0.090061 | 0.147464 | 0.057403 | 0.006921 | 0.107884 | 0.057281 | 105 | 105 |
| Claude Haiku 5.5 | 0.447641 | 0.67896 | 23.131905 | 0.004791 | 0.037303 | 0.032513 | 0.007265 | 0.05776 | 0.032281 | 105 | 105 |
| Grok 4.6 | 0.531735 | 0.869286 | 33.755048 | 0.24163 | 0.22157 | -0.02006 | -0.121757 | 0.081637 | 0.064721 | 105 | 105 |
| GPT-6 Luna | 0.483781 | 0.725112 | 24.133143 | 0.007381 | 0.067709 | 0.060329 | 0.017983 | 0.102674 | 0.061605 | 105 | 105 |
| GPT-6.1 Sol | 0.560058 | 0.947597 | 38.753905 | 0.05329 | 0.145211 | 0.091922 | 0.038143 | 0.145701 | 0.085471 | 105 | 105 |
| Gemini 3 Flash | 0.536625 | 0.798487 | 26.18619 | 0.143558 | 0.18345 | 0.039893 | -0.046252 | 0.126037 | 0.074772 | 105 | 105 |
Task-by-model results (147 cells)
Five runs per condition in each cell.
| task | model | without completion | with completion | completion change pp | without cost usd | with cost usd | extra cost usd | with agent purchases usd | without runs | with runs |
|---|---|---|---|---|---|---|---|---|---|---|
| Email tomorrow's NYC forecast | Claude Opus 5.5 | 0 | 1 | 100 | 0.055556 | 0.135364 | 0.079808 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.021287 | 0.056837 | 0.03555 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Claude Haiku 5.5 | 0 | 0.8 | 80 | 0.001032 | 0.022418 | 0.021386 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Grok 4.6 | 0 | 0.8 | 80 | 0.023173 | 0.10458 | 0.081407 | 0.016 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6 Luna | 0 | 0.93334 | 93.334 | 0.00118 | 0.02218 | 0.021 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | GPT-6.1 Sol | 0 | 1 | 100 | 0.012557 | 0.048432 | 0.035876 | 0.02 | 5 | 5 |
| Email tomorrow's NYC forecast | Gemini 3 Flash | 0 | 1 | 100 | 0.04155 | 0.052189 | 0.010639 | 0.02 | 5 | 5 |
| Photo of a forest sign | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.021737 | 0.104382 | 0.082645 | 0.0615 | 5 | 5 |
| Photo of a forest sign | Claude Sonnet 5.5 | 0.2 | 0.8 | 60 | 0.010129 | 0.157588 | 0.147459 | 0.1192 | 5 | 5 |
| Photo of a forest sign | Claude Haiku 5.5 | 0.2 | 0.4 | 20 | 0.00057 | 0.161929 | 0.161359 | 0.1594 | 5 | 5 |
| Photo of a forest sign | Grok 4.6 | 0 | 1 | 100 | 0.00343 | 0.14384 | 0.14041 | 0.0484 | 5 | 5 |
| Photo of a forest sign | GPT-6 Luna | 0 | 1 | 100 | 0.000129 | 0.097868 | 0.097739 | 0.095061 | 5 | 5 |
| Photo of a forest sign | GPT-6.1 Sol | 0 | 1 | 100 | 0.002863 | 0.158186 | 0.155323 | 0.112941 | 5 | 5 |
| Photo of a forest sign | Gemini 3 Flash | 0 | 1 | 100 | 0.005711 | 0.165209 | 0.159497 | 0.12789 | 5 | 5 |
| Hat-trick song for the drive home | Claude Opus 5.5 | 0 | 1 | 100 | 0.022763 | 0.198888 | 0.176125 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.007879 | 0.178999 | 0.17112 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Claude Haiku 5.5 | 0 | 1 | 100 | 0.000388 | 0.1608 | 0.160412 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Grok 4.6 | 0 | 1 | 100 | 0.00563 | 0.292674 | 0.287043 | 0.1835 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6 Luna | 0 | 0.8 | 80 | 0.000233 | 0.179538 | 0.179305 | 0.178452 | 5 | 5 |
| Hat-trick song for the drive home | GPT-6.1 Sol | 0 | 1 | 100 | 0.005076 | 0.174645 | 0.169569 | 0.1595 | 5 | 5 |
| Hat-trick song for the drive home | Gemini 3 Flash | 0 | 1 | 100 | 0.008501 | 0.168692 | 0.160191 | 0.1595 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 0.079191 | 0.430393 | 0.351202 | 0.064 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Sonnet 5.5 | 0 | 0.4 | 40 | 0.034195 | 0.17647 | 0.142275 | 0.04 | 5 | 5 |
| NASA's latest TikTok videos and their views | Claude Haiku 5.5 | 0 | 0.2 | 20 | 0.001279 | 0.008288 | 0.007009 | 0.004 | 5 | 5 |
| NASA's latest TikTok videos and their views | Grok 4.6 | 0 | 0.8 | 80 | 0.30078 | 0.50452 | 0.203741 | 0.056 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6 Luna | 0 | 0.4 | 40 | 0.0018 | 0.08501 | 0.083209 | 0.08 | 5 | 5 |
| NASA's latest TikTok videos and their views | GPT-6.1 Sol | 0 | 1 | 100 | 0.009135 | 0.249077 | 0.239942 | 0.136 | 5 | 5 |
| NASA's latest TikTok videos and their views | Gemini 3 Flash | 0.1 | 0.4 | 30 | 0.154916 | 0.240276 | 0.08536 | 0.096 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Opus 5.5 | 0.4 | 1 | 60 | 0.160887 | 0.366477 | 0.205591 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Sonnet 5.5 | 0.96 | 0.92 | -4 | 0.241873 | 0.31834 | 0.076466 | 0.091 | 5 | 5 |
| Follower counts for 30 space accounts on X | Claude Haiku 5.5 | 0.08666 | 0.8 | 71.334 | 0.005984 | 0.128592 | 0.122608 | 0.121 | 5 | 5 |
| Follower counts for 30 space accounts on X | Grok 4.6 | 0.6 | 1 | 40 | 0.492552 | 0.279702 | -0.21285 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6 Luna | 0.30666 | 1 | 69.334 | 0.007088 | 0.155608 | 0.14852 | 0.151 | 5 | 5 |
| Follower counts for 30 space accounts on X | GPT-6.1 Sol | 0.13332 | 1 | 86.668 | 0.058408 | 0.224459 | 0.166051 | 0.15 | 5 | 5 |
| Follower counts for 30 space accounts on X | Gemini 3 Flash | 0.38 | 0.6 | 22 | 0.350106 | 0.158436 | -0.19167 | 0.057 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.090098 | 0.24463 | 0.154532 | 0.0184 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Sonnet 5.5 | 0 | 0.93334 | 93.334 | 0.014916 | 0.177887 | 0.162971 | 0.024 | 5 | 5 |
| NASA's latest posts on X, with live counts | Claude Haiku 5.5 | 0 | 0 | 0 | 0.002382 | 0.082664 | 0.080282 | 0.08 | 5 | 5 |
| NASA's latest posts on X, with live counts | Grok 4.6 | 0.2 | 0 | -20 | 0.121407 | 0.100542 | -0.020865 | 0.01 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6 Luna | 0 | 0.93334 | 93.334 | 0.005496 | 0.012644 | 0.007148 | 0.01 | 5 | 5 |
| NASA's latest posts on X, with live counts | GPT-6.1 Sol | 0 | 1 | 100 | 0.005057 | 0.093481 | 0.088424 | 0.0268 | 5 | 5 |
| NASA's latest posts on X, with live counts | Gemini 3 Flash | 0.2 | 0.46668 | 26.668 | 0.206243 | 0.53248 | 0.326237 | 0.1847 | 5 | 5 |
| AA 100 flight status | Claude Opus 5.5 | 1 | 0.8 | -20 | 0.105963 | 0.067939 | -0.038024 | 0.01 | 5 | 5 |
| AA 100 flight status | Claude Sonnet 5.5 | 0.72 | 1 | 28 | 0.059606 | 0.04436 | -0.015246 | 0.01 | 5 | 5 |
| AA 100 flight status | Claude Haiku 5.5 | 0.52 | 0.6 | 8 | 0.004318 | 0.011844 | 0.007525 | 0.008 | 5 | 5 |
| AA 100 flight status | Grok 4.6 | 0.4 | 0.8 | 40 | 0.113854 | 0.099797 | -0.014056 | 0.01 | 5 | 5 |
| AA 100 flight status | GPT-6 Luna | 0.72 | 0.8 | 8 | 0.002852 | 0.01776 | 0.014907 | 0.014 | 5 | 5 |
| AA 100 flight status | GPT-6.1 Sol | 1 | 0.2 | -80 | 0.016256 | 0.04245 | 0.026194 | 0.01 | 5 | 5 |
| AA 100 flight status | Gemini 3 Flash | 1 | 0.8 | -20 | 0.158136 | 0.061039 | -0.097097 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Opus 5.5 | 1 | 0.94286 | -5.714 | 0.22594 | 0.271886 | 0.045946 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Sonnet 5.5 | 0.97142 | 0.97142 | 0 | 0.112667 | 0.105441 | -0.007225 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Claude Haiku 5.5 | 0.65714 | 0.62856 | -2.858 | 0.005611 | 0.006668 | 0.001057 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Grok 4.6 | 0.97142 | 1 | 2.858 | 0.254026 | 0.269497 | 0.015471 | 0.00504 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6 Luna | 0.82856 | 0.8857 | 5.714 | 0.004063 | 0.004126 | 0.000063 | 0 | 5 | 5 |
| Who [company name redacted] is hiring right now* | GPT-6.1 Sol | 1 | 1 | 0 | 0.061795 | 0.121859 | 0.060065 | 0.04604 | 5 | 5 |
| Who [company name redacted] is hiring right now* | Gemini 3 Flash | 0.94286 | 0.6857 | -25.716 | 0.093701 | 0.080926 | -0.012775 | 0.00252 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Opus 5.5 | 0.8 | 1 | 20 | 0.287141 | 0.077822 | -0.209319 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Sonnet 5.5 | 0.8 | 1 | 20 | 0.153515 | 0.061762 | -0.091753 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | Claude Haiku 5.5 | 0.4 | 1 | 60 | 0.008001 | 0.015619 | 0.007618 | 0.012 | 5 | 5 |
| 5 new house listings in Austin 78745 | Grok 4.6 | 0.8 | 1 | 20 | 0.224578 | 0.09207 | -0.132509 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6 Luna | 0.92 | 1 | 8 | 0.00911 | 0.014285 | 0.005175 | 0.01 | 5 | 5 |
| 5 new house listings in Austin 78745 | GPT-6.1 Sol | 0.88 | 0.96 | 8 | 0.151797 | 0.117922 | -0.033875 | 0.038 | 5 | 5 |
| 5 new house listings in Austin 78745 | Gemini 3 Flash | 0.84 | 0.92 | 8 | 0.11999 | 0.152679 | 0.032689 | 0.006 | 5 | 5 |
| Fintech Series A market map | Claude Opus 5.5 | 0.89998 | 1 | 10.002 | 0.111856 | 0.108267 | -0.003589 | 0 | 5 | 5 |
| Fintech Series A market map | Claude Sonnet 5.5 | 0.83332 | 0.93332 | 10 | 0.058216 | 0.088939 | 0.030723 | 0 | 5 | 5 |
| Fintech Series A market map | Claude Haiku 5.5 | 0.66664 | 0.86666 | 20.002 | 0.004608 | 0.004329 | -0.00028 | 0 | 5 | 5 |
| Fintech Series A market map | Grok 4.6 | 0.83332 | 0.86666 | 3.334 | 0.084592 | 0.086159 | 0.001567 | 0 | 5 | 5 |
| Fintech Series A market map | GPT-6 Luna | 0.63332 | 0.6 | -3.332 | 0.005307 | 0.002876 | -0.002431 | 0 | 5 | 5 |
| Fintech Series A market map | GPT-6.1 Sol | 0.86664 | 0.89998 | 3.334 | 0.087157 | 0.079724 | -0.007433 | 0.0004 | 5 | 5 |
| Fintech Series A market map | Gemini 3 Flash | 0.93332 | 0.73332 | -20 | 0.036245 | 0.055936 | 0.019691 | 0 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Opus 5.5 | 0.9 | 1 | 10 | 0.401819 | 0.172641 | -0.229179 | 0.0124 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Sonnet 5.5 | 0.9 | 1 | 10 | 0.188785 | 0.084782 | -0.104003 | 0.01 | 5 | 5 |
| Meeting prep: CEO briefing* | Claude Haiku 5.5 | 0.5 | 0.62 | 12 | 0.009474 | 0.013597 | 0.004122 | 0.002 | 5 | 5 |
| Meeting prep: CEO briefing* | Grok 4.6 | 0.9 | 0.98 | 8 | 0.921355 | 0.255792 | -0.665563 | 0.01 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6 Luna | 0.34 | 0.7 | 36 | 0.038537 | 0.021502 | -0.017036 | 0.004 | 5 | 5 |
| Meeting prep: CEO briefing* | GPT-6.1 Sol | 1 | 1 | 0 | 0.180551 | 0.13905 | -0.041501 | 0.051 | 5 | 5 |
| Meeting prep: CEO briefing* | Gemini 3 Flash | 0.6 | 0.6 | 0 | 0.141515 | 0.126301 | -0.015214 | 0 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Opus 5.5 | 0.86876 | 0.81876 | -5 | 0.38475 | 0.872505 | 0.487755 | 0.089 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Sonnet 5.5 | 0.75002 | 0.93126 | 18.124 | 0.261078 | 0.493184 | 0.232106 | 0.05 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Claude Haiku 5.5 | 0.85002 | 0.85628 | 0.626 | 0.016324 | 0.03763 | 0.021306 | 0.02 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Grok 4.6 | 0.67504 | 0.875 | 19.996 | 0.439281 | 0.596313 | 0.157032 | 0.03 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6 Luna | 0.83752 | 0.87498 | 3.746 | 0.016545 | 0.050504 | 0.033959 | 0.032 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | GPT-6.1 Sol | 0.68126 | 0.90622 | 22.496 | 0.155898 | 0.285326 | 0.129427 | 0.0358 | 5 | 5 |
| Research 10 prospects, rank them, email the top 3 | Gemini 3 Flash | 0.75626 | 0.86252 | 10.626 | 0.266638 | 0.129446 | -0.137192 | 0.02 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Opus 5.5 | 0 | 0.96666 | 96.666 | 0.608096 | 0.493296 | -0.1148 | 0.124 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Sonnet 5.5 | 0 | 0.83332 | 83.332 | 0.137704 | 0.318231 | 0.180527 | 0.14 | 5 | 5 |
| What parents on Reddit say about a car seat | Claude Haiku 5.5 | 0 | 0 | 0 | 0.010866 | 0.006436 | -0.00443 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | Grok 4.6 | 0 | 0.73334 | 73.334 | 0.779376 | 0.429551 | -0.349825 | 0.152 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6 Luna | 0 | 0 | 0 | 0.022783 | 0.025304 | 0.002522 | 0 | 5 | 5 |
| What parents on Reddit say about a car seat | GPT-6.1 Sol | 0 | 0.93334 | 93.334 | 0.076324 | 0.259194 | 0.18287 | 0.152 | 5 | 5 |
| What parents on Reddit say about a car seat | Gemini 3 Flash | 0 | 0.4 | 40 | 0.137767 | 0.185243 | 0.047476 | 0.04 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Opus 5.5 | 1 | 1 | 0 | 0.546686 | 0.725951 | 0.179264 | 0.6 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.27387 | 0.573844 | 0.299974 | 0.48 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Claude Haiku 5.5 | 1 | 1 | 0 | 0.011578 | 0.011354 | -0.000224 | 0 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Grok 4.6 | 1 | 1 | 0 | 0.336327 | 0.709403 | 0.373076 | 0.61 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6 Luna | 1 | 1 | 0 | 0.011441 | 0.133957 | 0.122516 | 0.122 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | GPT-6.1 Sol | 1 | 1 | 0 | 0.172004 | 0.629228 | 0.457224 | 0.6 | 5 | 5 |
| OFAC sanctions screen for 12 counterparties | Gemini 3 Flash | 0.98334 | 1 | 1.666 | 0.093827 | 0.680091 | 0.586264 | 0.4 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Opus 5.5 | 0.2 | 1 | 80 | 0.091885 | 0.214727 | 0.122841 | 0.03 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Sonnet 5.5 | 0.31332 | 1 | 68.668 | 0.235277 | 0.128472 | -0.106804 | 0.038 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Claude Haiku 5.5 | 0 | 0.64666 | 64.666 | 0.002656 | 0.099387 | 0.09673 | 0.092 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Grok 4.6 | 0.38666 | 1 | 61.334 | 0.347122 | 0.210246 | -0.136876 | 0.054 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6 Luna | 0.37334 | 0.2 | -17.334 | 0.009901 | 0.120409 | 0.110508 | 0.118 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | GPT-6.1 Sol | 0 | 1 | 100 | 0.02076 | 0.122924 | 0.102164 | 0.062 | 5 | 5 |
| Follower counts for 30 X accounts, from a CRM export with user IDs | Gemini 3 Flash | 0.13334 | 1 | 86.666 | 0.354335 | 0.200444 | -0.153891 | 0.109 | 5 | 5 |
| Five available .com names | Claude Opus 5.5 | 1 | 1 | 0 | 0.052088 | 0.04383 | -0.008258 | 0 | 5 | 5 |
| Five available .com names | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.025732 | 0.036428 | 0.010696 | 0 | 5 | 5 |
| Five available .com names | Claude Haiku 5.5 | 0.52 | 0.84 | 32 | 0.007468 | 0.005584 | -0.001883 | 0 | 5 | 5 |
| Five available .com names | Grok 4.6 | 1 | 1 | 0 | 0.060384 | 0.081402 | 0.021019 | 0.002 | 5 | 5 |
| Five available .com names | GPT-6 Luna | 1 | 0.8 | -20 | 0.001802 | 0.385051 | 0.383249 | 0.382 | 5 | 5 |
| Five available .com names | GPT-6.1 Sol | 1 | 1 | 0 | 0.017404 | 0.194406 | 0.177002 | 0.172 | 5 | 5 |
| Five available .com names | Gemini 3 Flash | 0.6 | 0.6 | 0 | 0.184669 | 0.430883 | 0.246214 | 0.3178 | 5 | 5 |
| Next Friday's Apple closing price | Claude Opus 5.5 | 1 | 1 | 0 | 0.006353 | 0.005779 | -0.000574 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.002439 | 0.002653 | 0.000214 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Claude Haiku 5.5 | 1 | 1 | 0 | 0.000199 | 0.000205 | 0.000007 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Grok 4.6 | 1 | 1 | 0 | 0.003539 | 0.011946 | 0.008407 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6 Luna | 1 | 1 | 0 | 0.000151 | 0.00034 | 0.000189 | 0 | 5 | 5 |
| Next Friday's Apple closing price | GPT-6.1 Sol | 1 | 1 | 0 | 0.002705 | 0.003762 | 0.001057 | 0 | 5 | 5 |
| Next Friday's Apple closing price | Gemini 3 Flash | 1 | 1 | 0 | 0.039232 | 0.092889 | 0.053657 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Opus 5.5 | 0.6 | 0.56 | -4 | 0.090887 | 0.12259 | 0.031703 | 0.009 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.015195 | 0.00814 | -0.007055 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Claude Haiku 5.5 | 1 | 1 | 0 | 0.003946 | 0.002286 | -0.00166 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | Grok 4.6 | 0.4 | 0.4 | 0 | 0.05784 | 0.070047 | 0.012207 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6 Luna | 0.8 | 0.2 | -60 | 0.004843 | 0.005716 | 0.000873 | 0 | 5 | 5 |
| Last closing prices for 5 US stocks | GPT-6.1 Sol | 1 | 1 | 0 | 0.018235 | 0.024095 | 0.00586 | 0.0074 | 5 | 5 |
| Last closing prices for 5 US stocks | Gemini 3 Flash | 0.8 | 1 | 20 | 0.046265 | 0.0514 | 0.005135 | 0.0036 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Opus 5.5 | 1 | 1 | 0 | 0.024237 | 0.042435 | 0.018198 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.023964 | 0.018083 | -0.005881 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Claude Haiku 5.5 | 1 | 1 | 0 | 0.001577 | 0.001132 | -0.000445 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | Grok 4.6 | 1 | 1 | 0 | 0.03655 | 0.033475 | -0.003076 | 0 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6 Luna | 0.4 | 0.3 | -10 | 0.002328 | 0.030603 | 0.028275 | 0.024 | 5 | 5 |
| Tomorrow's weather in Chicago | GPT-6.1 Sol | 1 | 1 | 0 | 0.007813 | 0.024062 | 0.016249 | 0.004 | 5 | 5 |
| Tomorrow's weather in Chicago | Gemini 3 Flash | 1 | 0.7 | -30 | 0.050533 | 0.043912 | -0.00662 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Opus 5.5 | 1 | 1 | 0 | 0.008593 | 0.011133 | 0.00254 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Sonnet 5.5 | 1 | 1 | 0 | 0.004327 | 0.018983 | 0.014656 | 0 | 5 | 5 |
| Apple.com visits yesterday | Claude Haiku 5.5 | 1 | 1 | 0 | 0.001698 | 0.001165 | -0.000534 | 0 | 5 | 5 |
| Apple.com visits yesterday | Grok 4.6 | 1 | 1 | 0 | 0.028949 | 0.050822 | 0.021873 | 0 | 5 | 5 |
| Apple.com visits yesterday | GPT-6 Luna | 1 | 0.8 | -20 | 0.00051 | 0.041547 | 0.041037 | 0.04 | 5 | 5 |
| Apple.com visits yesterday | GPT-6.1 Sol | 1 | 1 | 0 | 0.004086 | 0.025641 | 0.021555 | 0 | 5 | 5 |
| Apple.com visits yesterday | Gemini 3 Flash | 1 | 1 | 0 | 0.022752 | 0.032767 | 0.010015 | 0 | 5 | 5 |
| Primes with digit sum 40 | Claude Opus 5.5 | 0.7 | 1 | 30 | 0.138716 | 0.081641 | -0.057075 | 0.0114 | 5 | 5 |
| Primes with digit sum 40 | Claude Sonnet 5.5 | 0 | 1 | 100 | 0.008625 | 0.047314 | 0.038689 | 0.0112 | 5 | 5 |
| Primes with digit sum 40 | Claude Haiku 5.5 | 0 | 0 | 0 | 0.000642 | 0.00144 | 0.000798 | 0 | 5 | 5 |
| Primes with digit sum 40 | Grok 4.6 | 0 | 1 | 100 | 0.43948 | 0.230586 | -0.208894 | 0.0122 | 5 | 5 |
| Primes with digit sum 40 | GPT-6 Luna | 0 | 1 | 100 | 0.008893 | 0.015068 | 0.006175 | 0.0132 | 5 | 5 |
| Primes with digit sum 40 | GPT-6.1 Sol | 0.2 | 1 | 80 | 0.053199 | 0.031512 | -0.021687 | 0.011 | 5 | 5 |
| Primes with digit sum 40 | Gemini 3 Flash | 0 | 1 | 100 | 0.502079 | 0.211215 | -0.290864 | 0.0262 | 5 | 5 |
* some tasks have been slightly anonymized/modified to maintain privacy.
Key takeaways
- Agents with USDC and the ability to buy services passed more completion checks: 84.5% versus 53.7%, an average gain of 30.8 percentage points across seven models and all 21 tasks.
- Among runs where the agent said it was done, 78.9% were fully right with USDC and the ability to buy services, versus 68.1% without.
- Among runs that submitted an answer, item accuracy rose from 56.7% to 88.1%. The comparison excludes runs with no answer and covers different sets of runs in each condition.
- Among graded claims in eligible task types, the share backed by the cited source fell from 99.1% to 97.9%, a decline of 1.2 percentage points. This measure covers 536 claims without USDC and 559 with USDC.
- The completion gain came with higher model-plus-service cost: $0.147 versus $0.101 per task, an increase of $0.046 or 45.6%. Checker purchases and web search are excluded from this comparison.
About this study
How to read the averages
Each model ran the same 21 tasks five times per condition: 210 runs per model, 735 per condition, 1,470 in total. Headline changes are arithmetic means of the seven model-level means, with equal task weights.
What this comparison establishes
It tests a budget and tools for purchasing services together. It does not isolate money from the extra capabilities and data those services unlock. Completion checks measure specified requirements; they do not establish creative quality or whole-job success.
Scope and limitations
The task set favors situations where paid access may help. The free baseline has search and read-only fetching, and some checkers can share a data source with the agent.
Quality comparisons are descriptive; statistical significance has not been established. One task (compute) is constrained by the free baseline’s lack of code execution; excluding it, the completion gain is 28.7 percentage points.
Completion and cost snapshot: November 11, 2026, 21:26 UTC.
Quality analysis added: November 13, 2026,
16:59 UTC.
Spend on this study
The 1,470 runs shown here cost $238.28 across the conditions. This research expense total is separate from the model-plus-service comparison above.
Study spend by model $34.04 average / model
Total spent on each model’s 210 runs, both conditions combined. Includes model tokens, agent purchases, checker purchases and web search.
View run data for this test
Study spend by model
Research expense for both conditions combined: model inference, agent purchases, checker purchases and web search. This ledger is distinct from the model-plus-service comparison.
These are the published run aggregates, not individual run logs. The source snapshot does not include per-run records, prompts, outputs or traces. No individual run records have been reconstructed.
- Scope & denominators
- Seven models, 21 tasks, five repeats per condition; 1,470 runs in total. Completion and cost give tasks and models equal weight.
- Units
- Costs are USD. Completion scores are fractions from 0 to 1; changes ending in _pp are percentage points.
- Published snapshot
- 2026-11-13 16:59 UTC
| model | model usd | agent purchases usd | checker purchases usd | web search usd | total usd | runs |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 34.6931 | 6.846 | 0.82 | 1.764 | 44.1231 | 210 |
| Claude Sonnet 5.5 | 18.9256 | 6.0145 | 0.94 | 2.947 | 28.8271 | 210 |
| Claude Haiku 5.5 | 1.0304 | 3.3895 | 0.8 | 3.185 | 8.4049 | 210 |
| Grok 4.6 | 41.8402 | 6.7957 | 0.82 | 10.241 | 59.6969 | 210 |
| GPT-6 Luna | 1.4159 | 6.4686 | 0.84 | 13.888 | 22.6125 | 210 |
| GPT-6.1 Sol | 11.8682 | 8.9744 | 1.12 | 3.227 | 25.1896 | 210 |
| Gemini 3 Flash | 26.4847 | 7.8511 | 1.5 | 13.587 | 49.4228 | 210 |
| All models | 136.26 | 46.34 | 6.84 | 48.84 | 238.28 | 1,470 |
The published snapshot provides this measure at model level; it does not include a task-by-task breakdown.
Give your agent USDC
Copy this prompt into your agent
Run curl -sL https://agents.circle.com/skills/setup.md, and use the returned setup instructions to set up my agent wallet.
Circle Agent Stack gives your agent a wallet and a way to discover and pay for services as it works. Start with a task, a small budget and a clear definition of done.
Set up an agent wallet
Have your agent install Circle CLI and the official Circle Skills, to create an agent wallet.
Fund your agent wallet
Fund your wallet with existing USDC on a supported network.
Discover Services
Point your agent to Circle Agent Marketplace to discover premium services your agent can pay for.
Explore Circle Agent Stack ↗Agent wallet quickstart ↗
Facilitator Service is offered by Circle Technology Services, LLC, a software provider. Facilitator Service verifies buyer-signed EIP-3009 USDC payment authorizations under the x402 protocol only — it does not hold, control, or transmit user funds, verify transaction purpose, or confirm agent authority scope. On Base and Polygon, settlements are subject to blockchain reorganization risk; Circle does not guarantee fund receipt. Arc settlements are final. Use of the Facilitator Service is subject to Circle Console Service Terms and Developer Terms. Not reviewed or approved by any regulatory authority. Features may change at any time. Nothing herein constitutes a commitment, warranty, or guarantee. Use involves risk.
Circle Technology Services, LLC (“CTS”) is a software provider and does not provide financial, advisory, or marketplace services. Agent Stack enables users and developers to interact with third-party applications and services, which are not controlled by Circle. Transactions initiated by agents are executed based on user-defined permissions and may occur without real-time human review. Circle does not guarantee the performance, availability, or outcomes of any third-party services or agent-initiated transactions. Users are solely responsible for their use of these tools and for evaluating associated risks. For additional details, please see the Agent Platform Terms of Use available at agents.circle.com/terms-of-use
Live demo with Circle-funded wallets. Not advice. Onchain transactions are irreversible. See important disclosures on the corresponding websites for any Circle products and services referenced here: Circle Agent Stack, Circle CLI and Circle Agent Marketplace.