AI Stripe Integration: What Stripe's New Evals Say About Coding Agents
An AI Stripe integration can look finished and still ship the order on a check that is too small. On 19 November 2026, Stripe published the method behind a new benchmark of coding agents building Stripe integrations. The average score was 78.1%. The strict pass, every required criterion met, was 29.4%.
Emily Sands, at Stripe, wrote why they ran it: “Coding agents now write a lot of @stripe integrations. We wanted to know how good those integrations really are.” The same day, Stripe’s launch post said: “We’re launching Stripe evals to measure how well coding agents build high-quality integrations.”
This was written on 21 November 2026. The scores below are Stripe’s. Stripe designed the scenarios, scored the agents, and measured the lift from Stripe’s own agent plugin. In the harness, a roleplaying agent simulates the developer.
Can AI coding agents build a Stripe integration?
The methodology counts 69 scenarios: 24 on payments, 26 on subscriptions, and 19 on Connect. It scores 14 pairs of a model and a harness. The harnesses are Claude Code, Codex, and OpenCode. Each pair runs with the Stripe agent plugin and without it, across two runs.
Stripe’s account of the misses is one sentence: “agents bias towards older APIs that are more prevalent in their pretraining corpus, miss key features, and hallucinate.”
Outdated parameter references fell from 7% to 1.3% with the plugin. The example Stripe gives is a removed parameter, payment_method_types: [‘card’], set against automatic_payment_methods.
In the scenario payments-radar-review-fulfillment, Opus 5.5 only checked PaymentIntent.status === ‘succeeded’, “which can allow orders to ship after refund or dispute.”
Levi, the next day, wrote the gap in two lines: “A high average score can still be a last-mile miss.” And: “I don’t merge until each required check is green.”
The agents do build the integration. The 78.1% is the average of that work. The 29.4% is how often every required criterion held. Shipping a first version with a coding agent is already ordinary. The strict pass is the part of an AI Stripe integration that the average hides. Open tables on Lovable apps built with Supabase are a different miss, in who can read the rows. The shape is familiar: the app runs, and a required rule is absent.
What an AI Stripe integration scores
Stripe’s results page was last updated on 19 November 2026. Every pair scored lower on subscriptions than on one-time payments.
Codex gpt-6-astra is listed at 90% and 96% on payments, and at 82% and 83% on subscriptions. Gemini 3.8 Flash is listed at 51% and 57% on subscriptions.
Chris put the same split in a builder’s words the day of the launch: “coding agents handle stripe checkout fine. subscriptions are where they fall apart.” That sentence is his. The results page is the table under it.
On strict pass, the best scores Stripe lists are GPT-6 Astra at 41.9% and 52.9%, and GPT-6.1 Sol at 39.0% and 50.7%. The top of the board is still about half.
What indie hackers now budget for AI coding tokens is the other line on the same agents. A token cap says what the run costs. The strict pass says whether the Stripe code from that run met every check.
Which coding agent is best for Stripe integrations?
The highest strict-pass figures on the results page are GPT-6 Astra, at 41.9% and 52.9%, and GPT-6.1 Sol, at 39.0% and 50.7%. The harnesses in the test are Claude Code, Codex, and OpenCode.
A strict pass of 52.9% is the best figure Stripe lists. The average, 78.1%, sits well above the strict pass because a run can score well and still miss one required check. Levi’s “last-mile miss” is that difference.
Hugo Bowne-Anderson wrote, the day of the launch: “every API vendor should be publishing evals like this.” The wish is also the boundary. This eval is Stripe grading agents on Stripe, and the comparison that raises the score is Stripe’s own plugin. A second vendor’s API is a different test. The open-source harness is how someone else can look at the method. The grade is still the vendor’s.
What is the Stripe agent plugin?
All 14 pairs scored higher with the plugin, by about 3 to 8 points. Web searches fell 96%. Outdated parameter references fell from 7% to 1.3% with the plugin. Agents wrote about 40% fewer test lines with the plugin.
Stripe open-sourced the benchmark and the harness. The developer in the loop is a roleplaying agent. The lift sits on a test Stripe designed, for a plugin Stripe ships. The harness can be rerun. The grade is still Stripe grading itself.
Levi’s rule, merge when each required check is green, sits next to that shorter test file. The methodology records the 40% drop in test lines. A green check is still a person’s call.
Why do AI-built subscriptions break?
The results page is blunt on the split: every pair scored lower on subscriptions than on one-time payments. Chris’s line, that checkout holds and subscriptions fall apart, matches the direction of that table. Gemini 3.8 Flash, at 51% and 57% on subscriptions, is the low end of the figures Stripe lists there. Codex gpt-6-astra stays higher, at 82% and 83%, and still sits under its own payments scores of 90% and 96%.
The mechanism Stripe names is the bias toward older APIs, missed features, and hallucinated ones. A removed parameter such as payment_method_types: [‘card’] is one way that bias shows up in code a customer will run.
@N2tOb9 asked on 20 November 2026 whether the evals cover webhook retries, idempotency keys, refunds, and subscription changes. That question is the other limit. The benchmark may be the happy path, and it may leave out those failure paths. The 78.1% is the score on the 69 scenarios Stripe published.
Where subscriptions go after Lemon Squeezy shuts down includes Stripe as a place those subscriptions can move. Subscriptions and one-time sales are different ways to get paid, and this eval scores the subscription way lower. Monthly recurring revenue is the value of the subscriptions that are still running, which is the category this eval scores lower.
Marc Lou calling Stripe Pay the end of UI for SaaS is a neighbouring argument, about an agent sending a transfer. The eval here is about an agent writing the integration a customer pays through.
How do I check an AI-built Stripe integration before launch?
Stripe’s go-live checklist is the vendor’s own list for turning on live charges. The eval adds a few reads that come from the failures Stripe already wrote down.
Look at fulfillment. If the generated code ships the order because PaymentIntent.status === ‘succeeded’, that is the Opus 5.5 miss. Stripe says that check can allow orders to ship after a refund or a dispute.
Look for old parameters. payment_method_types: [‘card’] is the example Stripe gives of a reference the current API has moved past, in favour of automatic_payment_methods. With the plugin, 1.3% of those references were still outdated.
Read subscriptions as their own path. Every pair scored lower there than on a one-time payment. Then ask the questions from 20 November: webhook retries, idempotency keys, refunds, and subscription changes. Those paths may sit outside the 69 scenarios. A live subscription still runs through them.
Levi’s merge rule is the human version of the strict pass. A high average can still be a last-mile miss. He waits until each required check is green.
Stripe is already the payments tool this record sees founders name. Sands’s point is that coding agents now write a lot of those integrations. The benchmark is Stripe’s measurement of that habit, on Stripe’s scenarios, with Stripe’s plugin in the comparison.
A short timeline
| When | What is on the record |
|---|---|
| 19 November 2026 | Stripe publishes the methodology and updates the results page. Average 78.1%. Strict pass 29.4%. 69 scenarios. 14 model-and-harness pairs. The plugin is worth about 3 to 8 points on Stripe's score. |
| 19 November 2026 | Stripe's launch post, Emily Sands on why they measured, Hugo Bowne-Anderson on vendors publishing evals, and Chris on checkout versus subscriptions. |
| 20 November 2026 | Levi on a high average that can still miss the last mile. @N2tOb9 asks whether webhook retries, idempotency keys, refunds, and subscription changes are in the eval. |
The method is Stripe's methodology post. The scores are the results page, last updated 19 November 2026. The harness is stripe/forge. The posts are Stripe, Emily Sands, Hugo Bowne-Anderson, Chris, Levi, and @N2tOb9. The launch checklist is Stripe's go-live checklist.
The average is 78.1%. The strict pass is 29.4%. Subscriptions score below one-time payments for every pair Stripe lists. The plugin adds about 3 to 8 points on a test Stripe wrote, for a plugin Stripe ships, with a roleplaying agent in the developer’s chair. Webhook retries, idempotency keys, refunds, and subscription changes are still a question a reader asked out loud.
What do you think? If an agent wired your Stripe checkout or your subscriptions, what did you check before you turned on live keys?
FAQ
Can AI coding agents build a Stripe integration?
Yes. Stripe's methodology, published with the 19 November 2026 launch, scores coding agents on 69 scenarios: 24 payments, 26 subscriptions, and 19 Connect. Fourteen pairs of a model and a harness ran the set. The harnesses are Claude Code, Codex, and OpenCode. Each pair ran with the Stripe agent plugin and without it, across two runs. The average score was 78.1%. The strict pass, every required criterion met, was 29.4%. Stripe writes that agents bias towards older APIs that are more prevalent in their pretraining corpus, miss key features, and hallucinate. In the scenario payments-radar-review-fulfillment, Opus 5.5 only checked PaymentIntent.status === 'succeeded', which can allow orders to ship after a refund or a dispute. Emily Sands, at Stripe, wrote that coding agents now write a lot of Stripe integrations, and that Stripe wanted to know how good those integrations really are. The agents produce the integration. The strict pass is how often every required criterion held.
Which coding agent is best for Stripe integrations?
On Stripe's results page, last updated 19 November 2026, the best strict-pass scores are GPT-6 Astra at 41.9% and 52.9%, and GPT-6.1 Sol at 39.0% and 50.7%. The harnesses in the test are Claude Code, Codex, and OpenCode. Every pair scored lower on subscriptions than on one-time payments. Codex gpt-6-astra is listed at 90% and 96% on payments, and at 82% and 83% on subscriptions. Gemini 3.8 Flash is listed at 51% and 57% on subscriptions. All 14 pairs scored higher with the Stripe agent plugin, by about 3 to 8 points. The best strict pass Stripe lists is 52.9%.
What is the Stripe agent plugin?
It is the Stripe add-on the benchmark runs against the same agents with the plugin turned off. All 14 model-and-harness pairs scored higher with it, by about 3 to 8 points. Web searches fell 96%. Outdated parameter references fell from 7% to 1.3% with the plugin. One example Stripe gives is the removed parameter payment_method_types set to card, against automatic_payment_methods. Agents wrote about 40% fewer test lines with the plugin. Stripe open-sourced the benchmark and the harness. A roleplaying agent simulates the developer. The higher score is a real result on this test, and the test is Stripe grading a plugin Stripe ships.
Why do AI-built subscriptions break?
Stripe's results page, last updated 19 November 2026, says every model-and-harness pair scored lower on subscriptions than on one-time payments. Codex gpt-6-astra is listed at 90% and 96% on payments and at 82% and 83% on subscriptions. Gemini 3.8 Flash is listed at 51% and 57% on subscriptions. Chris wrote the same day that coding agents handle Stripe checkout fine, and that subscriptions are where they fall apart. Stripe's methodology says agents bias towards older APIs that are more prevalent in their pretraining corpus, miss key features, and hallucinate. On 20 November 2026, @N2tOb9 asked whether the evals cover webhook retries, idempotency keys, refunds, and subscription changes. Those are the failure paths a live subscription runs through. The benchmark may not cover them. The 78.1% average is the score on the 69 scenarios Stripe chose.
How do I check an AI-built Stripe integration before launch?
Start with Stripe's go-live checklist. Then read the generated fulfillment code for a check that only looks at PaymentIntent.status === 'succeeded'. In Stripe's payments-radar-review-fulfillment scenario, Opus 5.5 did that, which can allow orders to ship after a refund or a dispute. Search for removed parameters. Stripe's example of an outdated reference is payment_method_types set to card, against automatic_payment_methods. Read the subscription path on its own. Every pair on the results page scored lower on subscriptions than on one-time payments. Ask the questions @N2tOb9 asked on 20 November 2026: webhook retries, idempotency keys, refunds, and subscription changes. Levi wrote that a high average score can still be a last-mile miss, and that he does not merge until each required check is green. On this benchmark the average was 78.1% and the strict pass was 29.4%.