Anthropic, Gorgias and Pylon Show AI Agent Scores Depend on Who Measures Them
Anthropic says buyers its sales agent escalates become opportunities more than twice as often as form leads. On Gorgias's benchmark, Intercom's agent resolves 42% of conversations; Intercom advertises 76%.
October 7, 2026
Not long ago, a company that wanted to buy Claude for its staff filled out Anthropic's Contact Sales form and sometimes waited days for a reply. Usually it only wanted to know what a plan cost or whether there was a seat minimum. On September 30, Anthropic's sales development lead, Carl Johnson, described the replacement. It’s a buying agent built on Claude Managed Agents that sits on the pricing and Contact Sales pages, inside the product and in email, and holds thousands of conversations a day. Buyers who get escalated from it to a rep turn into opportunities more than twice as often as leads from the old form did, and they close about five days faster.
Six days later, SaaStr's Jason Lemkin published the October numbers from Gorgias's public benchmark of AI support agents. On the board's own four-week table, the top vendors resolve between 64% and 72% of conversations with no human involved. The median vendor resolves 46%, Intercom 42% and Zendesk 37%.
Side by side, the two show that AI agents doing revenue work now get judged on a published number rather than a demo. The two numbers also show that the score depends on who did the measuring and what they agreed to count as a win. Pylon's CEO, Marty Kausas, launched Nexus 1.0 on October 6 as an AI harness for B2B support teams that routes work across frontier and open-weight models. He promised benchmarks "in future versions." Until those arrive, a buyer can set the claim aside.
Anthropic graded its agent against the form it replaced
Read Anthropic's post for its baseline. The 2x compares the leads the agent escalates with leads from the old form, the slowest version of inbound Anthropic had. Tens of thousands of monthly requests had outgrown that queue, and the BDRs working it answered questions the documentation already covered. The agent is also opt-in, so buyers choose at the start whether to talk to it or to a rep. Anyone who picks the agent and then gets handed to a person has already self-selected and been sorted by deal size, with the pricing and security questions out of the way. A higher opportunity rate is what that filter should produce, whatever the agent's skill.
The number can be right and still mostly measure the funnel. The post is also candid about two things that cut both ways. The share of conversations that needed a person to close has fallen by about half, which shrinks the pool of escalated leads at the same time as it improves their quality. And the agent often steers small teams to the Team plan rather than Enterprise. Anthropic calls that a feature, on retention grounds, but it means revenue per inbound lead can fall while opportunity rate rises. The post gives the rate and the speed, but it doesn't give the revenue.
The metric Anthropic chose is still the right one. When SaaStr wrote up its own inbound agent in June, built on Qualified, the headline was 614 meetings booked for one event, and the post conceded those meetings didn't all close. Meetings booked is the number an inbound agent can inflate most easily. Opportunity rate and days to close sit one stage nearer to cash. That’s the comparison worth copying, with a control group in place of a retired form.
Gorgias's board puts Intercom 34 points below its advertised rate
The support benchmark makes the measurement problem visible because it puts a vendor's own claim next to a common test. Gorgias runs every agent through the same 10-turn conversation on live mid-market ecommerce stores, 228 of them by the board's current count. A blind model judge scores the answers against 26 checks. A conversation counts as automated only if no human touched it and the agent didn't push the shopper out of the chat to email or a phone line. On that test Intercom's agent resolved 42%. Intercom's Fin site advertises a 76% average resolution rate across more than 12,000 customers.
Both figures can be honest. They count different things. SaaStr notes that Gorgias's own customer reporting counts a conversation as automated once 72 hours pass without a human stepping in, so a shopper who gave up and left counts as a resolution there and as a failure on the benchmark. Gorgias charges $0.90 per resolved conversation, SaaStr adds, which makes the definition of resolved the pricing unit as well as the performance claim.
The board has holes of its own. Gorgias wrote the rubric and chose the weights, and it ranks fifth on automation on its own page. SaaStr's fund led Gorgias's seed round. The stores are all ecommerce, where "where's my order" dominates the queue, so a B2B support team's mix will score differently. The vendor list has shifted too. SaaStr's September launch write-up counted 18 vendors, and the October one ranks 13. SaaStr's figures pool four weeks, while the board says its published ranking runs on 90 days and warns that shorter windows strip out evidence. And nine of the ten vendors with enough history lost automation over those four weeks, so a pilot rate is a reading for one month. Unlike a vendor's homepage, the board publishes the rubric and the transcripts, so a buyer can grade their own agent the same way.
Klarna judged its agent on cost and later paid for it in quality
The case against reading either number as a forecast starts with transfer. Anthropic's inbound demand is unusual, and the product the agent sells is the model the agent runs on. A mid-market SaaS vendor with a thinner queue and a weaker agent shouldn't expect 2x. Gorgias's results come from shoppers asking about returns, and B2B admins asking about SSO are a different test.
Klarna is the stronger version of the objection. In February 2024 it said its AI assistant had handled 2.3 million conversations in its first month. That was two-thirds of its service chats, the work of 700 full-time agents. Those were published numbers too. By May 2025 its CEO, Sebastian Siemiatkowski, was telling Bloomberg the company was hiring people back. Cost had been too dominant a factor in how the program was judged, he said, and the quality had suffered. The volume figures were never shown to be false. They measured the thing that was easy to count. In growth marketing, enso raised its Series A on a similar turn, with a named person now running the agents it sells.
So this fall's numbers mostly show what buyers will now demand. A published number has become the price of a seat at the table. What it's worth depends on the baseline and definition behind it, and on how long the window ran.
What this means for revenue and support leaders
For an inbound agent, run it against a holdout of live traffic that still reaches a rep quickly, and skip the comparison with last year's form. Count opportunity rate, days to close and revenue per lead, so a cheaper-plan recommendation shows up as the trade it is. Meetings booked can sit on the dashboard but shouldn't decide the rollout.
For a support agent, write the definition of resolved into the pilot contract before the pilot starts, and make it the same definition the vendor bills on. Then grade a sample of your own tickets with a public rubric like the one Gorgias posts, weighted by your ticket mix. Do it again a month later, since nine vendors out of ten moved down in four weeks.
When a vendor launches the way Pylon did, ask for its rate on your tickets under your definition before the harness gets a slot in the evaluation. My bet is that the next funded AI support launch ships with a head-to-head score against named competitors, because Gorgias has shown a vendor can rank fifth on its own board and still lead the conversation about it. If Pylon's first benchmarked Nexus reports only its own number under its own definition, the category hasn't caught up yet.
A published number has become the price of a seat at the table. What it's worth depends on the baseline and definition behind it, and on how long the window ran.
Write the definition of resolved into the pilot contract before a support agent pilot starts, make it the one the vendor bills on, and regrade a sample of your own tickets a month later.
