Claude 3.5 Sonnet at a glance
The upgraded Claude 3.5 Sonnet rose from 33.4% to 49.0% on SWE-bench Verified at the same price, and became the first frontier model offered in public beta to operate a computer through screenshots, mouse and keyboard.
What changed with Claude 3.5 Sonnet
On October 22, 2024, Anthropic released an upgraded Claude 3.5 Sonnet at the same price and speed as before. Its score on SWE-bench Verified, a test of fixing real GitHub issues, rose from 33.4% to 49.0%, which Anthropic said was higher than any publicly available model. It also announced Claude 3.5 Haiku, scoring 40.6% on the same test, for release later that month.
The headline was computer use, in public beta: Claude could look at a screen, move a cursor, click and type to operate software the way a person does. Anthropic reported 14.9% on OSWorld in the screenshot-only category, versus 7.8% for the next-best system, and described the feature as still experimental, at times cumbersome and error-prone. It was available on the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI. Asana, Canva, Cognition, DoorDash, Replit and The Browser Company were early testers.
Claude 3.5 Sonnet in numbers
Values are copied from the linked source on the date shown. They are not IA Compa measurements.
- SWE-bench Verified (upgraded Claude 3.5 Sonnet)as of · Source for SWE-bench Verified (upgraded Claude 3.5 Sonnet): anthropic.com (opens in a new tab)
- 49.0%
Where to compare Claude 3.5 Sonnet
Independent leaderboards and official resources for comparing this model yourself. Rankings change often; each site shows its own current data.
General leaderboards
Official resources
Why Claude 3.5 Sonnet matters for businesses
For businesses, computer use was the first practical sign of agents that could work in tools without an API, such as legacy back-office systems. At launch it was slow and needed close supervision, but it set the direction for browser and desktop agents.
Put it to work: AI Agents
Sources for this story
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (opens in a new tab)Official
Anthropic · anthropic.com
- Claude 3.5 Sonnet (Oct) model page (opens in a new tab)Benchmark
Artificial Analysis · artificialanalysis.ai
Last verified: . IA Compa summarizes public announcements for business readers. We are not affiliated with the companies mentioned. Always confirm details, pricing, and availability with the official source.
- #llm
- #anthropic
- #claude
- #computer-use
- #agents
- #coding
