Coding Benchmark Reliability Concerns and Bun's Rust Rewrite
코딩/개발 | Thu Jul 09 2026 00:00:00 GMT+0000 (Coordinated Universal Time) | 3 sources
OpenAI uncovered flaws in SWE-Bench Pro, while Bun transitioned to Rust following its acquisition by Anthropic.
Analysis
[OpenAI] released audit findings on SWE-Bench Pro benchmark flaws [1][2]
- Estimated approximately 30% of tasks contain flaws
- Datapoint analysis pipeline flagged 200 (27.4%) flawed tasks
- Human review campaign identified 249 (34.1%) flawed tasks
- Five experienced software engineers conducted independent review
[SWE-Bench Pro] identified four major flaw categories [1]
- Overly strict tests invalidating functionally correct submissions
- Hidden requirements not inferable from under-specified prompts
- Low-coverage tests passing incomplete fixes
- Misleading prompts directing wrong approaches
[OpenAI] proposed an agent-based data quality verification methodology [1][2]
- Utilized multi investigator-agent pass approach
- Reviewed model attempts
- task metadata
- and failure traces
- Emphasized the utility of agents in scalable data quality checks
- Used in safety decisions under the OpenAI Preparedness Framework
[Bun] rewrote codebase from Zig to Rust [3]
- Project started with Zig in April 2021
- Grew to 22 million monthly downloads
- Adopted as runtime by Claude Code and OpenCode
- Used pre-release Claude Fable 5 for Rust rewrite
[Anthropic] acquired Bun and supported Rust rewrite with Claude models [3]
- Completed acquisition of Bun in December 2025
- Bun team joined Anthropic
- Pre-release version of Claude Fable 5 used for Rust rewrite
- Vercel
- Railway
- and DigitalOcean provide 1st-party support for Bun
[Bun v1.3.14] fixed numerous memory safety bugs [3]
- Fixed use-after-free crash in node:zlib
- Resolved use-after-free related to node:http2 reentrant callbacks
- Fixed vulnerability related to ArrayBuffer detach in UDPSocket.send()
- Resolved memory leaks in crypto.scrypt and tlsSocket.setSession()