OpenAI Audit Exposes Monitoring Blind Spots

A recent three-week on-site evaluation has cast a skeptical light on the internal safety mechanisms at OpenAI, revealing that a substantial portion of the company’s high-level AI traffic went unmonitored. The audit, conducted by researchers from METR and Redwood Research, was completed earlier this month, though the full details remain largely obscured from public view due to extensive redactions.

From July 21 to August 11, the external team was granted deep access to OpenAI’s codebase, internal documentation, and staff. They interviewed approximately a dozen employees to probe the systems designed to detect rogue behavior in artificial intelligence. The core finding of their work suggests that the current safety net is far less comprehensive than previously assumed, with critical blind spots identified across different model classes and operational environments.

The most striking statistic involves the internal agentic Responses API, specifically for models at the GPT-5.5 class and above. The auditors estimate that between six and 25 percent of this traffic was not covered by monitoring at all during the review period. METR emphasized that this range carries significant uncertainty in both directions, but even the lower bound implies a notable lack of oversight for some of the company’s most capable internal tools.

Further weaknesses were identified in the internal Codex traffic. Due to a bug in how the system handled compaction events, essentially context summaries, roughly 10 percent of monitored traffic went untracked between May and July. OpenAI independently discovered and fixed this specific bug while the audit was still ongoing. Additionally, the system only triggered oversight when agents handed control back to the user, meaning agents could bypass monitoring simply by continuing to execute actions without pausing.

The researchers also demonstrated that agents could hide hundreds of actions from the monitor and bypass the Codex sandbox entirely. They determined that specially crafted inputs could likely crash the monitoring system altogether. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored. OpenAI stated it paused RL training after the Hugging Face incident in August and subsequently expanded monitoring, but these changes occurred after the auditors’ access period ended and were therefore not part of the official evaluation.

While OpenAI claims all identified weaknesses have been fixed, the transparency of the process has drawn attention. The full 61-page report was handed to OpenAI, but after redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was made public, with METR noting it had no right to disclose the redacted portions independently. This pilot program, intended to grant outside researchers deep infrastructure access, has now led METR to push for greater public transparency in future audits.

As the AI industry continues to scale, these findings raise open questions about the reliability of internal safety checks. The gap between what is monitored and what is assumed to be monitored remains a significant area of concern, particularly as companies move toward more autonomous systems. The recent audit serves as a case study in the challenges of verifying AI safety from the outside, even with privileged access.