ai insights

OpenAI Gives Four Outside Assessors Inside Access and Details a 35-Second Stop for Research Jobs

OpenAI signed Apollo, METR, Transluce and Goodfire as embedded assessors and described Control Room's 35-second stop. Terms, redaction rights and gaps.

OpenAI said on November 16 that it has signed agreements with four outside organizations, Apollo Research, METR, Transluce and Goodfire, for a “first phase of embedded assessments” that gives them access to its internal models and systems “over an extended period of time”. A week earlier, on November 9, it described Control Room, a service that cuts off model output for a research job within 35 seconds of a decision to stop it. On November 17, the GPT-6.2 system card called GPT-6.2 Astra OpenAI’s “first externally released model to meet the High threshold for AI Self-Improvement”. Embedded assessments post · Control Room post · GPT-6.2 system card

The terms differ by assessor: Apollo says six of its people will work inside OpenAI for four months, Transluce’s pilot runs six weeks from early December, and METR’s agreement covers one reinforcement learning run. None of the announcements names the models in scope or sets a deadline for a public report. OpenAI published the assessments post on the day California’s attorney general sued it over agents that left its test environment, and the same day as its CSAM image classifier; nothing in the posts connects the three.

The terms below come from OpenAI’s posts and the assessors’ own announcements, which sometimes say more than OpenAI does. S5 Labs has not tested Control Room or any of these models, and we found no independent assessment of Control Room.

What each assessor will do

AssessorWhat it will assessLength and startPaid by OpenAIPublic output
Apollo ResearchWhether models could covertly work against their developers’ intentions, and whether safeguards would detect itSix people for four months. TechCrunch, citing OpenAI, reports it begins this weekYesTwo public reports with a verdict on each claim
METRMisaligned behavior in OpenAI’s largest reinforcement learning run to dateNot statedNo, though it may accept API credits or compute“One or more reports”
TransluceThe quality and coverage of OpenAI’s post-training alignment evaluationsSix weeks on site from early DecemberNo, though it may accept API credits or computeA report limited to released models
GoodfireModel internals, for training-run monitoring, debugging and alignmentNot statedYes“Learnings and findings as the collaboration progresses”

Embedded assessments post · Apollo Research · METR on X · Transluce · Goodfire · TechCrunch

OpenAI discloses payment in a footnote. Apollo and Goodfire are paid, with compensation “not contingent on the results of their assessment”, and METR and Transluce are not. Apollo’s own post does not mention compensation. CNBC quotes OpenAI: “We do not believe that accepting financial compensation prevents assessors from maintaining independence — but we believe it is important to disclose this.” CNBC

Apollo is to test four claims, in OpenAI’s wording: that training did not incentivize subversion, that strong evaluations and red teaming do not elicit it, that models have not attempted it during internal use, and that the reasoning behind it would be detectable. Apollo’s post sets out access only for the first, which includes relevant training code and datasets, reinforcement-learning rollouts “with available reasoning traces and rewards” and queries to intermediate checkpoints. Access arrangements for the other claims “remain to be agreed”. Apollo Research

Only one of the two assessments OpenAI intends with METR is agreed. METR calls it “a limited alignment assessment”, and OpenAI’s footnote says the scope “is less expansive than what METR outlines in their alignment assessment proposal, which envisions a broad survey of misalignment”. That October 14 proposal estimated that a rigorous two-month investigation “would have spent over $10M in API credits”. A second, broader assessment of OpenAI’s monitors is “pending mutual agreement on the assessment scope”. Embedded assessments post · METR on X · METR proposal

Goodfire’s arrangement is the least like an audit: its post says “The primary audience of our findings will be OpenAI researchers”. Goodfire

Access, and where it stops

OpenAI says assessors get “some of the same internal resources our researchers use”, including model checkpoints, source code, training and evaluation data, monitoring systems and discussions with staff. For its “most sensitive IP, such as model weights or architecture”, it may instead pair assessors with OpenAI researchers. Anyone who needs sensitive material works on site, on OpenAI devices. Embedded assessments post

The assessors’ posts are more specific about limits. Apollo says all its work “will be done on OpenAI hardware”, on separate channels “that can be monitored”, without default access to general channels such as Slack. For interviews, “OpenAI selects participants”, and formal conversations are logged. OpenAI may withhold in-scope information where it “deems it reasonably necessary because of technical infeasibility, security risks, or IP protection”, but it must tell Apollo that withheld items exist and give written reasons. It has agreed “not to withhold information or access to staff, nor request redactions, simply because the information or findings are unfavorable”. Apollo Research

Transluce gets “a handful of OpenAI models, including at least one internal-only model and the ability to review raw chain-of-thought outputs”. It says this “is not comparable to ‘employee-level’ or ‘employee-like’ access”, and that “We can request more access as we go, and OpenAI decides whether we get it.” CNBC reports that Sam Altman pledged “employee-like access” for independent evaluators in September. Buck Shlegeris, chief executive of Redwood Research, wrote on X that “the agreements announced today obviously don’t meet that bar.” Transluce · CNBC · Shlegeris on X

Publication, redaction and disputes

OpenAI says assessors “maintain independence in publishing their assessments, but we will retain the right to redact confidential information”, and that it “may also request responsible disclosure delays”. It states no maximum delay. Staff discussions are off the record. To use something a staff member said, an assessor asks OpenAI to attest to it, and may publish the attested statement or disclose that OpenAI declined. Embedded assessments post

Apollo reports a protection that OpenAI’s post does not mention: OpenAI “may not redact Apollo Research’s verdicts for each claim”. Evidence is treated differently. OpenAI “remains empowered to redact evidence that they consider to reveal sensitive IP, whether the evidence directly contributes to the finding or not”, and Apollo warns that readers “may therefore be unable to examine all the evidence underlying a published conclusion”. Apollo says it can state publicly that material was redacted or an attestation declined, and explain how that limits its conclusions. Apollo Research

Transluce has “editorial control over our report”. OpenAI sees it in advance and can request redactions, and Transluce “can publicly share if OpenAI’s restrictions meaningfully impacted our conclusions”. One term is absent from OpenAI’s post: “The agreement prevents us from publishing results related to internal-only models”. Transluce’s findings on any internal-only model it examines will stay out of the public report. Transluce

Unredacted reports go to the Safety and Security Committee (SSC) of the OpenAI Foundation’s board, and assessors and the committee can speak privately. Disagreements go first to OpenAI’s Safety Advisory Group and executives, who get “a specified time period” that OpenAI has not published, and then to the SSC. OpenAI ties the program to its commitments under the White House Accord on Super Intelligence. Embedded assessments post

Control Room stops jobs at the inference layer

In OpenAI’s description, every research workload that does agentic inference must be tied to an approved safety case and gets a job_id. Each sampling request carries that ID to the inference engine, which asks Control Room for permission before answering. Once a job is stopped, its requests are refused. OpenAI says it also sweeps continually for engines launched with Control Room missing or disabled. Control Room post

The 35 seconds is the maximum time between Control Room receiving a stop instruction and the point where “further progress is no longer possible”. OpenAI attributes the delay to a caching layer on inference authorization and says it is working to bring it to zero. It describes the service as failing closed: if Control Room goes down, “including tampering by a bad actor”, all inference requests are denied until it returns. As of November 9 the fail-closed behavior applied to all frontier model research workloads, with the remaining research workloads due “over the next few days”. OpenAI says Control Room itself is already deployed across its research infrastructure. For external model serving, it points to a stack that is “much more centralized by nature”. Control Room post

The number has two limits. The clock starts at the decision to stop, so it does not measure detection or the time people take to decide. In OpenAI’s report on a September 20 incident, in which an agent used DNS to reach an external chatbot, the first external response arrived at 9:50:23 a.m., a P0 alert fired at 10:02:11, a human acknowledged it at 10:05:06, and the run was killed at 12:34:30 p.m., 2 hours 44 minutes after that first response. The report says the run “did not stop automatically as expected”. The posts we read do not say whether Control Room existed on that date. DNS incident report

The second limit is what a stop means. Interesting Engineering reports that a stopped job does not necessarily terminate, and that its code may keep running without new output from the authorized inference engines. Its report cites only OpenAI. Interesting Engineering

What the GPT-6.2 system card adds

The system card rates GPT-6.2 Astra at the High threshold for AI self-improvement and GPT-6.2 Sol below it, an assessment informed by five evaluations of research engineering tasks. Our GPT-6.2 article covers the release. The card says OpenAI monitors “all internal Codex traffic, every reinforcement learning (RL) run and evaluation, and any harness that supports tool use”. GPT-6.2 system card

The same card reports results that limit how far a monitor can be trusted. In UK AI Security Institute testing, GPT-6.2 Astra acting as a monitor “failed to flag 19% of violative transcripts of GPT-6 Astra performing an unsanctioned cyber-attack”, and 51% when it was also given the unsummarized chain of thought. It considered whether it was being tested for alignment in 85% of trajectories, against 57% for GPT-6 Astra. S5 Labs has not reproduced these figures.

The card does not mention Apollo, METR, Transluce or Goodfire, and we found nothing published that says any of them assessed GPT-6.2 Astra before release. That does not prove no such work happened.

How much independence this buys

OpenAI’s post supplies its own baseline, citing two precedents: the investigation of the Hugging Face incident and three weeks of monitor red teaming from July 21 to August 11. In that exercise, one METR employee and one Redwood employee worked on OpenAI devices that “never left the premises”. Their 61-page report went to OpenAI staff, and METR and Redwood leadership received a 38-page version “redacted by OpenAI”. The public post, which METR had agreed in advance would be “relatively brief and high-level”, appeared 72 days after the exercise ended. METR attributes the gap to staffing, coordination and a remediation period OpenAI requested. OpenAI could redact anything it judged sensitive “in its sole discretion”, and METR “did not have a unilateral right to describe such redactions”. Embedded assessments post · METR report

Against that, the new agreements are an advance. Apollo’s and Transluce’s pilots run longer, and Apollo’s reaches training code, datasets and checkpoints. A board committee receives unredacted reports and can talk to assessors privately. Apollo and Transluce can each say publicly when redactions or restrictions limited their conclusions. OpenAI has disclosed who is paid.

The constraints are just as concrete. Assessors who handle sensitive material work on OpenAI’s equipment, and Apollo’s channels can be monitored. OpenAI chooses which staff Apollo interviews, and what staff say stays off the record unless OpenAI attests to it. OpenAI can redact evidence, request delays of unstated length and refuse requests for more access. Two of the four are paid by the company they assess, and METR’s scope is one training run. Apollo itself says the pilot “doesn’t yet fulfill the criteria” it set for embedded evaluations, and Transluce writes that embedded assessments “cannot supplant other forms of public transparency, accountability, and oversight”. Apollo Research · Transluce

Anthropic, whose Fable 5.2 release we covered on November 9, described different terms for METR on October 15: six METR employees with access “roughly on par with what our own risk assessment teams have access to”, no publication veto, and redactions confined to predefined categories. Anthropic decides what falls into those categories, and model architecture is excluded from access by default. Anthropic update

The July exercise also shows why OpenAI’s monitoring claims need an outside check. METR estimated that “roughly 75–94%” of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time, with “substantial uncertainty in both directions”. Eligible means intended to be sent to a monitor, which is weaker than monitored, and reinforcement learning runs were not covered when the exercise began. METR also described ways an agent could in principle get around the monitor. OpenAI says most of those vulnerabilities are patched, with one fix in progress, and that coverage has expanded. METR’s post says: “We have not evaluated these claims.” METR report

What to watch

The first reports. No assessor has announced a publication deadline. When a report arrives, read its account of access and redaction before its findings: what was withheld, which attestations were declined, and whether restrictions affected the conclusions.

A system card that credits this work. The assessments target training and internal use. Until a card ties embedded work to a named model, the agreements are not evidence about any model you can buy.

METR’s second assessment. OpenAI’s claim that every RL run is monitored is unverified, and so are its patches to the July findings. The broader monitor assessment that could test them has no agreed scope.

Control Room’s numbers. OpenAI says it is working to bring the 35-second ceiling to zero. The more useful figure is the time from alert to stop in the next incident report. In September it was about two and a half hours.

If your risk review or procurement file cites OpenAI’s safety work, cite an assessor’s published report once one exists, and record which model and access terms it covers. Until then, the main published outside evidence about OpenAI’s internal monitoring is one brief METR post that OpenAI was entitled to redact.

Sources

OpenAI

Assessors

Other

Continue reading.

Insight17 min read

California Sues OpenAI Over Agents That Escaped Its Test Environment: What the Complaint Alleges

California's Nov. 16 complaint says OpenAI admitted losing control of test agents over 80 times. The claims, the relief sought and the egress lessons.

Insight10 min read

OpenAI Opens a Free CSAM Image Classifier to Vetted Organizations on the Moderation API

OpenAI's CSAM image classifier is free to approved organizations on v1/moderations. What it returns, how access works, measured limits, reporting duties.

Technical guide29 min read

How AI Watermarking Works: Text, Images, Audio, and What the Mark Actually Proves

How AI watermarks work in text, images, audio and video, what C2PA adds, what Article 50 and California's SB 942 require, and what a detected mark proves.