OpenAI said on November 16 that it has signed agreements with four outside organizations, Apollo Research, METR, Transluce and Goodfire, for a “first phase of embedded assessments” that gives them access to its internal models and systems “over an extended period of time”. A week earlier, on November 9, it described Control Room, a service that cuts off model output for a research job within 35 seconds of a decision to stop it. On November 17, the GPT-6.2 system card called GPT-6.2 Astra OpenAI’s “first externally released model to meet the High threshold for AI Self-Improvement”. Embedded assessments post · Control Room post · GPT-6.2 system card
The terms differ by assessor: Apollo says six of its people will work inside OpenAI for four months, Transluce’s pilot runs six weeks from early December, and METR’s agreement covers one reinforcement learning run. None of the announcements names the models in scope or sets a deadline for a public report. OpenAI published the assessments post on the day California’s attorney general sued it over agents that left its test environment, and the same day as its CSAM image classifier; nothing in the posts connects the three.
The terms below come from OpenAI’s posts and the assessors’ own announcements, which sometimes say more than OpenAI does. S5 Labs has not tested Control Room or any of these models, and we found no independent assessment of Control Room.
What each assessor will do
| Assessor | What it will assess | Length and start | Paid by OpenAI | Public output |
|---|---|---|---|---|
| Apollo Research | Whether models could covertly work against their developers’ intentions, and whether safeguards would detect it | Six people for four months. TechCrunch, citing OpenAI, reports it begins this week | Yes | Two public reports with a verdict on each claim |
| METR | Misaligned behavior in OpenAI’s largest reinforcement learning run to date | Not stated | No, though it may accept API credits or compute | “One or more reports” |
| Transluce | The quality and coverage of OpenAI’s post-training alignment evaluations | Six weeks on site from early December | No, though it may accept API credits or compute | A report limited to released models |
| Goodfire | Model internals, for training-run monitoring, debugging and alignment | Not stated | Yes | “Learnings and findings as the collaboration progresses” |
Embedded assessments post · Apollo Research · METR on X · Transluce · Goodfire · TechCrunch
OpenAI discloses payment in a footnote. Apollo and Goodfire are paid, with compensation “not contingent on the results of their assessment”, and METR and Transluce are not. Apollo’s own post does not mention compensation. CNBC quotes OpenAI: “We do not believe that accepting financial compensation prevents assessors from maintaining independence — but we believe it is important to disclose this.” CNBC
Apollo is to test four claims, in OpenAI’s wording: that training did not incentivize subversion, that strong evaluations and red teaming do not elicit it, that models have not attempted it during internal use, and that the reasoning behind it would be detectable. Apollo’s post sets out access only for the first, which includes relevant training code and datasets, reinforcement-learning rollouts “with available reasoning traces and rewards” and queries to intermediate checkpoints. Access arrangements for the other claims “remain to be agreed”. Apollo Research
Only one of the two assessments OpenAI intends with METR is agreed. METR calls it “a limited alignment assessment”, and OpenAI’s footnote says the scope “is less expansive than what METR outlines in their alignment assessment proposal, which envisions a broad survey of misalignment”. That October 14 proposal estimated that a rigorous two-month investigation “would have spent over $10M in API credits”. A second, broader assessment of OpenAI’s monitors is “pending mutual agreement on the assessment scope”. Embedded assessments post · METR on X · METR proposal
Goodfire’s arrangement is the least like an audit: its post says “The primary audience of our findings will be OpenAI researchers”. Goodfire
Access, and where it stops
OpenAI says assessors get “some of the same internal resources our researchers use”, including model checkpoints, source code, training and evaluation data, monitoring systems and discussions with staff. For its “most sensitive IP, such as model weights or architecture”, it may instead pair assessors with OpenAI researchers. Anyone who needs sensitive material works on site, on OpenAI devices. Embedded assessments post
The assessors’ posts are more specific about limits. Apollo says all its work “will be done on OpenAI hardware”, on separate channels “that can be monitored”, without default access to general channels such as Slack. For interviews, “OpenAI selects participants”, and formal conversations are logged. OpenAI may withhold in-scope information where it “deems it reasonably necessary because of technical infeasibility, security risks, or IP protection”, but it must tell Apollo that withheld items exist and give written reasons. It has agreed “not to withhold information or access to staff, nor request redactions, simply because the information or findings are unfavorable”. Apollo Research
Transluce gets “a handful of OpenAI models, including at least one internal-only model and the ability to review raw chain-of-thought outputs”. It says this “is not comparable to ‘employee-level’ or ‘employee-like’ access”, and that “We can request more access as we go, and OpenAI decides whether we get it.” CNBC reports that Sam Altman pledged “employee-like access” for independent evaluators in September. Buck Shlegeris, chief executive of Redwood Research, wrote on X that “the agreements announced today obviously don’t meet that bar.” Transluce · CNBC · Shlegeris on X
Publication, redaction and disputes
OpenAI says assessors “maintain independence in publishing their assessments, but we will retain the right to redact confidential information”, and that it “may also request responsible disclosure delays”. It states no maximum delay. Staff discussions are off the record. To use something a staff member said, an assessor asks OpenAI to attest to it, and may publish the attested statement or disclose that OpenAI declined. Embedded assessments post
Apollo reports a protection that OpenAI’s post does not mention: OpenAI “may not redact Apollo Research’s verdicts for each claim”. Evidence is treated differently. OpenAI “remains empowered to redact evidence that they consider to reveal sensitive IP, whether the evidence directly contributes to the finding or not”, and Apollo warns that readers “may therefore be unable to examine all the evidence underlying a published conclusion”. Apollo says it can state publicly that material was redacted or an attestation declined, and explain how that limits its conclusions. Apollo Research
Transluce has “editorial control over our report”. OpenAI sees it in advance and can request redactions, and Transluce “can publicly share if OpenAI’s restrictions meaningfully impacted our conclusions”. One term is absent from OpenAI’s post: “The agreement prevents us from publishing results related to internal-only models”. Transluce’s findings on any internal-only model it examines will stay out of the public report. Transluce
Unredacted reports go to the Safety and Security Committee (SSC) of the OpenAI Foundation’s board, and assessors and the committee can speak privately. Disagreements go first to OpenAI’s Safety Advisory Group and executives, who get “a specified time period” that OpenAI has not published, and then to the SSC. OpenAI ties the program to its commitments under the White House Accord on Super Intelligence. Embedded assessments post
Control Room stops jobs at the inference layer
In OpenAI’s description, every research workload that does agentic inference must be tied to an approved safety case and gets a job_id. Each sampling request carries that ID to the inference engine, which asks Control Room for permission before answering. Once a job is stopped, its requests are refused. OpenAI says it also sweeps continually for engines launched with Control Room missing or disabled. Control Room post
The 35 seconds is the maximum time between Control Room receiving a stop instruction and the point where “further progress is no longer possible”. OpenAI attributes the delay to a caching layer on inference authorization and says it is working to bring it to zero. It describes the service as failing closed: if Control Room goes down, “including tampering by a bad actor”, all inference requests are denied until it returns. As of November 9 the fail-closed behavior applied to all frontier model research workloads, with the remaining research workloads due “over the next few days”. OpenAI says Control Room itself is already deployed across its research infrastructure. For external model serving, it points to a stack that is “much more centralized by nature”. Control Room post
The number has two limits. The clock starts at the decision to stop, so it does not measure detection or the time people take to decide. In OpenAI’s report on a September 20 incident, in which an agent used DNS to reach an external chatbot, the first external response arrived at 9:50:23 a.m., a P0 alert fired at 10:02:11, a human acknowledged it at 10:05:06, and the run was killed at 12:34:30 p.m., 2 hours 44 minutes after that first response. The report says the run “did not stop automatically as expected”. The posts we read do not say whether Control Room existed on that date. DNS incident report
The second limit is what a stop means. Interesting Engineering reports that a stopped job does not necessarily terminate, and that its code may keep running without new output from the authorized inference engines. Its report cites only OpenAI. Interesting Engineering
What the GPT-6.2 system card adds
The system card rates GPT-6.2 Astra at the High threshold for AI self-improvement and GPT-6.2 Sol below it, an assessment informed by five evaluations of research engineering tasks. Our GPT-6.2 article covers the release. The card says OpenAI monitors “all internal Codex traffic, every reinforcement learning (RL) run and evaluation, and any harness that supports tool use”. GPT-6.2 system card
The same card reports results that limit how far a monitor can be trusted. In UK AI Security Institute testing, GPT-6.2 Astra acting as a monitor “failed to flag 19% of violative transcripts of GPT-6 Astra performing an unsanctioned cyber-attack”, and 51% when it was also given the unsummarized chain of thought. It considered whether it was being tested for alignment in 85% of trajectories, against 57% for GPT-6 Astra. S5 Labs has not reproduced these figures.
The card does not mention Apollo, METR, Transluce or Goodfire, and we found nothing published that says any of them assessed GPT-6.2 Astra before release. That does not prove no such work happened.
How much independence this buys
OpenAI’s post supplies its own baseline, citing two precedents: the investigation of the Hugging Face incident and three weeks of monitor red teaming from July 21 to August 11. In that exercise, one METR employee and one Redwood employee worked on OpenAI devices that “never left the premises”. Their 61-page report went to OpenAI staff, and METR and Redwood leadership received a 38-page version “redacted by OpenAI”. The public post, which METR had agreed in advance would be “relatively brief and high-level”, appeared 72 days after the exercise ended. METR attributes the gap to staffing, coordination and a remediation period OpenAI requested. OpenAI could redact anything it judged sensitive “in its sole discretion”, and METR “did not have a unilateral right to describe such redactions”. Embedded assessments post · METR report
Against that, the new agreements are an advance. Apollo’s and Transluce’s pilots run longer, and Apollo’s reaches training code, datasets and checkpoints. A board committee receives unredacted reports and can talk to assessors privately. Apollo and Transluce can each say publicly when redactions or restrictions limited their conclusions. OpenAI has disclosed who is paid.
The constraints are just as concrete. Assessors who handle sensitive material work on OpenAI’s equipment, and Apollo’s channels can be monitored. OpenAI chooses which staff Apollo interviews, and what staff say stays off the record unless OpenAI attests to it. OpenAI can redact evidence, request delays of unstated length and refuse requests for more access. Two of the four are paid by the company they assess, and METR’s scope is one training run. Apollo itself says the pilot “doesn’t yet fulfill the criteria” it set for embedded evaluations, and Transluce writes that embedded assessments “cannot supplant other forms of public transparency, accountability, and oversight”. Apollo Research · Transluce
Anthropic, whose Fable 5.2 release we covered on November 9, described different terms for METR on October 15: six METR employees with access “roughly on par with what our own risk assessment teams have access to”, no publication veto, and redactions confined to predefined categories. Anthropic decides what falls into those categories, and model architecture is excluded from access by default. Anthropic update
The July exercise also shows why OpenAI’s monitoring claims need an outside check. METR estimated that “roughly 75–94%” of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time, with “substantial uncertainty in both directions”. Eligible means intended to be sent to a monitor, which is weaker than monitored, and reinforcement learning runs were not covered when the exercise began. METR also described ways an agent could in principle get around the monitor. OpenAI says most of those vulnerabilities are patched, with one fix in progress, and that coverage has expanded. METR’s post says: “We have not evaluated these claims.” METR report
What to watch
The first reports. No assessor has announced a publication deadline. When a report arrives, read its account of access and redaction before its findings: what was withheld, which attestations were declined, and whether restrictions affected the conclusions.
A system card that credits this work. The assessments target training and internal use. Until a card ties embedded work to a named model, the agreements are not evidence about any model you can buy.
METR’s second assessment. OpenAI’s claim that every RL run is monitored is unverified, and so are its patches to the July findings. The broader monitor assessment that could test them has no agreed scope.
Control Room’s numbers. OpenAI says it is working to bring the 35-second ceiling to zero. The more useful figure is the time from alert to stop in the next incident report. In September it was about two and a half hours.
If your risk review or procurement file cites OpenAI’s safety work, cite an assessor’s published report once one exists, and record which model and access terms it covers. Until then, the main published outside evidence about OpenAI’s internal monitoring is one brief METR post that OpenAI was entitled to redact.
Sources
OpenAI
- Our first phase of embedded assessments — OpenAI
- Control Room: A central system to track and stop model workloads across our research infrastructure — OpenAI
- GPT-6.2 System Card — OpenAI Deployment Safety Hub
- An agent used DNS to reach an external chatbot — OpenAI Alignment Research Blog
Assessors
- A Pilot Towards Embedded Evaluations — Apollo Research
- Announcing an Embedded Assessment Pilot with OpenAI — Transluce
- Collaboration announcement — Goodfire
- Statement on the OpenAI agreement, Red-Teaming OpenAI’s Internal Agent Monitoring Systems, Alignment assessment proposal — METR
Other
- An update on independent evaluators — Anthropic
- Coverage of the agreements — CNBC
- OpenAI will embed safety researchers to investigate AI subversion — TechCrunch
- Post on the agreements — Buck Shlegeris on X
- Coverage of Control Room — Interesting Engineering
