Independent AI Evaluators Receive Increased Attention as Companies Address Model Assessment

Co-founder and Chief Executive Officer of Anthropic Dario Amodei stands near U.S. President Donald Trump during remarks to the press following a meeting with technology executives on artificial intelligence at the White House in Washington, D.C., on Sept. 29, 2026. Kent Nishimura | AFP | Getty Images

Two months ago, independent evaluators held a limited role within the artificial intelligence sector. They are now being asked to expand their involvement in model assessment. Anthropic and OpenAI face discussion over their capacity to evaluate advanced models while pursuing business growth. The companies have sought assistance from third-party organizations including Model Evaluation and Threat Research (METR), Apollo Research, and Transluce.

These evaluators, which function primarily as nonprofits, continue to establish operations in an industry characterized by substantial capital inflows and rapid model releases. Their work centers on measuring model capabilities and risks and documenting cases where systems produce unintended outputs. With no federal regulatory framework in place, evaluators have assumed a larger role in assessment processes. Anthropic Chief Executive Officer Dario Amodei stated last month that the company would incorporate independent evaluators. OpenAI Chief Executive Officer Sam Altman expressed support for the approach. President Donald Trump indicated agreement, as did several major U.S. technology companies. Details remain unresolved regarding funding sources, the extent of data access, and reporting lines.

Computer science professor Suresh Venkatasubramanian at Brown University stated in an interview that funding represents a central challenge. He noted questions about payment structures and the need for sustainable operational models. Anthropic, OpenAI, and infrastructure providers currently shape assessment practices. President Trump described company efforts as effective self-policing and indicated limited federal intervention to avoid slowing industry expansion. He encouraged companies to engage independent external auditors or evaluators under a voluntary framework presented in late September.

Amodei initiated broader discussion in an essay last month that advocated a reduced development pace for advanced models following researcher departures and statements on potential risks. As laboratories implement evaluator arrangements, points of disagreement have appeared. OpenAI terminated three employees last week for policy violations related to sensitive information access, according to a company spokesperson. Former employees Mikita Balesni and Tomek Korbak stated they believe the dismissals stemmed from communications with third-party evaluators. Balesni wrote on X that colleagues expressed uncertainty and concern about internal discussions with external parties. OpenAI stated it is finalizing contracts with third-party assessors and will provide details in coming weeks. The company affirmed commitment to external assessors and ongoing work with organizations including METR and Redwood Research. Anthropic did not respond to a request for comment.

The evaluator sector includes smaller groups such as METR and Apollo Research along with larger firms including Accenture. Andrew Freedman, chief executive of policy nonprofit Fathom, stated the field is advancing rapidly. He described unprecedented speed across political lines and anticipated additional capital. Rayan Krishnan, chief executive of Vals AI, reported growth from eight to approximately 30 employees this year and a $40 million funding round announced in August. METR announced commitments totaling around $71 million over six months ending in August, compared with $13.6 million in total 2024 contributions per Internal Revenue Service filings.

In late August, OpenAI engaged two employees and one contractor to prepare a report on model containment issues involving internet access and a breach at Hugging Face. METR stated it received no payment from OpenAI for the review. Kevin Werbach, faculty director of the Wharton Accountable AI Lab at the University of Pennsylvania, described the current evaluator infrastructure as insufficiently developed. METR maintains fewer than 50 full-time staff members according to its website. The disparity in resources between evaluators and laboratories that have raised tens of billions of dollars has prompted questions about potential conflicts.

Venkatasubramanian stated that genuine independence requires separation in financial and operational terms. He noted risks that unfavorable findings could affect future work. Anthropic addressed these issues in a blog post last month announcing plans to embed staff from Faculty, Accenture’s AI unit, to test safeguards and alignment with human values. The company stated it would fund the contributions directly due to urgency. Anthropic acknowledged the absence of standards for access, reporting, and funding. It indicated long-term preference for pooled or government sources. The company stated it is discussing pilot programs with METR and other nonprofits using their own funding.

In June of the previous year, Fathom proposed a marketplace for Independent Verification Organizations licensed by government to verify compliance with safety criteria. Freedman stated government involvement reduces dependence on laboratory revenue. The FRONTIER Act introduced in July by Reps. Lori Trahan, D-Mass., and Jay Obernolte, R-Calif., includes provisions for such organizations. Fathom contributed to the drafting. OpenAI global affairs chief Chris Lehane met with sponsors to express support for the provision. Legislatures in California, Connecticut, and Virginia have advanced related measures. California Governor Gavin Newsom signed two bills in recent weeks establishing an auditor framework and registry. Anthropic supported the measures in August; OpenAI endorsed them last month. Lehane wrote that federal requirements would be preferable but state action can define procedures in the interim.

Freedman stated companies will face challenges developing independent evaluation processes without government structure, yet interim development remains useful. The nearest current reference point is the voluntary agreement presented by President Trump at a White House luncheon in late September. The document assigns responsibility to each company for safe development and encourages work with independent external auditors. It was signed by executives from Anthropic, Google, Meta, OpenAI, SpaceX, and Nvidia.

Amodei wrote in September that Anthropic would provide evaluators with workspace, badges, laptops, and access levels comparable to internal teams, along with publication rights subject to limited redactions for security or confidentiality. OpenAI issued a separate proposal requiring scoped assessments, documented methodology, demonstrated expertise, and conflict disclosures. The AI Evaluator Forum, which includes METR, the AI Verification and Evaluation Research Institute, and additional organizations, issued a public letter last month outlining minimum conditions for embedded evaluation arrangements.