Nvidia staffs new AI safety team amid its push for open models

Nvidia Forms AI Safety and Security Engineering Team Alongside Open-Weight Model Initiatives

Nvidia is assembling an AI safety and security engineering team, according to multiple job postings published in late July. The company is recruiting a distinguished engineer to act as founding technical leader for the group, along with a security research engineer, an evaluation engineer, and a senior manager.

The new team will assess AI agents prior to deployment and develop AI-based tools for addressing software vulnerabilities, according to the listings. One posting states that the team operates from the position that open-weight models, transparency, and broad scientific review support American AI leadership and cybersecurity defense.

Nvidia did not respond to a request for comment.

The hiring occurs as Nvidia increases its public statements supporting open-weight AI models. These models release trained weights publicly while keeping training data and source code private. This approach differs from the closed systems offered by OpenAI and Anthropic.

In July, Nvidia chief executive Jensen Huang posted on X a letter to U.S. policymakers that described open models as strengthening safety and cybersecurity. Shortly afterward, Nvidia joined the Open Secure AI Alliance as a founding member. The alliance focuses on developing open-source security tools for AI and includes more than 120 organizations such as Microsoft, Palantir, SpaceX, and Hugging Face.

The job postings for Nvidia’s internal team appeared before the alliance announcement and outlined responsibilities that align with the alliance’s stated goals. Hugging Face has used open models in responding to security incidents.

Open-weight models allow wider access to AI technology. Some observers have noted that this accessibility could extend to individuals seeking to misuse the technology, while others have stated that public weights enable broader testing and faster identification of issues through community review.

Nvidia’s actions also align with commercial considerations. Wider distribution of open models increases the number of users running AI workloads, which in turn raises demand for the company’s graphics processors and related hardware. Safety measures become relevant as organizations consider deploying AI systems that interact with internal data or perform actions in operational environments.

The company has continued to release statements and participate in industry groups focused on these topics. The job listings indicate that the safety team will contribute to evaluation processes and tool development in support of these efforts.

Geoff Weiss is a senior reporter on Business Insider’s tech team, where he writes about AI startups and Y Combinator, the intersection of AI and the media industry, and workplace dynamics within top AI labs and chip companies. Previously, Geoff was on the media desk, covering YouTube and Netflix, and themes like the intersection of Hollywood and the creator economy. His work on Netflix’s video podcasting ambitions and Mr Beast’s lessons for Hollywood won second and first prize, respectively, at the 2025 LA Press Club Awards. Prior to joining Business Insider, Geoff was the senior editor of Tubefilter and a staff writer at Entrepreneur. He graduated from New York University with a degree in English Literature. He can be reached at gweiss@businessinsider.com, on Signal @geoffweiss.25, and on LinkedIn.