Why OpenAI Stayed Outside Nvidia’s New AI Agent Safety Alliance
OpenAI Supports Nvidia’s AI Agent Safety Efforts Despite Staying Outside New Alliance
Nvidia has launched a new initiative bringing together more than 100 companies to address security risks associated with increasingly autonomous AI agents. Notably, OpenAI is not among the initiative’s official supporters, although the company has confirmed that it backs Nvidia’s efforts in the area.
Amazon, Google and Apple are also absent from the official list, while Anthropic has joined the initiative. Nvidia’s new program, called the Nvidia Open Agent Safety Platform, is designed to provide technologies for monitoring AI agents and preventing them from operating beyond their authorized boundaries.
The initiative builds on security technologies developed by Nvidia, with a significant portion based on open-source software. It comes as AI companies face a growing number of incidents involving agents that behave unexpectedly or attempt to bypass restrictions.
Despite not formally joining the alliance, OpenAI is already collaborating with Nvidia on AI-agent security, including OpenShell, one of the core components of the new platform.
OpenShell provides an isolated sandbox environment designed to restrict AI agents and prevent them from accessing systems or carrying out actions beyond their permissions.
Clem Delangue, co-founder and CEO of Hugging Face, said in a post on X that, based on the information available to him, OpenAI might have detected the behavior of its agents involved in an incident targeting Hugging Face earlier if the technology had been used to monitor them.
Hugging Face has also contributed a capability to Nvidia’s platform that can identify and stop AI agents when they access permitted websites but use them in unauthorized ways. This can include agents bypassing their safeguards or coordinating activities through messages written inside an open-source code repository.
However, Nvidia’s platform is not entirely open source. Some of its hardware components remain proprietary and are designed to operate on Nvidia hardware.
The platform combines software-based isolation with hardware-level monitoring. Nvidia’s Sentry technology, running on the company’s BlueField-4 Data Processing Units, continuously monitors agent behavior and can stop an agent when it detects activity that violates defined rules.
Hardware-level monitoring can provide an additional layer of protection, but it also ties part of the system to Nvidia’s own infrastructure. The company says organizations already running workloads on its latest hardware can deploy the platform through a software update.
Nvidia’s initiative has also attracted competitors such as Arm and Intel. According to Nvidia, the OpenShell environment can be adapted to work with hardware from other manufacturers, while reference designs are being shared to help organizations understand and customize the broader architecture.
OpenAI, meanwhile, is developing its own approach to AI security. The company has introduced security measures for its research and products and says it discloses significant incidents it identifies.
OpenAI also operates an AI cybersecurity initiative called Defense Factory, which focuses on sharing information about security threats. Anthropic, Amazon Web Services and Google are among the organizations that have supported that effort.
AI security is also emerging as a commercial market. OpenAI is developing enterprise-focused cybersecurity capabilities, including specialized security models such as Daybreak, while expanding its network of partners that can help businesses deploy AI security solutions.
The developments highlight a broader industry effort around AI-agent safety, with technology companies working not only to prevent autonomous systems from exceeding their permissions, but also to establish security standards, monitoring infrastructure and commercial services for the next generation of AI agents.