NVIDIA Uses a Team of AI Agents to Run Its DGX SuperPOD Clusters

One chat interface instead of many dashboards

As GPU clusters grow, operations span compute, network, hardware and scheduling. NVIDIA says SREs often have to examine several dashboards to identify why training performance dropped. The AI Factory Operations Agent replaces that routine with specialized agents for different operational domains, reachable through a single chat interface.

According to NVIDIA, in production these agents detect failures, correlate root causes across layers and surface clear recommendations. SREs can use natural language with any agent, switch between investigations and track every action for auditability. The sub-agents communicate through Model Context Protocol (MCP), which NVIDIA says any organization can adopt.

Architecture built on NemoClaw, OpenShell and Nemotron

The developer example is built around the NVIDIA NemoClaw blueprint. A user-facing agent runtime receives SRE queries and invokes the appropriate domain agents and tools. Agents connect to infrastructure through OpenClaw plugins and MCP servers, and each MCP server exposes a focused set of approved operational tools. Users can work through a built-in web UI or connect existing coding-agent interfaces through the API.

The framework uses NVIDIA OpenShell and can run fully on-premises, which NVIDIA says supports extensibility and data sovereignty. Inference uses locally served NVIDIA Nemotron open models, and the framework runs across internal NVIDIA DGX H100, DGX B200 and DGX GB300 production clusters.

The system runs as a Kubernetes control-plane application, with Helm deploying the UI, OpenClaw gateway, MCP tool servers, inference endpoint and OpenShell runtime as Kubernetes services. For shell execution, OpenShell creates a per-session Kubernetes sandbox rather than running commands in the main gateway pod. Kubernetes inspection goes through a server-side run_kubectl tool using registered read-only credentials, while OpenShell itself receives neither kubectl nor Kubernetes credentials. In NVIDIA's example, the agent can get pods and configmaps but cannot get secrets or create and update pods.

The domain agents

NVIDIA lists six agents in its internal deployment:

• Job Recovery Agent: detects training job failures and goodput degradation, runs multi-layer root cause analysis across scheduler, application, hardware and network layers, and recommends recovery steps once the root cause is confirmed.
• Hardware Agent: triages device-level faults such as PCIe errors, ECC errors, GPU issues, NVMe failures and thermal anomalies.
• Observability Agent: queries historical Prometheus, telemetry and Grafana Loki logs and builds dashboards dynamically in the web UI.
• Research Agent: searches preloaded documentation, including NVIDIA Mission Control manuals and DGX SuperPOD reference architectures, and accepts uploaded runbooks.
• Kubernetes Agent: provides read-only awareness of pods, services, deployments, GPU resources, node status and scheduling state.
• Cluster Management Agent: uses NVIDIA Base Command Manager APIs through diagnostic MCP tools for advisory management and troubleshooting.

Cross-cutting MCP tools include web search, code execution and Grafana dashboards. Organizations can also connect their own MCP servers and APIs to fit existing schedulers, observability platforms and cluster managers.

Lessons from production and deployment advice

NVIDIA says it continues to refine the framework by measuring root cause analysis accuracy, tracking SRE agreement and expanding the agent ecosystem. It names five lessons: keep agent boundaries clear, because overlapping ownership creates noise; start read-only, since accurate investigation builds SRE trust before automation; correlate across layers to find root causes that isolated tools miss; define MCP tools precisely to reduce wrong tool selection; and show tool calls, evidence and conclusions before SREs act.

For large deployments, NVIDIA prefers a dedicated GPU-accelerated control plane isolated from the production cluster, so agent reasoning does not consume training GPUs or disrupt workloads. Smaller deployments can use a local or cloud inference endpoint depending on data sovereignty, cluster size and available GPU capacity, including an RTX PRO Workstation or DGX Spark.

In multi-tenant AI factories, scoped agents, tools, sandboxes and Kubernetes permissions limit what each tenant can reach. Tenant SREs see only their own namespaces, jobs and nodes, while central operations keep broader cluster visibility. NVIDIA points readers to the AI Factory Operations Agent developer example and an open source framework on GitHub.