Operations | Monitoring | ITSM | DevOps | Cloud

GPU cloud security tends to get discussed as if it's the same problem as general cloud security. It isn't. GPUs sit between processes in ways CPUs don't. Training data passes through them in patterns that create specific exposure. Model weights derived from sensitive data are themselves sensitive material in ways most procurement processes don't recognize. And the multi-tenant nature of public GPU cloud creates failure modes that don't exist in CPU-only environments.

Serving large language models at scale is fundamentally a distributed systems problem. A single GPU, or even a single node, is rarely enough once you need multiple models, multiple replicas, tensor-parallel sharding across GPUs, or high-availability rollouts. Kubernetes solves general container orchestration well, but it has no native concept of a GPU-aware, actor-based compute cluster.

The story of 2026 engineering has a plot twist that on-call teams saw coming. AI coding assistants now write a large share of production code, pull requests per developer are up, and delivery velocity looks great on the dashboard. Then the pager goes off.

AI-related outages just moved from a fringe worry to a mainline reliability problem, and the on-call rotation is where that shift lands first. A new StackGen analysis of nearly 178,000 public status-page records found that incidents disclosed by AI model and AI application companies now account for more than one in ten reported outages, a sixfold jump from 1.7 percent in 2023 to 10.7 percent so far in 2026.

CloudZero had a full team on the ground at Ai4 in Las Vegas during the first week of August 2026. The team included CTO Erik Peterson, who spoke on a panel about AI cost economics. The same problem surfaced everywhere we went: teams can see what they’re spending, but not whether it’s working. DIY cost tooling that fails time and time again, agent sprawl, and a widening gap between finance and engineering kept coming up throughout the week.

AI tokens are the small chunks of text, roughly four characters or three quarters of a word each, that language models read and generate. Every prompt and every response is measured in tokens, and AI providers bill per million of them. That makes the token the base unit of AI spend: 1,000 tokens is about 750 words, and every AI feature you ship is a token meter running.

In 2025, 85% of organizations increased AI investment, and 91% plan to do the same this year, according to Deloitte. Despite continued spending, however, ROI lags behind, with just 6% seeing payback within one year. While AI use cases tend to have a longer payback period, often in the 2-4 year range, companies can’t afford to keep spending money without some measure of its practical impact both immediately and over time.

AI coding assistants have changed how fast application code gets written. Copilot, Cursor, Claude Code, and agentic tools built on top of them can generate a working feature in minutes. But none of them know your database's history. They don't know that a migration already renamed that column last sprint, that a policy forbids unqualified DELETE statements, or that the target environment has drifted from what your migrations say it should look like.