Supermicro, IBM and NVIDIA Launch KV Cache Storage Solution for AI Inference

Supermicro, IBM and NVIDIA have introduced a shared KV cache storage solution designed to improve the performance of large-scale AI inference workloads. The platform combines high-capacity flash storage, distributed storage software and NVIDIA's inference and networking technologies to help AI systems reuse previously processed information.

Announced on November 5, 2026, the solution brings together Supermicro's Petascale all-flash storage servers, IBM Storage Scale, NVIDIA Dynamo inference software and NVIDIA Spectrum-X Ethernet networking. The Supermicro servers use AMD EPYC 9005 Series processors.

The companies say the architecture can reduce the time AI models take to begin generating responses, particularly when applications handle long prompts, repeated conversations and complex agentic workflows.

How the KV Cache Storage Solution Improves AI Inference

Large language models process incoming information in stages. During the prefill phase, a model converts input text into tokens and calculates intermediate key-value states. These states help the model generate subsequent responses during the decode phase.

When an application repeatedly uses the same information, recalculating those states can consume time and computing resources. A persistent KV cache stores previously generated states so they can be reused across requests.

However, the amount of cached information can grow rapidly in applications handling lengthy conversations, large codebases, documents and retrieval-augmented generation workloads. Keeping all that information in GPU memory can become expensive and restrictive.

The new platform moves KV cache storage into a shared networked storage layer. This allows multiple GPU nodes to access cached information without relying entirely on their local memory. The approach is intended to reduce redundant computation, improve GPU utilisation and support higher inference throughput.

Companies Report Up to 56 Times Faster Response Initiation

The most notable result comes from joint performance testing involving a 120-billion-parameter language model. Supermicro and NVIDIA reported up to 56 times faster time-to-first-token with persistent KV caching at 130,000-token prompts, compared with a system operating without KV caching.

Time-to-first-token measures how long a user waits before receiving the first generated token. It is an important measure of responsiveness, particularly for AI assistants and applications that process lengthy inputs.

The companies also reported that as input length increased from 30,000 to 130,000 tokens, time-to-first-token rose by only 0.377 seconds with KV caching. Without the cache, the increase was 29.55 seconds.

These figures come from the companies' joint testing and should be understood within that specific workload and configuration. They do not establish a 56-fold improvement across every AI model or inference environment.

Shared Storage Targets Larger Enterprise AI Workloads

The platform is aimed at enterprises and cloud infrastructure providers running high-throughput inference services. Its potential applications include coding assistants, multi-turn AI conversations, long-context retrieval systems and agentic applications that repeatedly access large amounts of information.

Beyond response speed, the companies highlight higher GPU utilisation, improved throughput and the ability to expand cache capacity independently of GPU memory. The architecture also provides shared access to cached information across GPU nodes and clusters.

Supermicro says its factory rack integration and deployment services can reduce implementation time from weeks to days. The companies also report that they validated the integrated system under high-concurrency conditions and network traffic scenarios involving competing workloads.

The collaboration reflects a broader infrastructure challenge as AI adoption moves beyond model training towards sustained inference operations. As applications handle longer prompts and more complex tasks, efficient memory and storage management will become increasingly important.

By combining persistent caching with high-performance networking and storage, Supermicro, IBM and NVIDIA are targeting one of the bottlenecks that can limit large-scale AI inference.