Google open-sources TPU Sync to move KV cache between TPUs

A KV cache (key-value cache) is the working memory a language model builds while it reads a prompt: for every token already processed, it keeps intermediate values so it does not have to recompute them each time it produces a new word. The longer the conversation or document, the larger this cache grows, and it normally sits in the scarce and expensive high-bandwidth memory (HBM) of the accelerator chip.

Google has now released TPU Sync, an open-source native C++ library built to move and manage that cache across its tensor processing units (TPUs). According to TechBytes, which summarises a Google Cloud blog post by engineers Fenghui Zhang and Weiren Yu, the library also synchronises model parameters in reinforcement learning workflows. SemiAnalysis says TPU Sync was formerly called TPU-raiden, and Open Source For You describes the project as TPU Raiden, released under the Apache 2.0 licence.

Why does the KV cache have to move at all?

Language model inference has two phases, as Open Source For You explains: prefill, where the model reads the whole prompt, and decode, where it generates the answer one token at a time. Disaggregated serving runs these two phases on separate pools of hardware, so after prefill the resulting KV cache must be shipped to the chips that handle decode. Without a transfer library, SemiAnalysis notes, external TPU serving is limited to the aggregated mode, even though Google has run disaggregation internally for Gemini for years.

A second need is offloading. When the HBM of a chip cannot hold the cache of every user, SemiAnalysis explains, the data must travel from HBM to the host’s DRAM and back. Google’s Mark Lohmeyer, vice president and general manager of AI and computing infrastructure, told journalists on a call, as reported by The Next Platform on 9 November 2026, that TPUs in agentic workloads are often idle for a large share of the time. TPU Sync, he said, moves the working memory off the chip when an agent pauses and restores it when the agent resumes, which frees the accelerator for other tasks and lowers the cost per agentic workload. The Next Platform adds that, because TPU Sync is software, Google can back-port it to earlier generations of TPU systems.

What performance does Google claim?

The figures below come from TechBytes, dated 9 November 2026, which attributes them to the Google Cloud blog. We could not read the Google post itself, so they should be treated as vendor-reported numbers relayed by a secondary outlet.

• Key-value transfer latency is cut by more than 90% on open-weight Mixture of Experts models of 235 billion to 480 billion parameters, according to TechBytes.
• Time to First Token (TTFT), the delay before the first word of an answer appears, falls by 90% on average across prefix-caching and disaggregated-serving benchmarks, compared with standard setups, according to TechBytes.
• The raw_transfer component reaches up to 223 GB/s of aggregate device-to-host and host-to-device throughput on Ironwood TPU v7x hosts, against a physical PCIe Gen5 limit of 256 GB/s, according to TechBytes.
• Across the data center network, the software sustains about 192 Gbit/s on one 200 Gbit/s network card and 350 Gbit/s across two cards with NUMA-aware striping, according to TechBytes.

How does TPU Sync work under the hood?

TechBytes says TPU Sync relies on layout-preserving direct memory access (DMA, a hardware mechanism that copies memory without involving the processor) combined with in-place buffer aliasing. It avoids Python serialisation and intermediate copies by working directly with JAX PJRT buffers and PyTorch tensors, and the transfers happen outside the XLA compilation graph. SemiAnalysis describes the same idea as zero-copy transfers obtained by extracting the native hardware descriptors of PJRT buffers.

According to TechBytes, the runtime is organised in three modules. A global key-value store acts as a registry that maps prompt hashes to physical storage blocks and directs least-recently-used evictions. A cluster controller handles execution planning and lease management. A local cache manager tracks the state of device blocks, manages pinned host memory and keeps a warm-boot copy in shared memory under /dev/shm. The memory hierarchy spans HBM, pinned host DRAM, local SSDs and Google Cloud Managed Lustre.

On integrations, TechBytes lists official bindings for PyTorch through TorchTPU and for JAX, a connector for the vLLM, SGLang and llm-d serving stacks, and low-level primitives for Pathways and multi-controller JAX pipelines.

What are the limits, and how mature is it?

TechBytes states that transfers between HBM and host DRAM are production-ready, while tiering into shared storage such as Managed Lustre and external POSIX backends is still being validated experimentally. The reinforcement learning weight-synchronisation module is also experimental, and reaching the top network throughput requires two 200 Gbit/s network cards configured with NUMA-aware striping. Open Source For You, whose article is undated, goes further and calls the project still under active development and not yet intended for general production use, so the maturity picture differs between sources.

SemiAnalysis adds a competitive angle. Its InferenceX preview compares aggregated TPU v7 against aggregated Nvidia B200 and B300, and it says Nvidia’s GB200 and GB300 NVL72 systems currently lead on part of the curve in disaggregated serving. A disaggregated TPU v7 comparison is planned once TPU-Sync-based disaggregation is optimised in the external stack, and SemiAnalysis expects Mooncake Store support on TPU to be built on TPU-Sync primitives.

What does it change for a company?

For an international company that runs or rents long-context assistants and agents on Google Cloud, KV cache movement becomes something to configure rather than something to build. The practical steps are to test the connector with vLLM or SGLang on your own prompts, measure time to first token before and after, and check which memory tiers are still marked experimental before relying on them. Teams that train with reinforcement learning should validate the weight-synchronisation module on their own trainer and sampler topologies. None of the published figures replaces a benchmark on your workload.