# Xinyuanyi | KV Cache Tiering Storage for LLM Inference

> Xinyuanyi FX all-flash NVMe-oF arrays add a KV cache tier to LLM inference: +29–40% throughput and −26–32% TTFT on a 480B model, each figure backed by a test report.

URL: https://vuke.top/en · Updated November 12, 2026

FX all-flash NVMe-oF arrays catch the KV cache that overflows GPU memory. On a 480B model in production topology that means +29–40% inference throughput and −26–32% time to first token [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3). Every figure maps to a test report you can request and verify.

## Every number has a source

Each item states its baseline and its report ID. Results from different platforms are listed separately and never merged.

| Metric | Value | Baseline | Detail | Source |
| --- | --- | --- | --- | --- |
| Higher inference throughput | +29–40% | vs. a single local NVMe drive | Qwen3-Coder-480B-FP8 in TP=8 production topology under a long-context cold-recovery load: +29% at concurrency 8 and +40% at concurrency 16, the operating point with the largest gain; +35–36% for the whole machine in the dual-instance TP4×2 layout. | Measured · [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3) |
| Lower time to first token (TTFT) | −26–32% | vs. a single local NVMe drive | Across three concurrency levels of the same 480B TP8 load, TTFT p50 falls from 10.17–35.73 s to 7.53–26.35 s. | Measured · [R2](https://vuke.top/en/evidence/R2) |
| Faster than recomputing without external storage | 8.6–20× | vs. full recompute with no external storage | At concurrency 16, TTFT p50 is 149.5 s when recomputing and 11.85 s with FX100; throughput is 4.1 tok/s versus 74.9 tok/s. | Measured · [R2](https://vuke.top/en/evidence/R2) |
| TTFT gain from the LMCache parallel-read patch | 4.1× | vs. before the patch | One GPU, concurrency 16, cold read from disk (Qwen2.5-32B): TTFT 37.97 s → 9.30 s; read bandwidth 0.98 → 5.23 GB/s, about 5.3×. | Measured · [R1](https://vuke.top/en/evidence/R1) |
| Faster model loading | 6.2–9.3× | vs. NFS, on Huawei Ascend 910B | On Huawei Atlas 910B: DeepSeek-32B serving load 691 s → 112 s (6.2×); DeepSeek-70B 1399 s → 150 s (9.3×). | Measured · [R9](https://vuke.top/en/evidence/R9) |
| Faster training checkpoint saves | 1.9× | vs. the baseline in report R1 | 8-GPU 32B LoRA with 65.6 GB full-model snapshots: 178 s → 94 s; sustained write bandwidth 3.26 → 6.40 GB/s (+96%). | Measured · [R1](https://vuke.top/en/evidence/R1) |
| Effective compute utilization under model switching | 46.7% → 62.8% | at 20 model switches per hour | Modeled from measured load times: at 20 model switches per hour, effective compute utilization rises from 46.7% to 62.8%. | Measured · [R1](https://vuke.top/en/evidence/R1) |
| Concurrent model loading on 8 GPUs | 16% / 30% | vs. the baseline in report R1 | Eight GPUs cold-reading models at once: 16% faster wall-clock overall and 30% faster per model. | Measured · [R1](https://vuke.top/en/evidence/R1) |
| Deviation between two independent runs | ≤5% | same load level, measured twice | Two independent runs at the same load level deviate by no more than 5%; cross-instance hot sharing through the fs:// shared pool was verified in the same campaign. | Measured · [R3](https://vuke.top/en/evidence/R3) |
| HBM residency an external KV tier can stand in for | 128 GB | MetaX N260 single GPU, 14B model, seven-way comparison | Conclusion of a seven-way comparison on a 14B model: an external KV tier can stand in for 128 GB-class HBM residency. That campaign ran on a single MetaX N260 GPU; the method carries across platforms, the numbers are not mixed with MI308X results. | Measured · [R5](https://vuke.top/en/evidence/R5) |
| ComfyUI + LTX-Video 2.3 models running end to end | 7 / 7 | AMD MI308X, ROCm 7.2 | All 7 delivered models (3 VAEs, the fp8 text encoder, the text projection and 2 LoRAs) run end to end on MI308X / ROCm 7.2 with no dtype, operator or kernel errors. | Measured · [R6](https://vuke.top/en/evidence/R6) [R7](https://vuke.top/en/evidence/R7) |
| Arrays to inference nodes | 1 : 8 | production sizing basis | One FX100 serves 8 inference nodes; 100 GbE sustaining KV read-back for 8 nodes is a measured anchor. On the joint-test basis, storage spend stays within roughly 10% of the GPU budget. | Estimated · [R1](https://vuke.top/en/evidence/R1) [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3) |

## What GPU memory cannot hold goes to flash

Long-context sessions keep pushing KV cache out of GPU memory. Without an external tier, evicted sessions must be recomputed in full; with FX they are simply read back.

- L1 GPU memory (HBM): Sessions that are active right now (Managed by vLLM)
- L2 Host RAM: Sessions used recently (Managed by LMCache)
- L3 FX all-flash array: All historical sessions, shared across instances (NVMe-oF · RoCEv2)

[How it works](https://vuke.top/en/tech)

## One workload, three concurrency levels

Raw readings from report R2: FX100 against a single local NVMe drive at concurrency 8, 16 and 32.

| Concurrency | FX100 TTFT p50 (s) | Local NVMe TTFT p50 (s) | Full recompute TTFT p50 (s) | FX100 tok/s | Local NVMe tok/s | Full recompute tok/s |
| --- | --- | --- | --- | --- | --- | --- |
| 8 | 7.53 | 10.17 | — | 56.6 | 43.9 | — |
| 16 | 11.85 | 17.31 | 149.48 | 74.9 | 53.6 | 4.1 |
| 32 | 26.35 | 35.73 | — | 71.6 | 53.9 | — |

Qwen3-Coder-480B-FP8 · TP=8 · 8 × MI308X · long-context cold-recovery load. TTFT is p50 in seconds; throughput in tok/s; full recompute was measured at concurrency 16 only. Source: [R2](https://vuke.top/en/evidence/R2)

## The FX series all-flash storage platform

Four models with per-port rates from 100 Gb to 400 Gb. Specifications are vendor figures; FX100 has a full set of measured reports.

| Model | Interface generation | Rate per port | IOPS | Flash form factor | Status | Measured reports |
| --- | --- | --- | --- | --- | --- | --- |
| [FX100](https://vuke.top/en/products/fx100) | PCIe 3.0 | 100 Gb | 16M | U.2 | In production | [R1](https://vuke.top/en/evidence/R1) [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3) [R4](https://vuke.top/en/evidence/R4) [R5](https://vuke.top/en/evidence/R5) [R9](https://vuke.top/en/evidence/R9) |
| [FX200](https://vuke.top/en/products/fx200) | PCIe 4.0 | 200 Gb | 32M | U.2 | In production | — |
| [FX300](https://vuke.top/en/products/fx300) | PCIe 5.0 | 400 Gb | 60M | U.2 | In production | — |
| [FX400](https://vuke.top/en/products/fx400) | PCIe 6.0 | 400 Gb | 140M | E1.S | Test units; volume production planned for late 2026 | — |

## What else we do around this tier

- [Storage Acceleration: KV Cache Tiering](https://vuke.top/en/solutions/storage-acceleration): Storage acceleration
- [Enablement for Domestic and Non-NVIDIA Accelerators](https://vuke.top/en/solutions/domestic-compute): Accelerator enablement
- [AI Datacenter Efficiency Optimization](https://vuke.top/en/solutions/efficiency-optimization): Efficiency optimization
- [AI Datacenter Construction](https://vuke.top/en/solutions/datacenter-construction): Datacenter construction
- [Inference-Stack Software Development](https://vuke.top/en/solutions/software-development): Software development

## Test jointly first, decide afterwards

Every engagement starts with a gated joint test of about 10 weeks. Four gates are passed in sequence; if one is missed, the project stops there, so you never pay for a benefit that was not demonstrated.

- **G1 · Week 2 · Delivery acceptance**: All materials delivered and inspected; every array passes its power-on self-test with all 24 drives recognized.
- **G2 · Week 4 · Single-node baseline**: The single-node 8-GPU inference baseline is reproduced within ±10% of the R1 figures; a local stress test of one array reaches 83% line efficiency against the measured anchor.
- **G3 · Week 7 · Main joint-test gate**: KV tiering must deliver both a TTFT reduction of ≥25% and a throughput gain inside the measured +29–40% band; with 8 nodes sharing one array, read-back bandwidth approaches the NIC ceiling.
- **G4 · Week 10 · Stability and sign-off**: 72 hours of continuous load without interruption; single-drive and single-link fault injection stays invisible to the workload; a signed joint-test report is issued.

## Frequently asked questions

### What does Xinyuanyi do?

Xinyuanyi (Tianjin) Semiconductor Equipment Co., Ltd. builds storage acceleration for LLM inference. The product is the FX series of all-flash NVMe-oF arrays, which holds the KV cache that overflows GPU memory so that evicted sessions are read back instead of recomputed. Around it we offer enablement for domestic and non-NVIDIA accelerators, datacenter efficiency optimization, datacenter construction and inference-stack software development.

### What is the FX series storage acceleration platform?

The FX series is an integrated hardware and software storage acceleration system developed on Guangdong Luisuan GP-series hardware. It ships with a vLLM + LMCache KV cache tiering stack that includes Xinyuanyi's own parallel-read patch. FX100, FX200 and FX300 are in production; FX400 is planned for volume production in late 2026. FX100 offers 100 Gb per port and 16M IOPS; FX300 is a PCIe 5.0 platform with 400 Gb per port and 60M IOPS (vendor specifications).

### How much does KV cache tiering improve inference performance?

With Qwen3-Coder-480B-FP8 on 8 × AMD MI308X in TP=8 production topology, the measured results under a long-context cold-recovery load are: 29–40% more inference throughput and 26–32% lower TTFT p50 than a single local NVMe drive [R2](https://vuke.top/en/evidence/R2) [R3](https://vuke.top/en/evidence/R3), and 8.6–20× faster than full recompute without external storage [R2](https://vuke.top/en/evidence/R2). These are application-level throughput and time to first token, not microsecond latencies of the storage device.

### Where does it help most, and where is the benefit limited?

The gain comes from cache hits. Long contexts, multi-turn conversations and high-concurrency session recovery (agents, coding assistants, customer service) hit the cache often and benefit the most. Short, low-concurrency conversations hit it rarely and benefit little. That is why we recommend testing before buying: measure with your own session-length distribution and concurrency, then decide.

### How can these numbers be verified? Are they reproducible?

Every key figure carries a report ID (R1–R9), and the reports are available on request. R4 is an official report issued by a third-party testing organization. R8 is the test code export and raw data: the LMCache parallel-read patch, the load client, orchestration scripts and forensic probe scripts, so a third party can rerun the tests on its own cluster. Two independent runs at the same load level deviate by no more than 5% [R3](https://vuke.top/en/evidence/R3).

### Does adopting FX require changes to the inference engine?

No rewrite of the inference engine is needed. The integration point is the open-source vLLM + LMCache combination: LMCache moves KV cache that does not fit in GPU memory down to host RAM and external storage, and the FX array is that external tier. We adapted the stack at source level on ROCm, and the parallel-read patch is delivered as part of R8 [R1](https://vuke.top/en/evidence/R1) [R8](https://vuke.top/en/evidence/R8).

## Contact

- Company: Xinyuanyi (Tianjin) Semiconductor Equipment Co., Ltd.
- Contact: Wang Limin
- Phone: +86 131 6181 8898
- Email: mingxin@agentmail.to (The inbox is monitored and triaged by AI around the clock)