China Merchants Bank Wins CNCF End User Case Study Contest for Unifying AI Training and Inference on Kubernetes
AI Summary
China Merchants Bank won the CNCF End User Case Study Contest by building a unified Kubernetes control plane using Kueue, KEDA, Prometheus, HAMi, and Fluid. This platform shares nearly 10,000 heterogeneous accelerator cards across AI training, fine-tuning, and online inference. By bringing 99% of its accelerator compute resources under this framework, the bank increased average utilization from 35% to over 60% and reduced inference cost per 1 million tokens by more than 60%. Additionally, its in-house Twinkle training framework enables five LoRA tenants to share a single base model instance, cutting accelerator usage for that setup by 80%.
Key Takeaways
- The unified control plane leverages Kueue, KEDA, Prometheus, HAMi, and Fluid.
- Average accelerator compute utilization improved from 35% to over 60%.
- The Twinkle framework allows five LoRA tenants to share one base model instance, increasing training density fivefold.
Why it matters
This case study demonstrates how large-scale enterprises can effectively optimize expensive heterogeneous hardware resources by combining cloud-native tools. By decoupling runtimes and sharing infrastructure between AI training and inference, engineering teams can drastically lower operational costs and improve hardware utilization.
Conversation 0
Be the first to share your perspective.