Weighted State Souping for Efficient Long-Context Retrieval
AI Summary
State space models (SSMs) use fixed state sizes for fast inference, and state souping allows independent chunk encoding for cached recombination. However, standard state souping struggles with long, crowded contexts. This paper introduces weighted state souping, which learns query-dependent weights to compose cached states effectively, improving long-context performance across question answering, needle-in-a-haystack tasks, and tool use.
Key Takeaways
- Weighted state souping learns query-dependent weights for composing cached states.
- Can be adapted into a sparse variant with near-constant encoding cost relative to context length.
- Accepted at the PALM Workshop at NeurIPS 2026.
Why it matters
It offers an efficient approach for language models to handle large collections of structured contexts like documents and tools by combining reusable representations with selective retrieval.
Conversation 0
Be the first to share your perspective.