Quest is an efficient long-context LLM inference framework that leverages query-aware sparsity in KV cache to reduce memory movement during attention and thus boost throughput.

@InProceedings{pmlr-v235-tang24l, title = {{QUEST}: Query-Aware Sparsity for Efficient Long-Context {LLM} Inference}, author = {Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, …

Aug 26, 2024 · NanoFlow: Towards Optimal Large Language Model Serving Throughput Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian …

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference Jiaming Tang*, Yilong Zhao*, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han ICML 2024 / Abstract / Code …

Jun 16, 2024 · Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han

However, we observe the criticality of a token highly depends on the query. To this end, we propose Quest, a query-aware KV cache selection algorithm. Quest keeps track of the minimal …

Weidong Cao, Yilong Zhao (Co-First-Author), Adith Boloor, Yinhe Han, Xuan Zhang, and Li Jiang, Neural-PIM: Efficient Processing-In-Memory with Neural Approximation of Peripherals, in IEEE …

Quest is an efficient long-context LLM inference framework that leverages query-aware sparsity in KV cache to reduce memory movement during attention and thus boost throughput.

[c16] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han: QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. ICML 2024

May 2, 2024 · Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han Published: 02 May 2024, Last Modified: 25 Jun 2024 ICML 2024 Poster Everyone Revisions …