Search results

Continue to enable kvcache to be reused during request generation

Community Engagementhelp/insights needed from communityhelp/insights needed from community

Community want to contributePRs initiated from CommunityPRs initiated from Community

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

Scaffolding

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

#3733 In NVIDIA/TensorRT-LLM;

· WeiHaocheng opened on Apr 21, 2025

Support Cohere Command-A (Cohere2ForCausalLM arch)

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

Enhancement

#2912 In NVIDIA/TensorRT-LLM;

· aikitoria opened on Mar 14, 2025

enable_kv_cache_reuse don't work on Qwen L40

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

Bug

#2911 In NVIDIA/TensorRT-LLM;

· yeshihai opened on Mar 14, 2025

Wrong outputs with FP8 kv_cache reuse

bugSomething isn't workingSomething isn't working

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

#2699 In NVIDIA/TensorRT-LLM;

· lishicheng1996 opened on Jan 16, 2025

[Performance] KV cache reuse is slower when batch size > 1

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

#2631 In NVIDIA/TensorRT-LLM;

· ReginaZh opened on Dec 26, 2024

How to make sure enable_kv_cache_reuse working correctly?

bugSomething isn't workingSomething isn't working

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

#2462 In NVIDIA/TensorRT-LLM;

· chwma0 opened on Nov 19, 2024

int8_kv_cache scale

Investigating

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open.

#1557 In NVIDIA/TensorRT-LLM;

· liquanfeng opened on May 7, 2024

FEA: support use_paged_context_fmha feature to enable enable_kv_cache…

Community want to contributePRs initiated from CommunityPRs initiated from Community

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open (in progress).

NVIDIA/TensorRT-LLMnumber 909

#909 In NVIDIA/TensorRT-LLM;

· yunzhongyan0 opened on Jan 18, 2024

Support paged kv cache for benchmarks

Community want to contributePRs initiated from CommunityPRs initiated from Community

KV-Cache Managementkv-cache management for efficient LLM inferencekv-cache management for efficient LLM inference

triagedIssue has been triaged by maintainersIssue has been triaged by maintainers

Status: Open (in progress).

NVIDIA/TensorRT-LLMnumber 130

#130 In NVIDIA/TensorRT-LLM;

· yunfeng-scale opened on Oct 26, 2023