llama.rust

LLM inference in Rust

Components

• kv-cache: KVCache struct definition
• all-close: all_close function
• scaled-dot-product-attention: scaled_dot_product_attention function
• round-multiple: round_multiple function
• flash-attention: flash attention definition
• linear-attention: linear attention definition
• multihead-attention: multihead-attention definition
• paged-attention: paged attention definition

References

• llama.cpp
• llm
• vLLM
• mistral.rs
• candle