Vulkan Device memory allocation failed (ErrorOutOfDeviceMemory )

[Eliastrt]

Please include information about your system, the steps to reproduce the bug, and the version of llama.cpp that you are using. If possible, please provide a minimal code example that reproduces the bug.

---

System: OpenSuse Leap 15.4
GPU: AMD RX580 8GB
Vulkan Instance Version: 1.3.275
VkPhysicalDeviceMemoryProperties:

memoryHeaps: count = 3
memoryHeaps[0]:
size = 8321499136 (0x1f0000000) (7.75 GiB)
budget = 8310034432 (0x1ef511000) (7.74 GiB)
usage = 0 (0x00000000) (0.00 B)
flags: count = 1
MEMORY_HEAP_DEVICE_LOCAL_BIT
memoryHeaps[1]:
size = 4133625856 (0xf6622000) (3.85 GiB)
budget = 4124254208 (0xf5d32000) (3.84 GiB)
usage = 0 (0x00000000) (0.00 B)
flags:
None
memoryHeaps[2]:
size = 268435456 (0x10000000) (256.00 MiB)
budget = 256970752 (0x0f511000) (245.07 MiB)
usage = 0 (0x00000000) (0.00 B)
flags: count = 1
MEMORY_HEAP_DEVICE_LOCAL_BIT

Vulkan0: AMD RADV POLARIS10 | uma: 0 | fp16: 0 | warp size: 64

GGUF model: mistral-7b-instruct-v0.2.Q6_K.gguf

---

First of all, thanks to Occam for new Vulkan implementation of Llama.CPP!

I have tried to run llama.cpp according instruction "without Docker":

--> ./bin/main -m "PATH_TO_mistral-7b-instruct-v0.2.Q6_K.gguf" -p "Hi you how are you" -n 50 -e -ngl 33 -t 4

Got an error:
ggml_vulkan: Device memory allocation of size 4257734656 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
llama_model_load: error loading model: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/root/GPT/GGUF/mistral-7b-instruct-v0.2.Q6_K.gguf'
main: error: unable to load model

FYI:
If I reduce -ngl to 23 layers then everything worked properly, but slowly.

llm_load_tensors: CPU buffer size = 5666.09 MiB
llm_load_tensors: Vulkan0 buffer size = 3925.09 MiB
6.51 tokens per second (OpenCL version loads full model on the same GPU and shows ~12 tokens\sec)

[0cc4m]

OpenCL does not run the full model on the GPU, it just does matrix multiplications, scalar multiplications and scalar additions, all the rest is done by the CPU. That means it uses a little less VRAM, yes. I guess you're using the GPU for a GUI at the same time, so it has other programs occupying its VRAM and doesn't find enough for Vulkan. Try using q5_k or q4_k.

[Eliastrt]

Thanks for reply.

I have no GUI on this machine. Anyway, 23 layers uses ~4GB and run without problems, 24 layers rise this error.

According AMD docs: https://gpuopen.com/learn/vulkan-device-memory/

For Vulkan implementations could be used different Heap levels. And it looks like this implementation uses Heaps[1] instead of Heaps[0]. If so, it is the problem, because my vulkaninfo shows 8GB for Heaps[0] and only 4GB for Heaps[1]

[0cc4m]

Thanks for reply.

I have no GUI on this machine. Anyway, 23 layers uses ~4GB and run without problems, 24 layers rise this error.

According AMD docs: https://gpuopen.com/learn/vulkan-device-memory/

For Vulkan implementations could be used different Heap levels. And it looks like this implementation uses Heaps[1] instead of Heaps[0]. If so, it is the problem, because my vulkaninfo shows 8GB for Heaps[0] and only 4GB for Heaps[1]

You didn't post the whole vulkaninfo, but I'm pretty sure it's using the right heap. It looks for device-local memory first. The 4GB heap is host-visible memory (RAM). This is used, too, but only for layers you didn't put on the GPU and for staging buffers.

[Eliastrt]

full vulkaninfo output in attachment
vulkaninfo.txt

FYI, I have tried run with another RX580 8GB just to avoid hardware issues and got the same error.

[0cc4m]

full vulkaninfo output in attachment vulkaninfo.txt

FYI, I have tried run with another RX580 8GB just to avoid hardware issues and got the same error.

Looks perfectly fine. It's using the right memory. Try compiling with LLAMA_VULKAN_DEBUG=1 and LLAMA_VULKAN_VALIDATE=1 (this needs Vulkan validation layers installed). Then run it again in the way that causes it to error and upload the whole output. It's gonna be very verbose.

[Eliastrt]

Thanks! So, I can't understand this output, but here it is:
test_vulkan_rx580_8GB_part_1.txt
test_vulkan_rx580_8GB_part_2.txt

[0cc4m]

Thanks! So, I can't understand this output, but here it is: test_vulkan_rx580_8GB_part_1.txt test_vulkan_rx580_8GB_part_2.txt

So it's just trying to allocate a buffer of size 4257734656 bytes. Your driver reports a max buffer size of 4294967292 bytes, so this should work. Maybe the limit is lower, or it has an issue with memory fragmentation? Try this patch to halve the limit:

diff --git a/ggml-vulkan.cpp b/ggml-vulkan.cpp
index 7834e635..1d1cdc82 100644
--- a/ggml-vulkan.cpp
+++ b/ggml-vulkan.cpp
@@ -1186,6 +1186,8 @@ void ggml_vk_init(ggml_backend_vk_context * ctx, size_t idx) {
         ctx->device.lock()->max_memory_allocation_size = props3.maxMemoryAllocationSize;
     }

+    ctx->device.lock()->max_memory_allocation_size = 2147483646;
+
     ctx->device.lock()->vendor_id = ctx->device.lock()->properties.vendorID;
     ctx->device.lock()->subgroup_size = subgroup_props.subgroupSize;
     ctx->device.lock()->uma = ctx->device.lock()->properties.deviceType == vk::PhysicalDeviceType::eIntegratedGpu;

[Eliastrt]

Great! It's working now! ~11 tokens\sec

By the way, what is your plans for the feature of model split on several GPU? ;)

[0cc4m]

Great! But that's not a real solution yet. Let's keep the issue open until there's a way to lower the limit without code changes. Maybe an environment variable?

[Eliastrt]

I have found some interesting info in Vulkan Specs:
https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html#vkAllocateMemory

"Some platforms may have a limit on the maximum size of a single allocation. For example, certain systems may fail to create allocations with a size greater than or equal to 4GB. Such a limit is implementation-dependent, and if such a failure occurs then the error VK_ERROR_OUT_OF_DEVICE_MEMORY must be returned. This limit is advertised in VkPhysicalDeviceMaintenance3Properties::maxMemoryAllocationSize."

[0cc4m]

I have found some interesting info in Vulkan Specs: https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html#vkAllocateMemory

"Some platforms may have a limit on the maximum size of a single allocation. For example, certain systems may fail to create allocations with a size greater than or equal to 4GB. Such a limit is implementation-dependent, and if such a failure occurs then the error VK_ERROR_OUT_OF_DEVICE_MEMORY must be returned. This limit is advertised in VkPhysicalDeviceMaintenance3Properties::maxMemoryAllocationSize."

Yeah, if you look at the patch I gave you that is what I'm doing in the line above the new one. Your GPU reports a limit of 4GB, but fails when I actually try to allocate (slightly less than) 4GB. 2GB seems fine.

[Eliastrt]

I have made some R&D and can give you additional info:

1. For my device Heap[1] < maxMemoryAllocationSize

Vulkan0: AMD RADV POLARIS10 | uma: 0 | fp16: 0 | warp size: 64
DEBUG: maxMemoryAllocationSize=4294967292
DEBUG: maxBufferSize=0
DEBUG: MemoryHeap[1] Size=4133625856

2. If I assign size of Heap[1] everything working fine:

vk::PhysicalDeviceMemoryProperties mem_props = ctx->device.lock()->physical_device.getMemoryProperties();
....
if (maintenance4_support) {
ctx->device.lock()->max_memory_allocation_size = std::min(
std::min(props3.maxMemoryAllocationSize, props4.maxBufferSize), mem_props.memoryHeaps[1].size
);
} else {
ctx->device.lock()->max_memory_allocation_size = std::min(
props3.maxMemoryAllocationSize, mem_props.memoryHeaps[1].size
);
}

What are your params for Heap[1] and maxAllocationSize?