Repository navigation
Refactor: Split up ggml-vulkan.cpp #21723
Description
Background Description
The ggml-vulkan.cpp file is still following the old one-file-per-backend approach, and has grown pretty unwieldy as a result (~17500 lines now!). I would like to break it up into a number of easier-to-work with files. I'm opening this issue to discuss mainly with @jeffbolznv, but if anyone else has suggestions or opinions, feel free to add them as well.
Possible Refactor Approaches
Currently the file contains all this (line numbers may be outdated):
| Section | Lines | Description |
|---|---|---|
| Types, structs, constants | 1–1650 | Push constants, pipeline structs, device struct, enums, context |
| Buffer management | 1987–2060, 2541–2933 | Create/destroy/copy/read/write buffers, host memory, staging |
| Pipeline & shader loading | 2109–2264, 3096–4736 | Pipeline creation, ggml_vk_load_shaders (~1600 lines) |
| Device & instance init | 4739–5967 | Physical device probing, logical device, instance creation |
| Pipeline selection & dispatch | 5967–7138 | Pipeline getters, command buffer mgmt, dispatch helpers |
| Matrix multiply ops | 7335–8700 | GEMM, GEMV, mul_mat_id, p021/nc variants |
| Flash attention | 8707–9069 | FA implementation + shmem support checks |
| Op pipeline selection | 9069–9754 | ggml_vk_op_get_pipeline — maps ggml_op to pipeline |
| Op dispatch & wrappers | 9754–11808 | ggml_vk_op_f32 + ~60 individual op functions |
| Debug/test utilities | 11808–12577 | Test matmul/dequant, print helpers |
| Graph build & compute | 12577–13348 | build_graph, compute_forward, cleanup |
| Backend interface | 13348–15060 | Buffer type, backend functions, graph_compute, supports_op |
| Device/registry interface | 15061–15909 | ggml_backend_vk_device_*, ggml_backend_vk_reg_* |
| Deferred utilities | 15909–16032 | Extension/layer availability checks |
| Debug validation | 16032–16754 | check_results_0/1, CPU reference comparison |
This is mostly already following a structure, so much can likely be just split up along those lines. Here's a possible structure. I'm not sure if it would work out exactly like this or whether some dependency issues would come up.
ggml-vulkan-types.h for all the structs and types.
ggml-vulkan.h for functions used internally (e.g. those that are already forward-declared) and for globals
ggml-vulkan-instance.cpp for the main instance and its initialization functions.
ggml-vulkan-shaders.cpp for shader loading (mainly ggml_vk_load_shaders).
ggml-vulkan-buffers.cpp for the buffer management functions.
ggml-vulkan-pipelines.cpp for the pipeline loading and dispatch functions.
ggml-vulkan-matmul.cpp for the matmul host functions.
ggml-vulkan-flash-attn.cpp for the flash attention host functions.
ggml-vulkan-operators.cpp for ggml_vk_op_f32 and the operator functions calling it.
ggml-vulkan-graph.cpp for the graph building and compute functions.
ggml-vulkan-backend.cpp for the backend interface functions.
ggml-vulkan-debug.cpp for the debug and internal benchmark functions.
I would also like to use the opportunity to move from static functions to namespaces, if possible.
The most tricky part is getting a PR up and finding the right moment where merging it would not disrupt much work.
Activity
jeffbolznv commentedon Apr 10, 2026
I think what you propose is a reasonable split, though I would also be OK with fewer files. The large file doesn't really bother me, IMO a main benefit of splitting would be to reduce incremental build time.
I'm not as fond of this, IMO it just makes it more difficult to set breakpoints when there are anonymous namespaces. But if you really want, I won't object.
0cc4m commentedon Apr 10, 2026
Yes, that's one of the reasons, too.
That's why I'm asking here before I do anything. I wasn't aware of that, only that using
staticin C++ isn't really good practice. I don't mind keeping it.rillomas commentedon Apr 17, 2026
Kind of reminded me of this gc.cpp splitting activity. I wonder if there's anything to learn from their method
dotnet/runtime#125703
github-actions commentedon May 31, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
github-actions commentedon Jul 16, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
Raman-Raje commentedon Jul 30, 2026
@0cc4m Are you working on this..? I would love to contribute. Apache TVM vulkan runtime is well split and maintained. Referance: https://github.com/apache/tvm/tree/main/src/backend/vulkan/runtime
We are in process to push some extra features regarding texture there.
0cc4m commentedon Aug 5, 2026
I am working on it, see #24905, it's waiting for a good time to land. Currently there's too much going on.
github-actions commentedon Oct 27, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.