[akapoor3518]

Summary

Ollama statically links llama-context.cpp into its Go binary via cgo, while the ggml core (ggml.c) and the Tsavorite backend are built into separately dlopen'd shared libraries (libggml-base.so, libggml-tsavorite.so). Several symbols the "GGML Perf Summary" table depends on are defined in ggml.c and were link-time unreachable from llama-context.cpp in that configuration:

• ggml_perf_accumulate — undefined reference at link time (table never printed at all).
• ggml_perf_log_open, ggml_perf_write_detailed_csv, ggml_backend_type — same issue, only surfaced once the detailed (Target/TSI_KERNEL-RUN column) table variant was exercised.

Also, ggml-tsavorite.cpp's existing #ifdef OLLAMA backend struct literal already references .profile, .graph_reserve, .buffer_size, .reset, but ggml-backend-impl.h didn't yet define those ggml_backend_i fields, and ggml_backend_log_profile_info() (declared in ggml-backend.h) had no definition — both needed for that file to even compile under -DOLLAMA.

Changes

• include/llama.h, src/llama.cpp: add llama_backend_log_profile(); restore the ggml_backend_free(TSAVORITE) call in llama_backend_free().
• ggml/include/ggml-backend.h, ggml/src/ggml-backend.cpp: add ggml_backend_log_profile_info().
• ggml/src/ggml-backend-impl.h: add graph_reserve, buffer_size, reset, profile fields to struct ggml_backend_i.
• ggml/src/ggml-tsavorite/ggml-tsavorite.cpp: expose ggml_perf_accumulate, ggml_perf_log_open, ggml_perf_write_detailed_csv, ggml_backend_type via ggml_backend_tsavorite_get_proc_address().
• src/llama-context.cpp: add a shared llama_resolve_ggml_proc_address() helper plus #ifdef OLLAMA-gated wrappers (llama_perf_ggml_accumulate, llama_perf_ggml_log_open, llama_perf_ggml_write_detailed_csv, llama_perf_ggml_backend_type) that resolve these symbols dynamically at runtime through the backend registry. Native (non-OLLAMA) builds keep the original direct calls, unchanged.

Scope is intentionally llama.cpp-only. Ollama picks this up via a separate Makefile.sync commit-id bump once this merges -- no ollama-side changes in this PR.

Also included (found while investigating a separate, still-open multi_thread_enable:true intermittent crash in ollama, unrelated to the perf-symbol work above but touching the same file):

• Fix a real data race in tsi_init_per_txe_state_once(): unsynchronized std::vector mutation (packed_args, device_free) on first use from concurrent callers.
• Harden call_triton_matmul_full_packed(): its static descriptor buffers were populated with no synchronization before reaching the tsi_pack_mutex-protected dispatch call.

These are real, standalone concurrency fixes, but do not fully resolve the intermittent multi_thread_enable:true SIGSEGV in the Triton MAT_MUL dispatch path (still under investigation, tracked separately).

Validation

Tested in ollama (posix build, SDK_VERSION=0.4.24), via a temporary local sync pointed at this branch:

• Native llama.cpp posix debug build compiles and links cleanly with all changes (sanity check independent of ollama).
• Ollama posix debug build (GGML_PERF_DETAIL) — full detail table prints correctly, matching llama.cpp's own tsi-pkg-build.sh dev output format:

  === GGML Perf Summary ===
    Op               Target      Runs  TSI_KERNEL-RUN          Total us            Avg us
    ADD              OPU          480        30496556            677744           1411.97
    MUL              OPU          490        30847744            653504           1333.68
    RMS_NORM         OPU           10          970154             14996           1499.60
    MUL_MAT          OPU          165        51087589          63340037         383879.01
    CONT             OPU          240        15149184             16538             68.91
    GET_ROWS         OPU           30         2201664              2113             70.43
    SOFT_MAX         OPU          240        15096192             27038            112.66
    GLU              OPU          240        15274774            696186           2900.78
  

  Verified via qwen:0.5b, HTTP 200, no crash.
• Ollama posix release build (GGML_PERF_RELEASE) — plain customer-safe table prints correctly, no CPU/OPU breakdown:

  === GGML Perf Summary ===
    Op                   Runs        Total us            Avg us
    ADD                   144          287062           1993.49
    MUL                   147          558433           3798.86
    RMS_NORM                3           37089          12363.00
    MUL_MAT               165        61081193         370189.05
    CONT                   72           16335            226.88
    GET_ROWS                9            2070            230.00
    SOFT_MAX               72           24200            336.11
    GLU                    72          545814           7580.75
  

  Verified via qwen:0.5b, HTTP 200, no crash.
• Both tested with multi_thread_enable:false (the known-stable path); multi_thread_enable:true remains separately tracked as an open, intermittent issue unrelated to this fix.

---

Summary by cubic

Restores GGML perf-summary tables in ollama split-binary builds and adds a backend profile hook. Previously llama-context.cpp could not reach ggml_perf_*/ggml_backend_type and printed no tables; now runtime resolution restores detailed and release tables, and exports for detailed CSV are correctly guarded to prevent release build link errors. Addresses FIR-2216.

• Dynamic perf symbols for ollama: ggml-tsavorite.cpp exposes ggml_perf_accumulate and ggml_backend_type in all perf modes, and exposes ggml_perf_log_open/ggml_perf_write_detailed_csv only under GGML_PERF_DETAIL; src/llama-context.cpp adds OLLAMA-only wrappers that resolve these at runtime with atomic caches and retry after late backend registration; native llama.cpp builds keep direct calls.
• Backend profiling: adds ggml_backend_log_profile_info() and optional graph_reserve, buffer_size, reset, profile in struct ggml_backend_i. Introduces llama_backend_log_profile() and caches a single TSAVORITE backend; tracks when the profile hook finalizes the runtime to avoid double teardown in llama_backend_free().
• Concurrency hardening in ggml-tsavorite: serializes first-use lazy init in tsi_init_per_txe_state_once() via an atomic flag plus mutex; serializes call_triton_matmul_full_packed() descriptor populate+dispatch; on all teardown paths, clears packed_args and all scalar_* per‑TXE vectors and resets the init flag to force clean reallocation on the next init.
• Rollout: no migration required; ollama will pick this via Makefile sync.

^{Written for commit bcb7c7e. Summary will update on new commits.}

[qodo-code-review]

PR Summary by Qodo

Expose ggml perf/profile symbols for Ollama split-binary builds (llama.cpp + Tsavorite)

✨ Enhancement 🐞 Bug fix 🕐 40+ Minutes

AI Description

• Resolve ggml perf-summary symbols at runtime for Ollama’s split-binary backend loading model.
• Add backend profile logging hook and missing ggml_backend_i interface fields used by Tsavorite.
• Fix Tsavorite multi-thread races in lazy init and static descriptor population paths.

Diagram

graph TD
  A["Ollama Go binary"] --> B["llama-context.cpp"] --> C["ggml backend registry"]
  C --> D["libggml-*.so"] --> E["perf/profile symbols"]
  B --> F["perf summary output"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Export perf symbols from libggml-base and link against it

• ➕ Keeps direct calls (no runtime lookup/branching)
• ➕ Failures surface at link/load time rather than silently returning nullptr
• ➖ Doesn’t fit Ollama’s current split-binary/static-link constraints as cleanly
• ➖ Still requires coordinated build/link changes across Ollama packaging

2. Use dlsym(RTLD_DEFAULT) / platform loader APIs directly

• ➕ Simpler than iterating the ggml backend registry
• ➕ Decouples from registry semantics
• ➖ More platform-specific and fragile across loader flags/visibility
• ➖ Harder to control which backend provides the symbol when multiple are loaded

3. Move perf-summary helpers into a ggml-shared shim API

• ➕ Single stable API surface for split-binary consumers
• ➕ Avoids exposing internal function names as stringly-typed lookups
• ➖ More design/work upfront; larger scope than this targeted fix
• ➖ Requires ongoing API/versioning commitments

Recommendation: The PR’s registry-based runtime resolution is the most practical, low-scope solution for Ollama’s split-binary model: it keeps native llama.cpp builds unchanged while unblocking perf logging when symbols live in dlopen’d backends. Consider optionally adding a warning/log when a symbol cannot be resolved (instead of silently no-op) to improve diagnosability in misconfigured deployments.

Files changed (7) +135 / -6

Enhancement (5) +93 / -6

ggml-backend.hExpose ggml_backend_log_profile_info() in public API +1/-0

Expose ggml_backend_log_profile_info() in public API

• Adds a public API entry point to request backend-specific profile logging. This allows callers (e.g., llama.cpp / Ollama integration) to trigger backend profiling output without reaching into backend internals.

ggml/include/ggml-backend.h

ggml-backend-impl.hExtend ggml_backend_i with Ollama/Tsavorite hooks +9/-0

Extend ggml_backend_i with Ollama/Tsavorite hooks

• Adds optional interface fields (graph_reserve, buffer_size, reset, profile) to ggml_backend_i. These match fields already referenced by Tsavorite under -DOLLAMA and enable profile/logging plumbing.

ggml/src/ggml-backend-impl.h

ggml-backend.cppImplement ggml_backend_log_profile_info() dispatcher +10/-0

Implement ggml_backend_log_profile_info() dispatcher

• Implements a null-safe wrapper that calls the backend’s optional profile hook when present. This standardizes how higher layers request backend profile info.

ggml/src/ggml-backend.cpp

llama.hAdd llama_backend_log_profile() API +3/-0

Add llama_backend_log_profile() API

• Adds a new public llama API to request backend profile logging at program end (documented as MPI-related today). This mirrors llama_backend_free() in lifecycle placement.

include/llama.h

llama-context.cppOLLAMA-only dynamic resolution wrappers for ggml perf APIs +70/-6

OLLAMA-only dynamic resolution wrappers for ggml perf APIs

• Introduces a helper to resolve ggml symbols through the ggml backend registry at runtime, with cached function pointers per symbol. Replaces direct perf calls with wrappers so Ollama split-binary builds can still print perf summaries and detailed CSV while non-OLLAMA builds remain unchanged.

src/llama-context.cpp

Bug fix (2) +42 / -0

ggml-tsavorite.cppFix multi-thread races and export perf symbols via proc-address +36/-0

Fix multi-thread races and export perf symbols via proc-address

• Adds mutex protection around per-TXE lazy allocations and around shared static descriptor population in a Triton matmul path to prevent data races under multi_thread_enable=true. Also exposes ggml_perf_* and ggml_backend_type through ggml_backend_tsavorite_get_proc_address() so Ollama’s statically-linked llama-context can resolve them at runtime.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp

llama.cppWire llama_backend_log_profile() and restore Tsavorite backend free +6/-0

Wire llama_backend_log_profile() and restore Tsavorite backend free

• Implements llama_backend_log_profile() by delegating to ggml_backend_log_profile_info() for the TSAVORITE backend. Restores freeing the TSAVORITE backend in llama_backend_free() to avoid leaks in that configuration.

src/llama.cpp

[cubic-dev-ai]

All reported issues were addressed across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

1. Profile backend leaked ✓ Resolved 🐞 Bug ☼ Reliability

Description

llama_backend_log_profile() creates a new TSAVORITE backend via ggml_backend_init_by_name() and
calls ggml_backend_log_profile_info() without ever freeing the returned backend handle. Under
OLLAMA, the TSAVORITE profile hook also unloads blobs and calls tsi_finalize(), so leaving the
backend instance alive after that teardown risks inconsistent shutdown state.

Code

src/llama.cpp[R93-96]

+extern "C"
+void llama_backend_log_profile(void) {
+    ggml_backend_log_profile_info(ggml_backend_init_by_name("TSAVORITE", NULL));
+}

Relevance

●●● Strong

Resource leak/teardown symmetry fixes are typically accepted; backend init without free is clear
lifecycle bug.

PR-#149

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new API constructs a backend instance but does not destroy it; TSAVORITE’s registered profile
callback performs global finalization, so leaking the backend handle after profiling is unsafe.

src/llama.cpp[93-101]
ggml/src/ggml-backend-reg.cpp[410-416]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[7222-7242]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[7245-7265]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`llama_backend_log_profile()` initializes a TSAVORITE backend and calls `ggml_backend_log_profile_info()` but never frees the backend handle. This leaks backend resources and, for TSAVORITE under `OLLAMA`, the profiling hook performs global teardown (`tsi_finalize()`), so the backend lifetime should be explicitly managed.

## Issue Context
- `ggml_backend_init_by_name()` returns a newly initialized backend that the caller owns.
- TSAVORITE’s `.profile` implementation (`tsi_log_profile_info`) performs global finalization/unload.

## Fix Focus Areas
- src/llama.cpp[93-101]
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[7222-7242]
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[7245-7265]

## Suggested fix
1. Store the result of `ggml_backend_init_by_name("TSAVORITE", nullptr)` in a local `ggml_backend_t backend`.
2. If `backend != nullptr`, call `ggml_backend_log_profile_info(backend)`.
3. Ensure `ggml_backend_free(backend)` is called (or explicitly document/implement correct ordering if TSAVORITE’s profile hook finalizes global state in a way that makes `backend->iface.free()` unsafe afterward).
4. Consider making TSAVORITE profiling a backend-reg/device-level operation (no backend instance creation) if `.profile` is intended to be global teardown.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Cached NULL proc lookup ✓ Resolved 🐞 Bug ≡ Correctness

Description

The OLLAMA perf wrappers cache resolved function pointers in function-local statics, so if the first
lookup happens when no backend exports the symbol, the cached nullptr is never retried and perf
reporting remains disabled for that symbol. This is especially risky because backends can be loaded
dynamically after startup via the backend registry.

Code

src/llama-context.cpp[R985-986]

+    static auto fn = reinterpret_cast<ggml_perf_accumulate_t>(llama_resolve_ggml_proc_address("ggml_perf_accumulate"));
+    if (fn) {

Relevance

●●● Strong

Team has flagged function-local static caching pitfalls; likely to avoid permanently caching nullptr
from dynamic proc lookup.

PR-#149

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The wrapper stores a single lookup result in a static variable; backend registries can be populated
later via dynamic loading, so a first miss becomes permanent.

src/llama-context.cpp[962-1006]
ggml/src/ggml-backend-reg.cpp[436-443]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
In OLLAMA builds, the perf wrappers resolve symbols once into `static auto fn = ...`. If resolution returns `nullptr` on the first call (e.g., before a backend providing the symbol is loaded/registered), subsequent calls will never retry, permanently disabling that functionality for the process lifetime.

## Issue Context
Backends can be loaded dynamically through the registry; the symbol may become available after the first attempt.

## Fix Focus Areas
- src/llama-context.cpp[962-1006]
- ggml/src/ggml-backend-reg.cpp[436-443]

## Suggested fix
Replace the `static auto fn = ...` pattern with a cache that retries when null, e.g.:
- `static ggml_perf_accumulate_t fn = nullptr; if (!fn) fn = reinterpret_cast<...>(llama_resolve_ggml_proc_address(...));`
- Optionally add a lightweight guard to avoid repeated scanning once resolved.
Apply the same pattern to `ggml_perf_log_open`, `ggml_perf_write_detailed_csv`, and `ggml_backend_type`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. llama_perf_ggml_write_detailed_csv line too long ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

A newly added line in src/llama-context.cpp exceeds the 120-character limit, reducing readability
and violating the project's C++ line-length standard.

Code

src/llama-context.cpp[997]

+    static auto fn = reinterpret_cast<ggml_perf_write_detailed_csv_t>(llama_resolve_ggml_proc_address("ggml_perf_write_detailed_csv"));

Relevance

●●● Strong

Style/readability nits in this file have been accepted before; splitting long line is trivial.

PR-#66

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2281062 requires that modified C++ files have no lines longer than 120 characters.
The added line that initializes fn for ggml_perf_write_detailed_csv is a single long statement
that exceeds this limit.

Rule 2281062: Limit C++ source lines to 120 characters
src/llama-context.cpp[996-998]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A newly added C++ line exceeds the 120-character maximum.

## Issue Context
`src/llama-context.cpp` contains a long `reinterpret_cast` expression that should be wrapped across multiple lines to keep each line at 120 characters or fewer.

## Fix Focus Areas
- src/llama-context.cpp[996-998]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Init mutex always locked ✓ Resolved 🐞 Bug ➹ Performance

Description

tsi_init_per_txe_state_once() now takes tsi_init_mutex on every call, even after initialization
completes, adding a global lock to a function called on each multi-threaded dispatch. This can
create avoidable contention and throughput loss in hot paths.

Code

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[R1539-1540]

+    std::lock_guard<std::mutex> lock(tsi_init_mutex);
+

Relevance

●● Moderate

Locks added for race fix; team may keep for safety, but might optimize (call_once/atomic) later.

PR-#110
PR-#152

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new mutex is acquired at the start of tsi_init_per_txe_state_once(), and the function is invoked
before each dispatch, so the lock is on the hot path.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[1532-1540]
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[4559-4566]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`tsi_init_per_txe_state_once()` uses a mutex unconditionally, even when all lazy state is already initialized, despite being called on every dispatch under `multi_thread_enable=true`.

## Issue Context
The lock is needed for first-use safety, but once initialization is complete, a fast path can avoid repeated mutex acquisition.

## Fix Focus Areas
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[1532-1540]
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[4559-4566]

## Suggested fix
Implement a fast path such as:
- Use `std::once_flag`/`std::call_once` if initialization is truly one-time and `num_of_txes` is stable.
- Or use double-checked locking: check an atomic/flag (and required sizes) without locking; only lock and initialize/resize when needed.
Ensure the existing race fix remains intact.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (1)

5. Matmul serialized globally 🐞 Bug ➹ Performance

Description

call_triton_matmul_full_packed() now serializes the entire populate+dispatch sequence behind a
static process-wide mutex, preventing concurrent matmul launches from overlapping across
threads/devices. This can significantly reduce scalability when multi_thread_enable=true and
multiple calls are in flight.

Code

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[R4850-4852]

+    std::lock_guard<std::mutex> lock(g_full_packed_static_mutex);
+

Relevance

●● Moderate

Global serialization fixes shared-static race; performance concern plausible but correctness likely
prioritized unless better design offered.

PR-#149
PR-#152

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR introduces a static mutex and takes it at function entry, making this matmul path
single-threaded across the entire process for the duration of each call.

ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[4830-4852]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A new global mutex (`g_full_packed_static_mutex`) is used to guard shared static descriptor/payload buffers in `call_triton_matmul_full_packed()`, but it also serializes the entire dispatch path, reducing concurrency.

## Issue Context
The lock fixes a real shared-buffer race, but the long-term solution is to remove the shared mutable static buffers.

## Fix Focus Areas
- ggml/src/ggml-tsavorite/ggml-tsavorite.cpp[4830-4852]

## Suggested fix
Refactor to eliminate process-wide shared mutable statics, e.g.:
- Make descriptors/payload buffers `thread_local`, or
- Move them into per-device/per-worker storage (similar to existing per-device structures), or
- Allocate descriptors on the stack and use an aligned scalar storage strategy.
Keep synchronization sufficient to preserve correctness, but avoid a single global mutex as the scaling limiter.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context

✅ Compliance rules (platform): 6 rules

Tip of the day

💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[akapoor3518]

Addressed all 5 findings from this summary in aea1a75:

1. Profile backend leaked -- fixed. llama_backend_log_profile() and llama_backend_free() now share a single lazily-created TSAVORITE backend handle, created once and freed exactly once.
2. Line too long (llama_perf_ggml_write_detailed_csv) -- fixed, wrapped to stay under 120 columns (and audited the other 3 wrappers for the same).
3. Cached NULL proc lookup -- fixed. All four OLLAMA dynamic-lookup wrappers now retry resolution instead of permanently caching a null result.
4. Init mutex always locked -- fixed. tsi_init_per_txe_state_once() now has a fast atomic pre-check so the mutex is only touched during the actual first-use init window, not on every dispatch.
5. Matmul serialized globally -- not changing. Traced call_triton_matmul_full_packed()'s one call site (ggml_tsavorite_run_tmu_mul_mat, invoked from the single sequential per-node dispatch loop) -- it's never called concurrently with itself, so the mutex is uncontended in practice. The suggested thread_local/per-device buffer refactor is legitimate future work but out of scope for this targeted fix.

Native llama.cpp posix debug build and ollama posix debug build both verified clean after these changes.

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[atrivedi-tsavoritesi]

Hi Anoop,

Did you test these changes with Ollama ? As the llama-context.cpp and some other changes in the PR may conflict with changes in Ollama.

If you have can you also post a draft PR for Ollama as I am not clear on the Ollama side of the changes.

Thanks
Ashish

[akapoor3518]

Hi @atrivedi-tsavoritesi,

Ollama-side change is just a commit-id update in Makefile.sync -- no other Ollama code changes, so no conflict. Tested at Ollama (posix, qwen:0.5b, both release and debug table formats), results are in the PR description under Validation. Please take a look.