Llm and custom metrics

custom_metrics_config.json

@@ -0,0 +1,10 @@
+{
+  "model_config_list": [],
+  "mediapipe_config_list": [
+    {
+      "name": "small_llm",
+      "base_path": "/ovms/models/OpenVINO/Qwen2-0.5B-Instruct-int4-ov",
+      "graph_path": "/ovms/models/OpenVINO/Qwen2-0.5B-Instruct-int4-ov/graph.pbtxt"
+    }
+  ]
+}

docs/metrics.md

@@ -46,6 +46,8 @@ Optional metrics
 | :---    |    :----   |    :----   |    :----       |
 | gauge      | ovms_infer_req_queue_size | name,version | Inference request queue size (nireq). |
 | gauge      | ovms_infer_req_active | name,version | Number of currently consumed inference requests from the processing queue that are now either in the data loading or inference process. |
+| gauge      | ovms_llm_kv_cache_utilization_percent | name,node | GenAI-reported cache utilization (0-100) for a continuous-batching LLM or VLM pipeline. |
+| histogram      | ovms_llm_prefill_speed_tokens_per_second | name,node | Per-request input token count divided by GenAI time to first token in seconds, for continuous-batching LLM and VLM pipelines. |
 
 > **Note**: While `ovms_current_requests` and `ovms_infer_req_active` both indicate how much resources are engaged in the requests processing, they are quite distinct. A request is counted in `ovms_current_requests` metric starting as soon as it's received by the server and stays there until the response is sent back to the user. The `ovms_infer_req_active` counter informs about the number of OpenVINO Infer Requests that are bound to user requests and are either loading the data or already running inference. 
 
@@ -57,7 +59,53 @@ Labels description
 | method      | ModelMetadata, ModelReady, ModelInfer, Predict, GetModelStatus, GetModelMetadata, Unary, Stream | Interface methods. |
 | version      | 1, 2, ..., n | Model version. Note that GetModelStatus and ModelReady and all MediaPipe servables do not have the version label. |
 | name      | As defined in model server config | Model name, DAG name or MediaPipe graph name. |
+| node      | As defined in the graph configuration | LLM calculator node name; distinguishes independent pipelines within a graph. |
 
+## LLM-specific metrics
+
+The two `ovms_llm_*` families are optional and supported for continuous-batching
+language and vision-language models, not legacy/NPU or Omni pipelines. Enable them
+with a REST port and an explicit metrics list:
+
+```bash
+ovms --config_path /models/config.json --rest_port 8000 --metrics_enable \
+    --metrics_list ovms_llm_kv_cache_utilization_percent,ovms_llm_prefill_speed_tokens_per_second
+curl http://localhost:8000/metrics
+```
+
+An explicit list selects only the listed families. Include other metric names if
+you also need the default serving metrics. The same names can be selected in
+`monitoring.metrics.metrics_list` in the configuration file instead of the CLI.
+
+Cache utilization is published by the executor after each generation step and
+before waiting for new requests. It is GenAI's last-step maximum cache utilization
+percentage across registered cache types, not total device-memory utilization.
+The denominator can change with dynamic cache allocation. Idle pipelines retain
+the last reported sample; idle does not imply an empty cache or a zero value.
+
+Prefill speed is observed once per successfully completed request, for both unary
+and streaming responses, independently of the log level. Cancelled or failed
+requests are not observed. It is an effective TTFT-derived rate, not isolated
+prefill kernel throughput: scheduling, first-token generation and prefix-cache
+hits affect it. For VLMs, the GenAI TTFT excludes embedding preparation. Invalid
+or non-positive speed samples are skipped with a warning.
+
+The histogram exports `_bucket`, `_sum`, and `_count` series. Its finite bucket
+boundaries, in tokens per second, are `1, 10, 50, 100, 250, 500, 1000, 2500,
+5000, 10000, 25000, 50000, 100000`. For example, the mean per-request speed over
+five minutes is:
+
+```promql
+sum by (name, node) (rate(ovms_llm_prefill_speed_tokens_per_second_sum[5m]))
+/
+sum by (name, node) (rate(ovms_llm_prefill_speed_tokens_per_second_count[5m]))
+```
+
+This is not aggregate token throughput or a token-weighted average. Values are
+retained in the in-memory registry between scrapes; scraping does not query the
+GenAI pipeline. Series are deactivated and removed when the graph unloads or
+reloads, even if an in-flight request still retains the old pipeline. Replacement
+pipelines start fresh series; late updates from the old pipeline are ignored.
 
 ## Enable metrics
 
@@ -229,7 +277,129 @@ For [MediaPipe Graphs](./mediapipe.md) execution there are 6 generic metrics whi
 | histogram      | ovms_graph_processing_time_us | Tracks duration of successfully started mediapipe graphs in us. It can represent pipeline processing time for unary calls or the session length for streamed requests. |
 | histogram      | ovms_graph_request_latency_us | Difference between MediaPipe packet timestamps at the time of creation (request receival) and when the packet generation has concluded (full or partial response). Server automatically timestamps the incoming  packets and is used as a source of truth for receival time, unless client opts-in to manual timestamping (NOTE: manual timestamping causes the metric results to become invalid). |
 
-Exposing custom metrics in calculator implementations (MediaPipe graph nodes) is not supported yet.
+## Custom metrics in C++ calculators
+
+Calculators can register **custom counters** and update them during processing.
+The API is deliberately small:
+
+```cpp
+auto events = metrics->registerCounter("custom_events_total", "Events processed.");
+events->increment();       // Add one.
+events->increment(3.0);    // Add a batch.
+```
+
+`metrics` is an OVMS-provided `std::shared_ptr<ovms::CalculatorMetrics>`, scoped
+to one graph and one named calculator node. The returned
+`std::shared_ptr<ovms::CalculatorCounter>` is a thread-safe handle, not a local
+copy of the counter. OVMS stores the accumulated value in the shared Prometheus
+registry and exposes it on the existing `/metrics` endpoint. No separate cache,
+exporter, background thread or scrape callback is necessary.
+
+This API currently supports counters only, not gauges or histograms. It is a
+**C++ extension API**: implementing new metric logic requires compiling the
+calculator and linking it into OVMS; a graph file alone cannot define arbitrary
+counting logic. It does not expose a Python metrics API.
+
+### 1. Enable metrics and add the side packet
+
+Enable the normal metrics endpoint using `--rest_port 8000 --metrics_enable`, or
+the equivalent monitoring configuration. Then add this input side packet to the
+calculator node in its graph:
+
+```protobuf
+node {
+    name: "my_node"
+    calculator: "MyCalculator"
+    input_side_packet: "CALCULATOR_METRICS:calculator_metrics"
+    # Other inputs, outputs and calculator options...
+}
+```
+
+OVMS supplies the packet; do not configure an external side-packet value.
+Use the tag and packet name exactly as shown (`CALCULATOR_METRICS:calculator_metrics`).
+The node must have a stable, non-empty `name`. Each custom series automatically
+gets `name="<graph name>"` and `node="<calculator node name>"` labels.
+
+### 2. Register a counter in the calculator
+
+Include `src/metrics/calculator_metrics.hpp` and depend on
+`//src/metrics:libovmsmetrics` in the calculator's Bazel target.
+Declare the side-packet type in `GetContract`, acquire the node's scope in `Open`,
+and retain the returned handle as a calculator member:
+
+```cpp
+std::shared_ptr<ovms::CalculatorCounter> events;
+
+static absl::Status GetContract(mediapipe::CalculatorContract* cc) {
+    // Keep the calculator's existing input/output contract here.
+    cc->InputSidePackets()
+        .Tag(ovms::CALCULATOR_METRICS_TAG)
+        .Set<ovms::CalculatorMetricsMap>();
+    return absl::OkStatus();
+}
+
+absl::Status Open(mediapipe::CalculatorContext* cc) override {
+    const auto& scopes = cc->InputSidePackets()
+        .Tag(ovms::CALCULATOR_METRICS_TAG)
+        .Get<ovms::CalculatorMetricsMap>();
+    const auto scope = scopes.find(cc->NodeName());
+    RET_CHECK(scope != scopes.end()) << "Missing calculator metrics scope";
+    try {
+        events = scope->second->registerCounter(
+            "custom_events_total", "Events processed.");
+    } catch (const std::exception& e) {
+        return absl::InvalidArgumentError(e.what());
+    }
+    return absl::OkStatus();
+}
+```
+
+Use the normal MediaPipe headers for these methods, plus `<memory>` and
+`<exception>`. For an optional integration, guard both contract declaration and
+`Open` lookup with `cc->InputSidePackets().HasTag(ovms::CALCULATOR_METRICS_TAG)`;
+the built-in LLM example follows that pattern.
+
+Registration is idempotent: identical registrations from concurrent calculator
+instances in the same graph/node return the same counter. Register in `Open`,
+not for every packet. Different nodes or graphs get independent series.
+
+### 3. Update during processing
+
+Once an event has occurred, call the retained handle:
+
+```cpp
+events->increment();
+```
+
+To count multiple events in a packet, call
+`events->increment(static_cast<double>(eventCount))`.
+No registry lookup, metric-name construction or registration is needed in
+`Process`. Scrapes do not reset counters; use Prometheus `rate()` or `increase()`
+to calculate changes over time.
+
+### Naming, configuration and lifetime
+
+- Names must match `custom_<name>_total`, containing only ASCII letters, digits
+  and underscores, with a non-empty middle name and help text. The `custom_`
+  prefix keeps application counters separate from built-in `ovms_` metrics.
+- The same metric name must have identical help text across nodes and graphs.
+  Invalid names, conflicting definitions and negative/non-finite increments
+  are logged and throw exceptions; do not suppress those errors.
+- Metric names must be fixed in calculator code. Do not construct names from
+  words, users, requests or timestamps. This initial API intentionally exposes
+  only the fixed graph/node labels to keep cardinality bounded.
+- Custom counters are controlled by the global metrics-enable setting and the
+  calculator's own registration logic, **not** by the built-in `metrics_list`.
+  Do not put custom names in that list: its parser accepts built-in names only.
+  Custom counters still work when a list selects a subset of built-in metrics.
+- With metrics disabled, registration returns a valid no-op handle, so processing
+  need not contain null checks. Names and increments are still validated.
+- Counters persist across requests and calculator `Close` calls. Do not remove
+  them in `Close`: other calculator instances can share the series.
+- OVMS deactivates scopes and removes series on graph unload/reload. Old handles
+  safely ignore valid late updates and cannot affect replacement series.
+  Re-registering through an inactive scope fails. Reloads and server restarts
+  reset counters; Prometheus handles counter resets in `rate()`/`increase()`.
 
 
 ## Visualize with Grafana

src/audio/speech_to_text/stt_node_initializer.cpp

@@ -39,7 +39,9 @@ class SttNodeInitializer : public NodeInitializer {
         const std::string& graphName,
         const std::string& basePath,
         GraphSidePackets& sidePackets,
-        PythonBackend* /*pythonBackend*/) override {
+        PythonBackend* /*pythonBackend*/,
+        MetricRegistry* /*metricRegistry*/,
+        const MetricConfig* /*metricConfig*/) override {
         auto& sttServableMap = sidePackets.sttServableMap;
         if (!nodeConfig.node_options().size()) {
             SPDLOG_ERROR("SpeechToText node missing options in graph: {}. ", graphName);

src/audio/text_to_speech/tts_node_initializer.cpp

@@ -40,7 +40,9 @@ class TtsNodeInitializer : public NodeInitializer {
         const std::string& graphName,
         const std::string& basePath,
         GraphSidePackets& sidePackets,
-        PythonBackend* /*pythonBackend*/) override {
+        PythonBackend* /*pythonBackend*/,
+        MetricRegistry* /*metricRegistry*/,
+        const MetricConfig* /*metricConfig*/) override {
         auto& ttsServableMap = sidePackets.ttsServableMap;
         if (!nodeConfig.node_options().size()) {
             SPDLOG_ERROR("TextToSpeech node missing options in graph: {}. ", graphName);

src/embeddings/embeddings_node_initializer.cpp

@@ -51,7 +51,9 @@ class EmbeddingsNodeInitializer : public NodeInitializer {
         const std::string& graphName,
         const std::string& basePath,
         GraphSidePackets& sidePackets,
-        PythonBackend* /*pythonBackend*/) override {
+        PythonBackend* /*pythonBackend*/,
+        MetricRegistry* /*metricRegistry*/,
+        const MetricConfig* /*metricConfig*/) override {
         auto& embeddingsServableMap = sidePackets.embeddingsServableMap;
         if (!nodeConfig.node_options().size()) {
             SPDLOG_ERROR("Embeddings node missing options in graph: {}. ", graphName);

src/image_gen/image_gen_node_initializer.cpp

@@ -43,7 +43,9 @@ class ImageGenNodeInitializer : public NodeInitializer {
         const std::string& graphName,
         const std::string& basePath,
         GraphSidePackets& sidePackets,
-        PythonBackend* /*pythonBackend*/) override {
+        PythonBackend* /*pythonBackend*/,
+        MetricRegistry* /*metricRegistry*/,
+        const MetricConfig* /*metricConfig*/) override {
         auto& imageGenPipelinesMap = sidePackets.imageGenPipelinesMap;
         if (!nodeConfig.node_options().size()) {
             SPDLOG_ERROR("Image Gen node missing options in graph: {}. ", graphName);

src/llm/BUILD

@@ -658,6 +658,7 @@ ovms_cc_library(
             "omni_model/legacy/servable_initializer.cpp",
             "omni_model/legacy/legacy_executor.cpp"],
     deps = [
+        "//src/metrics:libovmsmetrics",
         "//third_party:openvino",
         "@mediapipe//mediapipe/framework:calculator_cc_proto",
         "@mediapipe//mediapipe/framework/port:ret_check",
@@ -759,5 +760,3 @@ ovms_cc_library(
     ],
     visibility = ["//visibility:public"],
 )
-
-

src/llm/language_model/continuous_batching/llm_executor.hpp

@@ -28,6 +28,7 @@
 #include <openvino/genai/continuous_batching_pipeline.hpp>
 
 #include "../../../logging.hpp"
+#include "../../../metrics/llm_metric_reporter.hpp"
 #include "../../../profiler.hpp"
 
 namespace ovms {
@@ -37,11 +38,13 @@ struct LLMExecutor {
     std::mutex mutex;
     std::condition_variable cv;
     std::shared_ptr<ov::genai::ContinuousBatchingPipeline> pipe = nullptr;
+    std::shared_ptr<LLMMetricReporter> metricReporter;
 
-    LLMExecutor(std::shared_ptr<ov::genai::ContinuousBatchingPipeline> pipe, bool isDynamicKVCacheSet = false) {
-        this->pipe = std::move(pipe);
-        this->isDynamicKVCache = isDynamicKVCacheSet;
-    }
+    LLMExecutor(std::shared_ptr<ov::genai::ContinuousBatchingPipeline> pipe, bool isDynamicKVCacheSet,
+        std::shared_ptr<LLMMetricReporter> metricReporter) :
+        isDynamicKVCache(isDynamicKVCacheSet),
+        pipe(std::move(pipe)),
+        metricReporter(std::move(metricReporter)) {}
 
     bool hasRequests() {
         return (pipe->has_non_finished_requests());
@@ -50,6 +53,13 @@ struct LLMExecutor {
     void step() {
         OVMS_PROFILE_FUNCTION();
         pipe->step();
+        publishCacheUtilization();
+    }
+
+    void publishCacheUtilization() {
+        if (metricReporter && metricReporter->isCacheUtilizationEnabled()) {
+            metricReporter->setCacheUtilization(pipe->get_metrics().cache_usage);
+        }
     }
 
     void waitForRequests(std::atomic<bool>* receivedEndSignal) {
@@ -125,6 +135,7 @@ class LLMExecutorWrapper {
                     stepCounter++;
                     llmExecutor->step();
                 } else {
+                    llmExecutor->publishCacheUtilization();
                     SPDLOG_LOGGER_INFO(llm_executor_logger, "All requests: {}; Scheduled requests: {};", 0, 0);
                     llmExecutor->waitForRequests(receivedEndSignal);
                 }
@@ -136,8 +147,9 @@ class LLMExecutorWrapper {
     }
 
 public:
-    LLMExecutorWrapper(std::shared_ptr<ov::genai::ContinuousBatchingPipeline> pipe, bool isDynamicKVCache = false) :
-        llmExecutor(std::move(pipe), isDynamicKVCache) {
+    LLMExecutorWrapper(std::shared_ptr<ov::genai::ContinuousBatchingPipeline> pipe, bool isDynamicKVCache = false,
+        std::shared_ptr<LLMMetricReporter> metricReporter = nullptr) :
+        llmExecutor(std::move(pipe), isDynamicKVCache, std::move(metricReporter)) {
         llmExecutorThread = std::thread(LLMExecutorWrapper::run, &llmExecutor, &finishExecutorThread);
     }
 

src/llm/language_model/continuous_batching/servable.cpp

@@ -37,9 +37,22 @@
 #include "../../text_utils.hpp"
 #include "llm_executor.hpp"
 #include "servable.hpp"
+#include "../../../metrics/llm_metric_reporter.hpp"
 
 namespace ovms {
 
+void ContinuousBatchingServable::recordPrefillMetrics(GenAiServableExecutionContext& executionContext) {
+    auto& cbExecutionContext = static_cast<ContinuousBatchingServableExecutionContext&>(executionContext);
+    if (!properties->metricReporter || !properties->metricReporter->isPrefillSpeedEnabled() ||
+        !executionContext.perfMetrics || cbExecutionContext.prefillMetricsRecorded) {
+        return;
+    }
+    cbExecutionContext.prefillMetricsRecorded = true;
+    auto& perfMetrics = *executionContext.perfMetrics;
+    properties->metricReporter->observePrefillSpeed(
+        calculatePrefillSpeed(perfMetrics.get_num_input_tokens(), perfMetrics.get_ttft().mean));
+}
+
 void ContinuousBatchingServable::logPerfMetrics(ov::genai::PerfMetrics& perfMetrics) {
     const size_t inputTokenCount = perfMetrics.get_num_input_tokens();
     const size_t outputTokenCount = perfMetrics.get_num_generated_tokens();
@@ -171,6 +184,9 @@ absl::Status ContinuousBatchingServable::readPartialExecutionResults(std::shared
 
 absl::Status ContinuousBatchingServable::prepareCompleteResponse(std::shared_ptr<GenAiServableExecutionContext>& executionContext) {
     auto status = GenAiServable::prepareCompleteResponse(executionContext);
+    if (status.ok()) {
+        recordPrefillMetrics(*executionContext);
+    }
     if (status.ok() && executionContext->perfMetrics && llm_calculator_logger->should_log(spdlog::level::debug)) {
         logPerfMetrics(*executionContext->perfMetrics);
     }
@@ -183,6 +199,9 @@ absl::Status ContinuousBatchingServable::preparePartialResponse(std::shared_ptr<
         stopGenerationOnResponseError(std::static_pointer_cast<ContinuousBatchingServableExecutionContext>(executionContext));
         return status;
     }
+    if (!executionContext->sendLoopbackSignal) {
+        recordPrefillMetrics(*executionContext);
+    }
     if (status.ok() &&
         !executionContext->sendLoopbackSignal &&
         executionContext->perfMetrics &&