Skip to content

server: fix Windows CI flake from the log color reset - #28747

Merged
ServeurpersoCom merged 1 commit into
ggml-org:masterfrom
ServeurpersoCom:server-router-child-cmd-newline
Sep 12, 2026
Merged

ServeurpersoCom merged 1 commit into
ggml-org:masterfrom
ServeurpersoCom:server-router-child-cmd-newline

Conversation

@ServeurpersoCom

@ServeurpersoCom ServeurpersoCom commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview

On Windows the child logs in color, and the logger emits its reset after the newline. That stray escape lands in front of cmd_child_to_router:state: on the shared pipe, the prefix check fails, and download_finished is logged instead of handled, leaving the model stuck downloading. Hence the intermittent test_router_delete_model failures.

Additional information

A leading newline on the command fixes it. Reproduced on Linux with LLAMA_ARG_LOG_COLORS=on, keeping the cached blob while the registry no longer knows the model. pytest unit/test_router.py passes 17/17.

Requirements

The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.

Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
@ServeurpersoCom
ServeurpersoCom requested a review from a team as a code owner September 11, 2026 09:44
@ggerganov

Copy link
Copy Markdown
Member

This is supposed to fix failures such as: https://github.com/ggml-org/llama.cpp/actions/runs/34582172702/job/103208051836#step:6:2054

@ServeurpersoCom

ServeurpersoCom commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Yes for the test_router_download_model failure in that job, its only command line is download_finished and it arrives with the reset glued in front of it. The test_router_delete_model failure in the same job is a different one though: there download_finished is handled correctly and the SSE event goes out, but DELETE /models still hangs the full 600s because the download child never closes its stdout, so the monitoring thread never joins. I open a separate PR for that one.

@ServeurpersoCom

ServeurpersoCom commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author

I sent my agent through every red Server run of the past week to sort the Windows flakes by root cause, here is what it found :

cause                              jobs (7d)  share of router  status

download_finished command lost        42           80%         PR #28747 <- this PR
behind the ANSI reset

DELETE 600s, exit command              5           10%         PR #28755
never sent to the child

DELETE 600s, exit acknowledged         5           10%         PR #28762
but EOF never reached

test_completion_unified               20            -          PR #28759
returning 500

build / infra, misc                   22            -          out of scope

#28755
#28759
#28762

@ServeurpersoCom
ServeurpersoCom merged commit c069aa7 into ggml-org:master Sep 12, 2026
20 of 26 checks passed
pl752 pushed a commit to pl752/llama.cpp that referenced this pull request Sep 15, 2026
…g#28747)

The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.

Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
quimmedes pushed a commit to quimmedes/cafe-llama.cpp that referenced this pull request Sep 16, 2026
…g#28747)

The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.

Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
zsogitbe pushed a commit to zsogitbe/llama.cpp that referenced this pull request Sep 17, 2026
…g#28747)

The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.

Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
Te-eMster pushed a commit to Te-eMster/mx-llama.cpp that referenced this pull request Sep 18, 2026
…g#28747)

The child writes its state commands on stdout while the logger writes
on stderr, and both share a single pipe. The logger emits the trailing
color reset after the newline of a debug, warn or error entry, so that
escape sequence has no newline of its own and the router reads it glued
in front of the next command. The line prefix check then fails and the
command is forwarded as a log line instead of being handled, which
leaves a finished download stuck in the downloading state.

Writing the command with a leading newline closes the pending line so
it always starts at a line boundary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants