Feature Request: Add watchdog with force-kill timeout for hung child processes in router mode #18237

Activity

1. ServeurpersoCom

   added

   enhancementNew feature or requestNew feature or request

   on Dec 20, 2025
2. ServeurpersoCom commented on Dec 20, 2025

   

   ServeurpersoCom

   on Dec 20, 2025

   ContributorAuthor

   More actions

   cc @ngxson to trace what we saw together during tests

   👍React with 👍1Reacted by Xuan-Son Nguyen
3. ServeurpersoCom commented on Dec 22, 2025

   

   ServeurpersoCom

   on Dec 22, 2025

   Last edited by ServeurpersoCom

   ContributorAuthor

   More actions

   The same DoS script (last from #18263 execution only for few seconds and CTRL + C) turns the router into a zombie (OOM on a child) Sometimes the server works again after a while, but sometimes I have to restart the llama-server router.

   An API request rate limitation could be added in addition to cleaning child processes (?)

   [45569] print_info: EOG token        = 151662 '<|fim_pad|>'
   [45569] print_info: EOG token        = 151663 '<|repo_name|>'
   [45569] print_info: EOG token        = 151664 '<|file_sep|>'
   [45569] print_info: max token length = 256
   [45569] load_tensors: loading model tensors, this can take a while... (mmap = true)
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  ensure_model: waiting until model name=MoE-Qwen3-30B-A3B-Thinking-2507 is fully loaded...
   srv  ensure_model: waiting until model name=MoE-Qwen3-30B-A3B-Thinking-2507 is fully loaded...
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   [45569] ggml_backend_cuda_buffer_type_alloc_buffer: allocating 23680.99 MiB on device 0: cudaMalloc failed: out of memory
   [45569] alloc_tensor_range: failed to allocate CUDA0 buffer of size 24831322112
   [45569] llama_model_load: error loading model: unable to allocate CUDA0 buffer
   [45569] llama_model_load_from_file_impl: failed to load model
   [45569] common_init_from_params: failed to load model 'mradermacher/Qwen3-30B-A3B-Thinking-2507-i1-GGUF/Qwen3-30B-A3B-Thinking-2507.i1-Q6_K.gguf'
   [45569] srv    load_model: failed to load model, 'mradermacher/Qwen3-30B-A3B-Thinking-2507-i1-GGUF/Qwen3-30B-A3B-Thinking-2507.i1-Q6_K.gguf'
   [45569] srv    operator(): operator(): cleaning up before exit...
   [45569] main: exiting due to model loading error
   srv    operator(): instance name=MoE-Qwen3-30B-A3B-Thinking-2507 exited with status 1
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv    operator(): got exception: {"error":{"code":500,"message":"model name=MoE-Qwen3-30B-A3B-Thinking-2507 failed to load","type":"server_error"}}
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  log_server_r: request: GET /v1/models 127.0.0.1 200
   srv  log_server_r: request: GET /v1/models 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   srv  proxy_reques: proxying request to model MoE-Qwen3-30B-A3B-Instruct-2507 on port 36009
   srv    operator(): http client error: Could not establish connection
   srv  log_server_r: request: GET /props 127.0.0.1 500
   srv  log_server_r: request: GET /props 127.0.0.1 200
   
4. ServeurpersoCom

   closed this as completedin #18350on Dec 24, 2025