Skip to content

[Bug]: Kubernetes, Slurm: effective resource requirements are not used #4019

Description

@un-def

Steps to reproduce

  1. Prepare a fleet:

    $ dstack fleet -v
    
     NAME     NODES  RESOURCES                             SPOT       BACKEND  PRICE  STATUS  CREATED      ERROR
     backend  0..    cpu=2.. mem=2GB.. disk=0GB.. gpu=0..  on-demand  *        -      active  23 mins ago
    

    Note: CPU >= 2, memory >=2

  2. Submit a run:

     $ dstack --cpu 1.. --memory 1GB.. -b slurm -b kubernetes
    
      Project              main
      User                 dmitry-local-admin
      Type                 dev-environment
      Resources            cpu=1.. mem=1GB.. disk=1GB.. gpu=0..
      Spot policy          on-demand
      Max price            off
      Retry policy         off
      Idle duration        5m
      Max duration         off
      Inactivity duration  off
    
      #  BACKEND              RESOURCES               INSTANCE TYPE                       PRICE
      1  kubernetes (nebius)  cpu=2 mem=2GB disk=1GB  computeinstance-e00ef6jy764y8b2jps  $0
      2  kubernetes (nebius)  cpu=2 mem=2GB disk=1GB  computeinstance-e00h474t5jwnkkwg8a  $0
      3  slurm (vm)           cpu=2 mem=2GB disk=1GB  worker-1                            $0
      4  slurm (vm)           cpu=2 mem=2GB disk=1GB  worker-2                            $0
    

    Note, the run requests CPU >=1, memory >= 1, but the offers are adjusted according to the effective (fleet/run combined) requirements (CPU >= 2, memory >=2)

  3. Check resources actually requested

Actual behaviour

$ kubectl get -n dstack-dmitry pod dstack-main-dev-environment-0-0-tjrihv30 -o jsonpath='{.spec.containers[*].resources}'
{"requests":{"cpu":"1","ephemeral-storage":"1Gi","memory":"1Gi"}}
$ scontrol show job 271 | grep ReqTRES
   ReqTRES=cpu=1,mem=1G,node=1,billing=1

Expected behaviour

{"requests":{"cpu":"2","ephemeral-storage":"1Gi","memory":"2Gi"}}
ReqTRES=cpu=2,mem=2G,node=1,billing=1

dstack version

0.20.27

Server logs

Additional information

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions