Slow PyTorch build on H100

[cleonard530]

I am on an H100 node with the following configs for my Pod:

...
  containers:
    - resources:
        limits:
          cpu: '64'
          memory: 256Gi
          rdma/rdma_shared_device_a: '1'
        requests:
          cpu: '4'
          memory: 32Gi
          rdma/rdma_shared_device_a: '1'
        claims:
          - name: gpu
...
    - name: MAX_JOBS
              value: '32'

which is the same configs I had on the H200 OpenShift Pod.

A full rebuild of PyTorch is taking me over 50 min, whereas on the H200 OpenShift cluster it was only about 20 minutes. I have checked htop and it looks like the cpu's I allocated are being fully utilized, so I am not sure what the issue is. I even increased MAX_JOBS to 64, but there was almost no gain.