Repository navigation
Error when trying to run a job #215
Description
I tried running a few different jobs and I get a message similar to the one below:
========================================================= ========================================================= ========================================================= logs from pod: c394fd78-1699-4d10-92ad-922f7bda2f37 ========================================================= ========================================================= ========================================================= failed to open log file "/var/log/pods/322513bd-524a-11e8-9e0d-000d3a0621a8/c394fd78-1699-4d10-92ad-922f7bda2f37_0.log": open /var/log/pods/322513bd-524a-11e8-9e0d-000d3a0621a8/c394fd78-1699-4d10-92ad-922f7bda2f37_0.log: no such file or directory ========================================================= end of logs from pod: c394fd78-1699-4d10-92ad-922f7bda2f37 =========================================================
I am using the alpha 1.5 release.
Is there a configuration I missed?
Activity
hongzhili commentedon May 8, 2018
Can you check if the job is running in kubernetes:
./deploy.py kubectl get pods
And if you can get the job log from kubernetes:
./deploy.py kubectl logs [pod_name]
If you cannot get the pod logs from kubectl. Please check the docker configurations on your host machine. E.g. if folder /var/log/pods exists and is accessible by docker daemon.
msalvaris commentedon May 9, 2018
Hongzhi Li (@hongzhili) Thanks for your help. Turned out to be an issue with the cuda drivers.