Copy Files to a pod

Is there a way to copy files to a pod?, I am trying with connect_get_namespaced_pod_exec but I think is very a limited workaround.

Thanks

Hi, unfortunately it doesn't seem there is another way to cope with that as even the kubectl implementation or this ticket seems to suggest this is the only way.
Have you managed to copy files via exec() ?

@aogier

Hi @aogier,
Yes, I used the exec to copy files. My user case is coping scripts to pods (a rundeck plugin), so using exec is enough. However, my workaround won't work with binary files.

@ltamaster

Hi @ltamaster, AFAICT current examples/implementation use tar on a standard stream pipe, like:

source -> tar -> stdout -> kubernetes -> stdin -> tar -> target

so I'd expect it should work for any file type, as long as tar can cope with them. If you don't mind share your current implementation I could try to help you make things work and maybe promote it to an example script to give to others too :)

@aogier

Hi @aogier ,

I am doing something like this:

    # Calling exec interactively.
    exec_command = [shell]
    resp = stream(api.connect_get_namespaced_pod_exec, name, namespace,
                  command=exec_command,
                  stderr=True, stdin=True,
                  stdout=True, tty=False,
                  _preload_content=False)

    file = open(source_file, "r")

    commands = []
    commands.append("cat <<'EOF' >" + destination_file + "\n")
    commands.append(file.read())
    commands.append("EOF\n")

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print("STDOUT: %s" % resp.read_stdout())
        if resp.peek_stderr():
            print("STDERR: %s" % resp.read_stderr())

        if commands:
            c = commands.pop(0)
            resp.write_stdin(c)
        else:
            break

    resp.close()

Ah, I've seen that now. There are some problems with strings send over the websocket, this will in turn also become a python 2/3 issue. I think the problem reside in kubernetes.stream.WSClient implementation, I've trying patching his write_channel() method that way:

def write_channel(self, channel, data):
    """Write data to a channel."""
    self.sock.send(bytes(chr(channel), 'utf-8') + data)

now your modified example does not give errors:

# Copying file
exec_command = ['/bin/sh']
resp = stream(api.connect_get_namespaced_pod_exec, name, 'default',
              command=exec_command,
              stderr=True, stdin=True,
              stdout=True, tty=False,
              _preload_content=False)

source_file = '/bin/sh'
destination_file = '/tmp/sh'

file = open(source_file, "rb")

buffer = b''
with open(source_file, "rb") as file:
    buffer += file.read()

commands = []
commands.append(bytes("cat <<'EOF' >" + destination_file + "\n", 'utf-8'))
commands.append(buffer)
commands.append(bytes("EOF\n", 'utf-8'))
commands.append(bytes("date\n", 'utf-8'))

while resp.is_open():
    resp.update(timeout=1)
    if resp.peek_stdout():
        print("STDOUT: %s" % resp.read_stdout())
    if resp.peek_stderr():
        print("STDERR: %s" % resp.read_stderr())
    if commands:
        c = commands.pop(0)
        #print("Running command... %s\n" % c)
        resp.write_stdin(c)
    else:
        break

resp.write_stdin(bytes("date\n", 'utf-8'))
sdate = resp.readline_stdout(timeout=3)
print("Server date command returns: %s" % sdate)
resp.write_stdin(bytes("whoami\n", 'utf-8'))
user = resp.readline_stdout(timeout=3)
print("Server user is: %s" % user)
resp.close()

however, output is unexpected:

Server date command returns: None
Server user is: None

and copied file is truncated. I'd like to use tar rather than shell heredocs just for.

What do you think ? Ciao

Ok this PoC works well:

# Copying file
exec_command = ['tar', 'xvf', '-', '-C', '/']
resp = stream(api.connect_get_namespaced_pod_exec, name, 'default',
              command=exec_command,
              stderr=True, stdin=True,
              stdout=True, tty=False,
              _preload_content=False)

source_file = '/tmp/dash.tar'
destination_file = '/tmp/sh'

file = open(source_file, "rb")

buffer = b''
with open(source_file, "rb") as file:
    buffer += file.read()

commands = []
commands.append(buffer)

while resp.is_open():
    resp.update(timeout=1)
    if resp.peek_stdout():
        print("STDOUT: %s" % resp.read_stdout())
    if resp.peek_stderr():
        print("STDERR: %s" % resp.read_stderr())
    if commands:
        c = commands.pop(0)
        #print("Running command... %s\n" % c)
        resp.write_stdin(c)
    else:
        break
resp.close()

Things to do:

programmatically produce tar stream
send a PR against python-base repo

Some preliminary tests will be needed, though, as well as six'ing websocket/example code

HTH, ciao !

real tar file PoC implemented here, let's wait for kubernetes-client/python-base#52 needed for this to properly work.

ATM only client -> pod binary traffic works on py3, either direction seems to work in py2, ymmv

ciao

@aogier

Hi @aogier ,

I tested your code and It worked great. I was trying to figure out how I can set a particular destination path on the tar command, is that possible?

Thanks
Luis

@ltamaster

Hi @ltamaster, first of all I'm very glad this is working yeah :)

Defining destination path is indeed possibile and it depends on how you want to do it. I'll suggest you couple of option starting from what i prefer the most:

Using proposed `tar -C /`

From manual, tar's -C flag:

Change to DIR before performing any operations.
This option is order-sensitive, i.e. it affects all options that follow.

so if you want to put under /usr/local/foo/bar/:

oggei@cane:/tmp$ cd $(mktemp -d)
oggei@cane:/tmp/tmp.u9nN0XILDP$ mkdir -p usr/local/foo/bar
oggei@cane:/tmp/tmp.u9nN0XILDP$ echo hello > usr/local/foo/bar/my_file
oggei@cane:/tmp/tmp.u9nN0XILDP$ tar cv . > ../package.tar
./
./usr/
./usr/local/
./usr/local/foo/
./usr/local/foo/bar/
./usr/local/foo/bar/my_file
oggei@cane:/tmp/tmp.u9nN0XILDP$

That way, you have a /tmp/package.tar that when decompressed place a usr/local/foo/bar/my_file under your root filesystem. Tar being tar, this is not limited to a single file BTW (in fact even the example above ships 4 dir and 1 file).

A similar concept is used eg. in Debian packaging system, where a filesystem root is produced under debian/$pkg_name and then compressed in .deb package.

Runtime `-C` flag definition

You could write your code so you define operations' root via -C flag, and then you tar a hierarchy relative to this root. It's more or less like the above, except you start from a custom root and not / (so you can tar c file > archive.tar and then place it where you want via -C). Seems less straightforward to me, but it works.

OK that's all, happy hacking on that :)

@aogier

Thanks @aogier

Adding this on your code made the trick:
tar.add(name=source_file,arcname=destination_path+"/"+file)

ah brilliant, this is a lot simpler ... I admit it, didn't studied well the tar module 🐐

thanks for sharing !

BTW, I got an error message on the line resp.close():

ERROR: websocket: close status: 891

The file is copied OK; the message is produced when that line is called.

I've pushed a new feature/476-copy-example branch with an updated submodule revision pointer could you please try this one ? No problem happens to me with either python 2 and 3, double check you're using 7b133cf for repo and kubernetes-client/python-base@7d67d12 for base submodule ! :)

Ok, I will try that

@ltamaster

@ltamaster FYI I've just deleted old tags and squashed/rebased all the stuff now everything reside at 5cb61bb and two PR submitted and I'm still curious if everything works well for you :)

hope this will get accepted soon my work is done on that ciao ! 👍

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot

@fejta-bot: Closing this issue.

Details

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/reopen

@Ark-kun

@Ark-kun: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ltamaster

Hi @ltamaster，

i want to copy foo.tar.gz from pod, when i doing like this

    # Calling exec interactively

    src_path = pod/to/foo.tar.gz
    target_path = local/to/foo.tar.gz
    _exec_copy_command = ['/bin/sh', '-c', 'cat {src_path}'.format(src_path=src_path)]
    with open(target_path, 'a') as tar_file:
        resp = stream(self.api_instance.connect_get_namespaced_pod_exec,
                      pod_name,
                      namespace,
                      command=_exec_copy_command,
                      container=container,
                      stderr=True, stdin=True,
                      stdout=True, tty=False,
                      _preload_content=False)

        while resp.is_open():
            resp.update(timeout=1)
            if resp.peek_stdout():
                out = resp.read_stdout()
                # print("STDOUT: %s" % len(out))
                tar_file.write(out)
            if resp.peek_stderr():
                # print('STDERR: {0}'.format(resp.read_stderr()))
                logger.error(f'STDERR: {resp.read_stderr()}')
        resp.close()

however，Compare the MD5 of the two files, inconsistent

is there anything wrong？

@aogier

Hey, @aogier, thanks for sharing your code snippets earlier. I know it's been a while, but could you help me compile the final example of copying a file with this library?
Something like that fits in this signature:

def copy_file(kube_conn, namespace: str, pod_name: str, source_file, dest_path: str):
    pass

Unfortunately, most of the links you shared earlier don't work nowadays:

real tar file PoC implemented here

Thanks!

Okay, after several hours of experiments, I finally made it.
For future explores:

def copy_file(kube_conn, namespace: str, pod_name: str, source_file: pathlib.Path, dest_path: str):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode='w:tar') as tar:
        tar.add(source_file, arcname=pathlib.Path(dest_path).joinpath(source_file.name))
    commands = [buf.getvalue()]

    # Copying file
    exec_command = ['tar', 'xvf', '-', '-C', '/']
    resp = stream.stream(kube_conn.connect_get_namespaced_pod_exec, pod_name, namespace,
                         command=exec_command,
                         stderr=True, stdin=True,
                         stdout=True, tty=False,
                         _preload_content=False)

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print(f"STDOUT: {resp.read_stdout()}")
        if resp.peek_stderr():
            print(f"STDERR: {resp.read_stderr()}")
        if commands:
            c = commands.pop(0)
            resp.write_stdin(c)
        else:
            break
    resp.close()

@QIvan

This is my version, which passes gzipped data for better performance, based on @QIvan 's solution:

def k8s_copy_file_from_pod(namespace: str, pod_name: str, source_file: str, dest_path: str):
    exec_command = ['/bin/sh', '-c', f'cd {Path(source_file).parent}; tar czf - {Path(source_file).name}']
    with tempfile.TemporaryFile() as buff:
        resp = k.stream.stream(k.client.CoreV1Api().connect_get_namespaced_pod_exec, pod_name, namespace,
                      command=exec_command,
                      binary=True,
                      stderr=True, stdin=True,
                      stdout=True, tty=False,
                      _preload_content=False)
        while resp.is_open():
            resp.update(timeout=1)
            if resp.peek_stdout():
                out = resp.read_stdout()
                logging.info(f"got {len(out)} bytes")
                buff.write(out)
            if resp.peek_stderr():
                logging.warning(f"STDERR: {resp.read_stderr().decode('utf-8')}")
        resp.close()
        buff.flush()
        buff.seek(0)
        with tarfile.open(fileobj=buff, mode='r:gz') as tar:
            subdir_and_files = [
                tarinfo for tarinfo in tar.getmembers()
            ]
            tar.extractall(path=dest_path, members=subdir_and_files)

Okay, after several hours of experiments, I finally made it. For future explores:

def copy_file(kube_conn, namespace: str, pod_name: str, source_file: pathlib.Path, dest_path: str):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode='w:tar') as tar:
        tar.add(source_file, arcname=pathlib.Path(dest_path).joinpath(source_file.name))
    commands = [buf.getvalue()]

    # Copying file
    exec_command = ['tar', 'xvf', '-', '-C', '/']
    resp = stream.stream(kube_conn.connect_get_namespaced_pod_exec, pod_name, namespace,
                         command=exec_command,
                         stderr=True, stdin=True,
                         stdout=True, tty=False,
                         _preload_content=False)

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print(f"STDOUT: {resp.read_stdout()}")
        if resp.peek_stderr():
            print(f"STDERR: {resp.read_stderr()}")
        if commands:
            c = commands.pop(0)
            resp.write_stdin(c)
        else:
            break
    resp.close()

I had issues transferring directories larger than 35MiB or so with this function. Transferring in chunks seems to have solved this (Only patient enough to test up to 1GiB):

import io
import tarfile
import pathlib
from kubernetes.stream import stream

def copy_file(kube_conn, namespace: str, pod_name: str, source_file: pathlib.Path, dest_path: str):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode='w:tar') as tar:  # To compress set 'w:gz'
        tar.add(source_file, arcname=pathlib.Path(dest_path).joinpath(source_file.name))
    buf.seek(0)

    # Copying file
    exec_command = ['tar', 'xvf', '-', '-C', '/']   # To decompress set 'xzvf'
    resp = stream.stream(kube_conn.connect_get_namespaced_pod_exec, pod_name, namespace,
                         command=exec_command,
                         stderr=True, stdin=True,
                         stdout=True, tty=False,
                         _preload_content=False)

    chunk_size = 10 * 1024 * 1024
    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print(f"STDOUT: {resp.read_stdout()}")
        if resp.peek_stderr():
            print(f"STDERR: {resp.read_stderr()}")
        if read := buf.read(chunk_size):
            resp.write_stdin(read)
        else:
            break
    resp.close()

For my own use case I did not get an overall benefit from compression

Navigation Menu

Repository navigation

Copy Files to a pod #476

Using proposed `tar -C /`

Runtime `-C` flag definition

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Copy Files to a pod #476

Description

Activity

aogier commented on Mar 20, 2018

ltamaster commented on Mar 20, 2018

aogier commented on Mar 20, 2018

ltamaster commented on Mar 21, 2018

aogier commented on Mar 21, 2018

aogier commented on Mar 21, 2018

aogier commented on Mar 21, 2018

ltamaster commented on Mar 22, 2018

aogier commented on Mar 22, 2018

Using proposed tar -C /

Runtime -C flag definition

ltamaster commented on Mar 23, 2018

aogier commented on Mar 23, 2018

ltamaster commented on Mar 23, 2018

aogier commented on Mar 23, 2018

ltamaster commented on Mar 23, 2018

aogier commented on Mar 25, 2018

14 remaining items

fejta-bot commented on Apr 26, 2019

fejta-bot commented on May 26, 2019

fejta-bot commented on Jul 21, 2019

k8s-ci-robot commented on Jul 21, 2019

Ark-kun commented on Feb 13, 2020

k8s-ci-robot commented on Feb 13, 2020

sona201 commented on Feb 22, 2023

QIvan commented on Dec 8, 2023

QIvan commented on Dec 8, 2023

iTaybb commented on Dec 23, 2024

marcelldls commented on Mar 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Using proposed `tar -C /`

Runtime `-C` flag definition