Skip to content

Proposal: kubectl extension #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pwittrock merged 1 commit into from
Apr 5, 2017
Merged

Proposal: kubectl extension #122

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions contributors/design-proposals/kubectl-extension.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->

<!-- BEGIN STRIP_FOR_RELEASE -->

<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">

<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>

If you are using a released version of Kubernetes, you should
refer to the docs that go with that version.

<!-- TAG RELEASE_LINK, added by the munger automatically -->
<strong>
The latest release of this document can be found
[here](http://releases.k8s.io/release-1.3/docs/proposals/kubectl-extension.md).

Documentation for other releases can be found at
[releases.k8s.io](http://releases.k8s.io).
</strong>
--

<!-- END STRIP_FOR_RELEASE -->

<!-- END MUNGE: UNVERSIONED_WARNING -->

# Kubectl Extension

Abstract
--------

Allow `kubectl` to be extended to include other commands that can provide new functionality without recompiling Kubectl


Motivation and Background
-------------------------

Kubernetes is designed to be a composable and extensible system, with the ability to add new APIs and features via Third Party Resources
or API federation, by making the server provide functionality that eases writing generic clients, and by supporting other authentication
systems. Given that `kubectl` is the primary method for interacting with the server, some new extensions are difficult to make usable
for end users without recompiling that command. In addition, it is difficult to prototype new functionality for kubectl outside of the
Kubernetes source tree.

Ecosystem tools like OpenShift, Deis, and Helm add additional workflow around kubectl targeted at the end user. It is beneficial
to encourage workflows to develop around Kubernetes without requiring them to be part of Kubernetes to both the end user community
and the Kubernetes developer community.

There are many tools that currently offer CLI extension for the same reasons - [Git](https://www.kernel.org/pub/software/scm/git/docs/howto/new-command.html) and
[Heroku](https://devcenter.heroku.com/articles/developing-cli-plug-ins#creating-the-package) are two relevant examples in the space.


Proposal
--------

Define a system for `kubectl` that allows new subcommands and subcommand trees to be added by placing an executable in a specific

@rootfs rootfs Dec 2, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to support both containerized command and in-directory command. It is more extensible to for kubectl to respond to specific handler definitions (e.g. similar to how clink works)

@fabiand fabiand Jan 16, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - IMHO the primary should be on containerized extensions.

We have a use-case where we are shipping TPRs and require custom commands for them. Thus shipping kubectl extensions in containers (or DaemonSets then) would be very convenient. And would fit much bette rinto the broader story.

Just my 2ct

@philips philips Jan 17, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love containers as much as anyone but I feel that plugins ala git are much simpler thing to tackle on all of the platforms kubectl supports today.

location on disk, like Git. Allow third parties to extend kubectl by placing their extensions in that directory. Ensure that help
and other logic correctly includes those extensions.

A kubectl command extension would be an executable located in `EXEC_PATH` (an arbitrary directory to be defined that follows similar

@pwittrock pwittrock Dec 14, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-goal: Provide extensions as new flags to non-extension commands. E.g. kubectl get pod -o <my-own-extension-format>

@philips philips Jan 17, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another non-goal that is often requested: new auth mechanisms

@smarterclayton smarterclayton Jan 17, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm - this one comes up a lot, simply because we have no way other than "recompile the world". Are you just skeptical of how we can solve this cleanly?

conventions in Linux) with a name pattern like `kubectl-COMMAND[-SUBCOMMAND[...]]` with one or many sub parts. The presence of
a command extension overrides any built in command.

@technosophos technosophos Jan 19, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking for Helm, we started with a similar naming convention and later backed off. While it worked for Git, we felt like placing requirements on the naming convention of the plugin executable was limiting to plugin authors. It also required lots of name munging in code.

So we went from that requirement to having a single source location in $HELM_HOME (analogous to ~/.kube where plugin metadata could go. The metadata included information on the name of the actual binary.

That model got us the additional benefit of being able to read in the metadata file in order to generate top-level help text. In that way, helm -h could read the Usage: field in the plugin metadata for each plugin, and then display plugin help text alongside built-in help text.

@liggitt liggitt Apr 4, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The presence of a command extension overrides any built in command.

This seems like asking for trouble.

  1. Letting plugins override core commands is very fragmenting (examples wouldn't work consistently)
  2. Multiple plugins could target overriding the same core command
  3. I would expect commands like kubectl get and kubectl describe to be high-value targets for plugins trying to add in pretty-printing for their own types, and those are extremely difficult to wrap accurately
  4. Even if plugins are trying to behave well and pick names that don't conflict, future kubectl additions could be rendered inert by a plugin that had claimed the name for itself
  5. It makes support really tough ("run kubectl get foo. wait, what plugins do you have installed?")


A key requirement is that the lookup be fast (since it would be invoked on every execution of `kubectl`) and so some true extension
behavior (such as complex inference of commands) may not be supported in order to reduce the complexity of the lookup.

Kubectl would lazily include the appropriate commands in preference to the internal command structure if detected (a user asking for
`kubectl a b c` would *first* check for `kubectl-a-b-c`, `kubectl-a-b`, or `kubectl-a` before loading the internal command).

@soltysh soltysh Dec 5, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the solution to address the usecase of providing my own getter (or other existing command) for TPR, iow. if I want to have kubectl get mynewtpr/foo I need to define kubectl-get-mynewtpr?

@liggitt liggitt Dec 7, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get/edit/delete by resource/name, create/update from file should work without custom extensions

if you want commands like kubectl create mytype some-name --custom-option-1=something, you would need custom extensions

@liggitt liggitt Dec 7, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd want to start with a limited number of places to hook in custom extensions (like kubectl create x, kubectl set x). We also shouldn't differentiate between thirdpartyresources and other types unknown to kubectl (they should behave the same externally from the API)

@fabiand fabiand Dec 12, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lookup scheme from @soltysh above looks okay to me. The only drawback is that the extensions needs to have the boilerplate to do nice output.
It would work in my case wher I'd like to output some useful informations for a TPR.

But that is a problem which can be addressed later by making the default kubectl output functionality reusable.

@pwittrock pwittrock Dec 14, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is kubectl describe <resource-type> another case targeted here?

@liggitt liggitt Dec 19, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is kubectl describe another case targeted here?

I don't think so... I think that was supposed to be addressed by enabling returning descriptive info about a resource from the server (#123)

@technosophos technosophos Jan 19, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Helm, we backed off this approach because it got very confusing to the user. It was nearly impossible to tell whether a command was executing a plugin or not.

Envision this from the user's perspective. User A has command kubectl foo bar baz. User B tries to execute the same command, and it doesn't work. User A doesn't know which of their many installed plugins provides this behavior. How do they resolve the situation?

Our solution was to only expose a top level subcommand to plugins. This leads to a flat top-level namespace, but also makes it very clear which plugin is being invoked.

As a side benefit, it made the help system much simpler.


All kubectl command extensions MUST:

* Support the `-h` and `--help` flags to display a help page

@pwittrock pwittrock Dec 14, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably make extensions authors live's easier by refactoring kubectl and providing pieces of it as libraries.

@technosophos technosophos Jan 19, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We made this same requirement, and also used the plugin metadata file to load help text into helm at runtime. The consistency is a HUGE boon to users. 👍

We also required that certain flags (like --debug) follow the same conventions as Helm.

Implementation-wise, what we actually did was parse these flags in the top-level binary, swallow the flags, and then converted them into environment variables. The benefit here (aside from being closer to Cobra's intent) is that the top-level CLI enforces consistency and the actual toggling of flags on and off is an implementation detail:

https://github.com/kubernetes/helm/blob/master/docs/plugins.md#a-note-on-flag-parsing

* Respect the semantics of KUBECONFIG lookup (to be further specified)

All kubectl command extensions SHOULD:

* Follow the display and output conventions of normal kubectl commands.

@pwittrock pwittrock Dec 14, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will probably be some details to work out around how extension versioning is handled:

  • different versions of the extensions themselves (are these versioned independently from k8s versions?)
  • extension kubernetes client (e.g. golang client) version vs server version skew
  • extension kubernetes client version vs kubectl version skew

@adohe-zz adohe-zz Dec 15, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extensions version should be independently from k8s version.

@pwittrock pwittrock Dec 15, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But may have a dependency on the k8s version? e.g. "requires 1.7"?

@pigmej pigmej Dec 15, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think support defining compatible k8s version is must have. Like >1.4.0 & <1.5.0 (or other syntax).

@adohe-zz adohe-zz Dec 15, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is definitely necessary, extension plugin should be able to define k8s version requirement, and itself version should be independent.

@pigmej pigmej Dec 15, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adohe indeed.